The text, inked late yesterday, was hailed as a breakthrough in a long-stalled national dialogue between political parties and the government aimed at drafting a new constitution for Yemen and preparing for elections in February.
The southern question has been a major stumbling-block for the talks launched in March, with hardline factions of the secessionist Southern Movement boycotting the discussions.
The dialogue is part of a transition backed by the United Nations and the Gulf countries which saw President Ali Abdullah Saleh step down after 33 years in power following massive Arab Spring-inspired protests in the region's poorest country.
But hardliners of the Southern Movement boycotted the session.
More From This Section
The document was signed by representatives of 17 political parties and civil society groups who pledged to secure "a just solution to the southern question... On the basis of federalism," according to a copy seen by AFP.
But three political groups rejected the document today, showing the divisions that linger around the issue.
Saleh's General People's Congress signed the document, but a faction within the party rejected it, as did the Yemeni Socialist Party and the nationalist Nasserist Party.
The talks, which had been due to close on September 18, accepted the principle of a federal state, but stumbled over the number of regions that should be set up.
Hadi and northern delegates had suggested a federal state should comprise several entities while southerners have demanded a federal state made up of a north and south only.
Following the end of British colonial rule in 1967, southern Yemen was independent until union with the north in 1990.