The deal came during a meeting of the joint working group on prisoners and detainees formed by UN special envoy Ismail Ould Cheikh Ahmed.
There has been mounting international pressure to end the Yemen conflict, which the United Nations estimates has killed more than 6,400 people and displaced 2.8 million since March last year.
"It was agreed during the meeting to release 50 percent of the prisoners and detainees within the next 20 days," Mane al-Matari, media adviser to Yemen's foreign minister who heads the government delegation, told AFP.
The two sides will meet again on Wednesday to finalise the mechanism on how and when the exchange will take place, Matari said.
Also Read
"The Yemeni government is committed to release all the prisoners as per the agreement," he said.
Matari estimated that their number is in the "thousands", but the rebel source said there may be only hundreds of prisoners involved.
Following a two-day interruption, the two delegations resumed face-to-face talks on Monday after mediation efforts and an appeal by the UN envoy.
Those issues include implementing a UN Security Council resolution which orders the Huthis to pull out of territory they occupied in a 2014 offensive and surrender heavy arms they captured.
The two delegations remained far apart as the rebels demanded the formation of a consensus transitional government to handle other issues and the delegation of President Abedrabbo Mansour Hadi insisted he is the legitimate head of state, the sources said.