The Wednesday operation, in which all the militants were "annihilated" and a still-unclear number of hostages killed, comes a week after a spectacular series of ambushes on army posts in which more than 50 soldiers and police were killed.
"The armed forces have successfully completed the assault on the headquarters of the Second Military Region at Mukalla and have thoroughly cleansed it of terrorist elements," a defence ministry source was quoted by the official Saba news agency as saying.
Medical and other military sources said at least 12 people, including five soldiers, were killed in the assault to retake the building in Mukalla, in southeast Yemen.
"We received this morning the bodies of 10 people" killed in the attack, a medical source at Ibn Sina public hospital in Mukalla told AFP.
More From This Section
A military official confirmed earlier that at least three of the soldiers taken hostage were among the 10 dead.
He later said two special forces soldiers were also killed.
"Efforts are ongoing to find other possible victims under the rubble," said the official, who said most of the building's third and top storey had been destroyed.
The army soon retook most of the building, except for the top floor, where the militants held the soldiers captive.
A military source told AFP the army had fired on the building with artillery.
Mukalla is the capital of the southeastern province of Hadramawt and is a major port city.
Al-Qaeda in the Arabian Peninsula (AQAP) has intensified its attacks in southern Yemen.
On September 20, suspected Al-Qaeda fighters killed more than 50 soldiers and police in coordinated dawn attacks in neighbouring Shabwa province.
Washington regards AQAP as the global jihadist network's most dangerous affiliate and has stepped up drone strikes targeting the militants.