Another seven soldiers were wounded in the attack targeting troops who had been sent from the main southern city of Aden to Lahj province to fight jihadists, the sources said.
A military official said it was unclear whether the attack had been carried out by Al-Qaeda or the Islamic State group, both of which have gained ground in the south and claimed deadly anti-government assaults in the past.
The bombing came hours after clashes between troops and jihadists in Lahj, the military officials told AFP, adding that Saudi-led coalition jets also took part in the fighting.
The gunmen on a motorbike opened fire at Colonel Abdullah Shamba, killing him immediately before driving off, the official said.
More From This Section
Shamba headed a local anti-Al-Qaeda militia in Abyan.
Yesterday, coalition jets attacked Al-Qaeda positions in Abyan's provincial capital of Zinjibar and in the nearby town of Jaar, military sources said.
Government forces backed by the Arab coalition began an all-out offensive in March against jihadists in south Yemen, recapturing main cities they had held.
But they later retreated from Zinjibar after Al-Qaeda militants struck back.
The Arab coalition which backs the Yemeni government against pro-Iran rebels has also turned its sights on the jihadists, and the United States has pressed its drone war against them.
Washington considers the Yemen-based Al-Qaeda in the Arabian Peninsula, or AQAP, to be the extremist network's deadliest franchise.