The jihadists were travelling in a vehicle towards the southern Shabwa province, witnesses said. The three civilians were passing by in another car.
The United States is the only country that operates drones in Yemen, but officials rarely discuss the covert programme.
Last month, Yemen's President Abdrabuh Mansur Hadi defended the use of drones against Al-Qaeda in his country, which has killed dozens of militants in a sharply intensified campaign over the past year.
The drone programme has come under criticism from human rights activists concerned over civilian casualties.
Also Read
The United Nations said 16 civilians were killed and at least 10 wounded when two separate wedding processions were targeted in December.
The victims had been mistakenly identified as members of Al-Qaeda, it quoted local security officials as saying at the time.
Following the deaths, Yemen's parliament voted for a ban on drone strikes, but analysts say lawmakers are unlikely to be able to halt the US campaign.
Yemen is the ancestral home of Osama bin Laden and the home base of Al-Qaeda in the Arabian Peninsula (AQAP), which has been linked to a number of failed attacks on the US homeland.
The group has taken advantage of the weakening of the central government since 2011, when a popular uprising erupted that eventually forced president Ali Abdullah Saleh to step down after 33 years in power.