It was not immediately clear who was behind the attack.
The refugees were hit by light weapons fire in waters off rebel-held Hodeida but the boat managed to dock in the city's port, an official there said.
The International Organization for Migration, which has operations in Yemen, said 42 bodies had been recovered.
The port official said dozens of Somalis who survived the attack, as well as three Yemeni traffickers, had been taken to the city's prison.
More From This Section
The rebel-controlled Saba agency accused the Saudi-led coalition battling the insurgents of attacking the refugees from the air but did not provide further details.
Major General Ahmed Assiri, spokesman for the coalition, dismissed the accusation and told AFP that the force had not been involved in fighting in Hodeida.
"There has been no firing by the coalition in this zone," the spokesman said.
The coalition has been backing pro-government forces waging an offensive from the south to retake Yemen's Red Sea coast from Huthi rebel forces.
It was unclear whether those on board were trying to leave Yemen or seek refuge there, but the IOM said it believed the boat was headed for Sudan.
Despite a war that has cost more than 7,000 lives since March 2015 and brought the country to the brink of famine, Yemen continues to attract people fleeing the horn of Africa.
Several refugee camps in southern Yemen host Somali refugees, although not in the Hodeida area.
The UNHCR said that as conditions worsen in Yemen, refugees are starting to use areas further to the north as a transit route.
It called on all sides in Yemen's war to protect civilians.
"UNHCR is appalled by this tragic incident, the latest in which civilians continue to disproportionately bear the brunt of conflict in Yemen," it said.