Officials filed charges of human trafficking against the detained men, who were among those rescued after the doomed boat went down early Wednesday.
The four victims, all of them women, died after the vessel took on water near Miami.
The boat was carrying a total of 15 people -- Haitians, Bahamians and Jamaicans -- who were trying to enter the United States illegally.
The US Coast Guard launched a sea and air rescue, plucking survivors from the ocean about 11 kilometres off the coast of Miami.