Navy spokesman General Ayoub Qassem told AFP that the traffickers were on a vessel escorting a migrant boat headed across the Mediterranean to Europe.
The coastguards opened fire after coming under attack, said Qassem.
Four of the group, armed with automatic weapons and rocket launchers, were killed, two arrested and one went missing, the spokesman said.
The migrants' boat had left the scene without being hit.
"The coastguards tried to avoid the confrontation and called on the armed group to follow their orders, but in vain," Qassem said.
Smuggling is big business for gangs who cram migrants into boats that are small and unsafe for the perilous journey to Italy just 300 kilometres from western Libya.
In the absence of an army or a regular police force in Libya, several militias act as coastguards but are often accused of complicity or involvement in the lucrative trade.