Abu Sayyaf leader, Abraham Hamid, had led the kidnapping of several foreigners from a tourist resort in the volatile southern Philippines last year, two of whom were later beheaded.
"The death of Hamid is a big blow to the (Abu Sayyaf) as it neutralised one of the notorious bandits and will degrade their capability for spotting and kidnapping victims in the future," said regional military spokesman Major Filemon Tan.
Two other militants were killed alongside Hamid in the shootout with Malaysian police in Lahad Datu in eastern Sabah, he said.
Tan said Hamid had also been involved in the kidnapping of four Indonesian crewmen in April.
More From This Section
There have been a spate of kidnappings of Malaysian and Indonesian sailors at sea in recent months that have been blamed on the Abu Sayyaf.
While Hamid and two militants were killed, Sabah security forces have arrested two others, Tan added.
Sabah police chief Abdul Rashid Harun told AFP the incident was the Malaysian authorities' first direct confrontation with suspected kidnappers in the waters off eastern Sabah.
The Abu Sayyaf, a loose network of militants based on remote islands in the southern Philippines, has defied more than a decade of military offensives.
The group was formed in the 1990s with seed money from Osama bin Laden's Al-Qaeda network, but has been on a lucrative kidnapping spree in recent years.