Ravelonarivo and President Hery Rajaonarimampianina have been in conflict over issues including the poor condition of the island's main roads and rising crime in the capital Antananarivo.
Rajaonarimampianina won elections in 2013 but has been beset by opposition to his rule, with lawmakers trying to unseat him for alleged constitutional violations and incompetence.
"The president has accepted the resignation of Prime Minister Jean Ravelonarivo," the presidency said in a statement.
It gave no official explanation for the resignations.
More From This Section
Madagascar endured several years of turmoil after Marc Ravalomanana was ousted as president in a 2009 coup that led to the withdrawal of foreign investment and donor money.
In 2013, a presidential election that was designed to resolve complex power struggles brought Rajaonarimampianina to power.
Ravelonarivo took office as prime minister last year.
Madagascar remains one of the world's poorest countries, heavily dependent on foreign aid, and any renewed political trouble could threaten development.
The country off Africa's southeastern coast with a population of 23 million is famed for its unique wildlife, the result of evolution though geographical isolation.