The Indian Ocean island's constitutional court will now decide whether the decision to dismiss Rajaonarimanpianina can be enacted.
The US embassy had pledged support for the president and called on parliament to put the stability of the country first, but the plea fell on deaf ears.
The motion was backed by 121 of the 125 lawmakers who voted, easily clearing the two-thirds majority required.
The parliamentary speaker, Jean Max Rakotomamonjy, said after the votes were counted: "It is officially declared that everyone accepts that the request of dismissal can be submitted to the constitutional court."
More From This Section
One who refused to take part, Lydia Raharimalala, claimed there had only been around 70 lawmakers in parliament when they were called to vote.
"There's something wrong, there was cheating," she said, and promised to take her claims to the constitutional court.
When Rajaonarimanpianina took power in January 2014 - after the first democratic elections since 2006 - it was hoped Madagascar would finally emerge from a deep political and economic crisis.
The island had been plunged into chaos in 2009 when Rajaonarimanpianina's predecessor Marc Ravalomanana was ousted in a coup.
But his opponents say he has failed to deliver on his economic promises and accuse him of stalling on the establishment of a high court and the implementation of promised reforms. Many of his political allies have turned their backs on him.
Andry Rajoelina, who briefly served as the island's transitional president in 2009, helped to get the president elected two years ago. But yesterday, he joined forces with Ravalomanana's camp to vote against him.
Lawmaker Tinoka Roberto, a Rajoelina supporter, said before the vote: "I am sorry, but nothing works in this country at the moment.