Rajoy, leader of the conservative Popular Party (PP), lost the vote of confidence with 180 votes against and 170 in favour -- the same result he obtained during the first vote on Wednesday.
He was backed only by members of his own group, by centrist upstarts Ciudadanos and a lone member from a regional party from the Canary Islands.
Political leaders now have two months from Wednesday's parliamentary vote to explore alternative alliances but if their efforts come to nothing vote-weary Spaniards will be asked to return to the polls in December.
Spain has never had a coalition government since it returned to democracy following the death of longtime dictator Francisco Franco in 1975 and the country's four main parties have been unable to cobble together a governing alliance.
More From This Section
In his speech before the second vote, Rajoy made a final appeal to Socialist MPs to support his bid to form a government, and so avoid "running the risk of having to return to the polls."
Socialist leader Pedro Sanchez hit back at his PP rival, arguing that allegations of illegal financing and graft that have dogged the conservative party under Rajoy's watch had undermined his credibility.
"Rajoy has violated Spaniards' trust," he said.
During the second parliamentary vote Rajoy only needed to get more votes in favour than against, meaning he only needed the Socialists who finished second in both elections to abstain.
But Sanchez fears he would lose his own credibility with his supporters if he were to abandon his long-time opposition to Rajoy, according to members of his entourage.
"Enough is enough. We are the laughing stock of the entire world," said Luis Garcia Montero, a 53-year-old bank worker, as he smoked a cigarette on a street bench in Madrid before going to work.
Although Rajoy remains in office as acting prime minister, he has no power to propose legislation or spend on new infrastructure projects such as roads and railways.