"I swear to faithfully fulfil the obligations of prime minister and to show loyalty to the king," Rajoy said with his hand on the Spanish constitution.
In a ceremony that was broadcast live on national television, the leader of the conservative Popular Party (PP) was then greeted by Felipe VI, who came to the throne in June 2014.
Rajoy is set to spend the next three days forming a new cabinet for his minority government, after a 10-month deadlock.
When the PP ruled from 2011 to 2015, it enjoyed an absolute majority in parliament.
Now, it must govern with 137 of 350 MPs, meaning that it will have to negotiate with the upstart centrist Ciudadanos, Basque and Catalan nationalists, and the main opposition Socialists on every bill it seeks to pass.