"No to war," chanted many among some 6,000 demonstrators according to march organisers as they rallied outside the Reina Sofia museum in the Spanish capital.
Protesters answered a "not in our name" petition from dozens of artists demanding Spain does not become embroiled in the Syrian conflict.
The petition has received some 34,000 online signatures to date as European governments consider armed intervention against Islamic State following the November 13 Paris attacks for which it claimed responsibility.
"Decisions have to be well thought though, as in any aspect of life," said Rajoy, who added Madrid was in touch with its allies pending a clear plan of action.
Also Read
Rajoy's Popular Party is mindful of how in March 2004 under his predecessor Jose Maria Aznar, who had backed the US intervention in Iraq a year earlier, lost general elections which took place three days after Islamic extremists killed 191 people in a series of Madrid train bombings.
Aznar's stance on Iraq was in stark contrast to that of the public in a traditionally pacifist country.