"We have witnessed a deep failure for the independence movement," Rajoy told a news conference, adding that constitutional reform was "the only legal and responsible course".
Catalonia's leaders said 2.3 million people in the region of 7.5 million turned out for Sunday's vote, which was stripped of legal force after challenges from Madrid.
Rajoy, who fiercely opposes the breakup of Spain, said the vote was an "act of political propaganda" to which "two out of three Catalans paid no attention".
"I have never refused dialogue, with Mr Mas or anyone," Rajoy said, but reiterated that a "real referendum... Cannot be" under Spain's constitution.
Also Read
"If Mr Mas intends to reform the constitution... He has a perfect right" to propose reforms in the regional parliament, Rajoy said. "What he doesn't appear to have is support for that."
Demands for independence in Catalonia have grown over recent years despite Madrid's resistance, fanned by the economic crisis.