The leaders termed the comment "unfortunate" and told reporters outside the House it was "disappointing" that Rajya Sabha chairman M Venkaiah Naidu did not allow a discussion on the notice.
They also objected to the "illegal disqualification" of rebel Janata Dal (United) leaders Sharad Yadav and Ali Anwar Ansari.
In place of Yadav and Ansari, Bihar Chief Minister and JD(U) president Nitish Kumar should be disqualified for breaking away from the grand alliance with the Rashtriya Janata Dal and the Congress, they said.
At an election rally in Gujarat on Sunday, Modi had suggested that Pakistan was trying to influence the state assembly polls.
More From This Section
He had claimed that some Pakistani officials and Manmohan Singh met at Congress leader Mani Shankar Aiyar's house over dinner on December 6.
Azad said the prime minister should come to the Rajya Sabha and explain his remarks to its members.
"But if it is false, if he levelled such a big allegation to polarise the people in view of the Gujarat polls, then he should apologise not only to the House but also to the people of the country," Azad said.
Raja said it was "not correct" on Modi's part to cast aspersions on Singh.
The CPI national secretary said Modi should have called up his predecessor if he had any information on the "conspiracy" issue.
Sharma said it was "undemocratic" that the Opposition's voice was being "muzzled".
He said the prime minister was "aware" of the notice and could have "used the opportunity to clarify his remarks, express regret and apologise".
"But the prime minister and the dispensation are arrogant and confrontational. And this is most unfortunate. This attitude does not augur well for democracy. It is an ominous development on the first day itself," Sharma said.