While all 131 MPs of Orban's ruling right-wing coalition voted in favour, the landmark bill failed to pick up an extra two votes to reach a required two-thirds majority in the 199-seat parliament.
The rejection is an unexpected setback for the populist strongman who has already amended the constitution six times since winning a supermajority in 2010 and reelection in 2014.
It follows a divisive referendum last month, in which 3.3 million voters backed Orban's rejection of the European Union's plan to relocate migrants via mandatory quotas.
Observers warned that today's defeat risked seriously wounding Orban's political ambitions.
More From This Section
"After the referendum, this is the second time that the scenario that Orban wanted is not being realised. For a party that prides itself on being effective, this is a rare and significant defeat," analyst Csaba Toth told AFP.
The vote could set the scene for a growing power struggle between Orban's Fidesz and Jobbik ahead of the next scheduled general election in 2018.
Jobbik has been a natural ally in Orban's hardline campaign to stop Hungary taking in any of the hundreds of thousands of migrants who arrived in Europe in 2015.
Spotting a rare opportunity for leverage, Jobbik's leader Gabor Vona recently announced it would only vote for the bill on the condition that the government scraps a controversial cash-for-residency bond scheme for wealthy foreigners.
Jobbik said the scheme, often sold by shady offshore companies, posed a national security risk that could be exploited by Islamic State jihadists.
Orban, caught offguard by the ultimatum, initially said he would "consider" Vona's gambit, but later told parliament that the government "would not give in to blackmail".