The xenophobic Alternative for Germany (AfD) obtained around 21 percent in its first bid for seats in the regional parliament of Mecklenburg-Western Vorpommern, according to exit polls shortly after voting ended.
Merkel's Christian Democratic Union, meanwhile, garnered just 19 percent in its worst ever showing in the north-eastern state, while the Social Democrats maintained top place with around 30 percent.
Calling it a "proud result," Leif-Erik Holm, AfD's lead candidate said, "And the cream of the cake is that we have left Merkel's CDU behind us... Maybe that is the beginning of the end of Merkel's time as chancellor."
The polls are also held exactly a year after the German leader made the momentous decision to let in tens of thousands of Syrian and other migrants marooned in eastern European countries.
More From This Section
Although she won praise at first, the optimism has given way to fears over how Europe's biggest economy will manage to integrate the million people who arrived last year alone.
Her decision has left her increasingly isolated in Europe, and exposed her to heavy criticism at home, including from her own conservative allies.
"I am voting AfD. The main reason is the question over asylum-seekers," said a pensioner and former teacher who declined to be named.
"A million refugees have come here. There is money for them, but no money to bring pensions in the east to the same levels as those of the west," he said, referring to the lower retirement payments that residents of former Communist states receive compared to those in the west.
Most of them have already decided to abandon the state, preferring to head "where there are jobs, people and shops," said Frieder Weinhold, CDU candidate.