Merkel's conservatives trailed center-left rivals in two states they had hoped to win.
The elections in the prosperous southwestern state of Baden-Wuerttemberg, neighboring Rhineland-Palatinate and relatively poor Saxony-Anhalt in the ex-communist east were the first major political test since Germany registered around 1.1 million people as asylum-seekers last year.
The three-year-old Alternative for Germany, or AfD - which has campaigned against Merkel's open-borders approach - easily entered all three legislatures, according to projections for ARD television based on exit polls and early counting.
"We are seeing above all in these elections that voters are turning away in large numbers from the big established parties and voting for our party," AfD leader Frauke Petry said.
Also Read
They "expect us finally to be the opposition that there hasn't been in the German parliament and some state parliaments," she added.
There were uncomfortable results both for Merkel's conservative Christian Democratic Union and their partners in the national government, the center-left Social Democrats.
It had hoped to beat left-leaning Green governor Winfried Kretschmann in Baden-Wuerttemberg, a traditional stronghold that the CDU ran for decades until 2011.
It also hoped to oust Social Democrat governor Malu Dreyer from the governor's office in Rhineland-Palatinate.
However, the projections showed the CDU challengers finishing up to 5 percentage points behind the popular incumbents in both states and dropping by more than 10 points to its worst-ever result in Baden-Wuerttemberg, with some 27 per cent.
The Social Democrats, meanwhile, suffered large losses in both Baden-Wuerttemberg and Saxony-Anhalt, where they were the junior partners in the outgoing governments.
In all three states, the results were set to leave the outgoing coalition governments without a majority - forcing regional leaders into what could be time-consuming negotiations with new, unusual partners.