The grenade was filled with explosives but it was not immediately clear whether it was equipped with a detonator, a police spokesman in the Black Forest town of Villingen-Schwenningen said in a statement.
It did not explode and no damage or injuries were reported.
"Security staff discovered the intact explosive device and notified the police," the statement said, adding that the grenade was thrown over the fence at the facility at about 1 AM (local time).
He called for a swift probe to bring the perpetrators to justice.
Also Read
Justice Minister Heiko Maas expressed disbelief about the "horrific" incident, saying he would call a meeting of his counterparts in Germany's 16 states in March to discuss a stronger crackdown on far-right crimes.
"With explosives flying at refugee homes, we mustn't wait until there is the first death," he said.
Prosecutor Johannes-Georg Roth said it would be decisive to the investigation whether the grenade was live and only by chance did not detonate.
A 75-member special police commission was formed to carry out the probe, bringing in personnel from the Office for the Protection of the Constitution, Germany's domestic security watchdog.
Homes for asylum seekers have been repeatedly attacked in Germany, which saw the arrival of nearly 1.1 million people fleeing war and poverty last year.
Police in Villingen-Schwenningen said about 20 residents of the shelter were temporarily evacuated but were able to return to their rooms in the early morning hours.
In the statement, they asked witnesses with information about the incident to come forward.
Another 10 cases were reported in January.