They were arrested on terrorism charges Saturday at Frankfurt airport and the following day brought before a judge who ordered them placed in pre-trial detention, the federal prosecutors service said.
"The detained suspects allegedly travelled to Somalia between 2012 and 2013 to join the al-Shebab there. They allegedly received weapons training in a camp of the group and then joined them in armed combat," prosecutors said in a statement.
The suspects were identified only as Steven N, 26, Abdullah W, 28, and Abdulsalam W, 23, in line with German practice of protecting the privacy of criminal defendants.
There was "no evidence that the accused had prepared concrete plans for attacks" in Germany.
Also Read
Police also searched the home of a fourth, unidentified, suspect, said the prosecution service.
Shebab, Al-Qaeda's main affiliate in Africa, controls vast swathes of rural Somalia from which it carries out regular guerrilla attacks against government institutions and international forces based in the capital Mogadishu.
The Shebab has recently stepped up attacks against countries that contribute to the 22,000-strong, UN-backed African Union force deployed against them in Somalia since 2007.