Interior Minister Manuel Valls had earlier called for Gilles Bourdouleix, a controversial lawmaker and mayor of the western town of Cholet, to be "severely punished" for the comments.
Bourdouleix reportedly muttered the remark on Sunday as he confronted members of the travelling community, also known as gypsies, who had illegally set up camp, according to a recording posted on the site of regional daily Courrier de l'Ouest.
"Maybe Hitler did not kill enough," Bourdouleix is heard saying after the Roma had reportedly given him the Nazi salute.
Local prosecutor Yves Gambert said his office had opened a preliminary investigation into the remarks yesterday, on charges of "defending crimes against humanity".
Also Read
"I believed that this justified the opening of an investigation," he said.
Bourdouleix faces up to five years in prison and a USD 60,000 fine if convicted on the charge.
Prosecutors have also ordered that the recording of the remark be analysed to see if it was altered.
Bourdouleix has said his comments were taken out of context and alleged the recording was tampered with.
Prime Minister Jean-Marc Ayrault said Bourdouleix's comments were "not worthy of an elected representative" and were "punishable by law".