"L'ordre du jour" (in English "Agenda") had been among the favourites for the Goncourt prize -- the most prestigious in the French-speaking world.
Vuillard, 49, said he was taken aback on hearing he won for his elegant, 160-page book which charts how the financial support of German industrialists was crucial in Hitler's push for power.
"One is always surprised, sometimes fatally," he told reporters at the Paris restaurant where the winner was announced.
His novelistic account of Hitler's rise, which sticks doggedly to the facts, turns on a secret meeting in February 1933 between Hitler and the heads of Krupp, Siemens, Opel, IG Farben and other major industrial groups where they agreed to bankroll his election campaign.
More From This Section
As they left the room, Hermann Goering laughed, "These elections will be the last for 10 years and maybe for a century."
The Renaudot award, often seen as a consolation prize for those not shortlisted for the Goncourt, went to "The Disappearance of Josef Mengele", another book about the Nazis which walked the tightrope of historical fact.
Mengele managed to escape to Argentina and even got a West German passport in his own name in the 1950s so he could return for a holiday in his hometown.
"I wanted to understand what is left of a person after they have done that kind of evil," Guez told AFP after he won.
"I wanted to know what Mengele's life was like afterwards, whether he had been punished or not. I think in Europe today (with the rise of the far-right) we need to understand the extraordinary mediocrity of evil," he added.
Although his book was greatly admired, many critics had tipped Veronique Olmi to make history by becoming the second woman writer in a row to win the century-old prize.
Her novel, "Bakhita", based on another real-life story of a Sudanese slave girl who became a Catholic saint, is already a bestseller. It has won the Fnac prize and is also in the running for the Femina prize, which will be decided on Wednesday.
Among the other finalists was Alice Zeniter's "The Art of Losing", a powerful account of an Algerian "harki" family who sided with the French during the bloody nine-year Algerian war of independence that ended in 1962.
"We did not choose the winner on the basis of the sex or origin of the writer," said the doyen of French letters Bernard Pivot, who chaired the jury.
"The book is all that counts," he told AFP.