"Everybody should know that when it comes to a brutal and inhuman regime, it is honourable to follow one's conscience and do the right thing," President Heinz Fischer said as the flat, X-shaped memorial was inaugurated in central Vienna.
"Hitler's army was not our army," Fischer said at a ceremony attended by four aged former deserters, adding that the long-held view that such people were traitors is "something to apologise for and to be ashamed of".
Hundreds of thousands of Austrians served in Hitler's armed forces during the 1939-45 conflict, with around a quarter of a million perishing on the Eastern Front and the many other theatres of war.
Around 30,000 death sentences were passed in total on those who refused to fight or were caught deserting, 20,000 of which were carried out, including on some 1,500 from Austria, renamed "Ostmark" by Hitler.
Also Read
Although many top henchmen from the Fuehrer downwards were Austrians, for many years the small Alpine country was slow to acknowledge its shared responsibility for the Holocaust and the other crimes of the Nazis, historians say.
"Until very recently deserters were considered by many people as traitors, even as murderers of their comrades," historian Walter Manoschek from Vienna University told AFP.
"It was part of the double-speak in Austria, which after the war portrayed itself as 'the first victim of Nazism' while seeing those who resisted Hitler as traitors," Manoschek said.
Richard Wadani, 92, who deserted on the western front in 1944 and is now president of an organisation campaigning for victims of Nazi injustice, said the Third Reich was "a regime that morally one could not fight for".