Demonstrators rallied at the city's Westbahnhof train station before marching down a major shopping thoroughfare.
The march took place as a service was held for the dead at St Stephen's Cathedral.
"We've had enough -- enough of the deaths, the suffering and the persecution," the archbishop of Vienna, Cardinal Christoph Schoenborn, told the congregation, which included senior government members.
He said it was "too awful" to think of the plight of the migrants in the truck, four of whom were children.
The march and service took place shortly after trains carrying hundreds of migrants arrived from Budapest at Westbahnhof station after they had been stopped at the Austrian border with Hungary for several hours.
After pulling into Westbahnhof, many migrants boarded a train to the Austrian city of Salzburg, while others climbed on to another one headed for Munich in southern Germany, with police looking on, an AFP correspondent at the scene said.