Zuma was left badly bruised when he fired Nhlanhla Nene last week in favour of little-known backbencher David van Rooyen, who was himself removed after only four days in office.
Zuma's African National Congress (ANC) party, which has ruled since the end of apartheid in 1994, won the general election easily last year but could lose power in some major municipalities at local polls in 2016.
"We had such hopes in 1994 because of all the struggles to remove apartheid," said Theresa Giorza, 55, a university teacher, at the protest in central Johannesburg.
"Over time, lots of people started to feel it was going wrong. It has gone from bad to worse. Now it's just rotten."
Also Read
The debacle over finance ministers triggered a market rout and fuelled opposition to Zuma, who has been buffeted by corruption scandals, a dire economy and charges of tarnishing Nelson Mandela's legacy.
Handing Van Rooyen the key finance portfolio fuelled fears that corrupt Zuma loyalists were calling the shots in government.
"This was another attempt to blunt the instruments of democracy," said Zwelinzima Vavi, one of the march organisers.
"Jacob Zuma has demonstrated beyond any reasonable doubt that he doesn't have what it takes to be a leader of a sophisticated democracy and economy."
Placards at the march in Johannesburg, which was held on the Day of Reconciliation, an annual holiday, read "21 Years of Undemocratic Rule" and "Recall Zuma Now".
Other protests were held in Cape Town and Pretoria.
In one of the biggest crises of his rule, Zuma was forced into a dramatic U-turn, sacking Van Rooyen and re-appointing Pravin Gordhan, a steadying hand who was finance minister from 2009 to 2014.