South Africa were 201 for seven at tea at Newlands, with Australia needing three more wickets in the remaining 33 overs to seal the first series win against the world's number one team since 2008/09.
It was a day of stubborn resistance by South Africa - who started on 71 for four - having abandoned any idea of chasing a target of 511.
AB de Villiers and nightwatchman Kyle Abbott defied Australia's bowlers for just over an hour and a half during an extended morning's play before Abbott, who batted for 116 minutes and faced 89 balls in scoring seven runs, did not play a shot to a ball from James Pattinson which cut back and clipped the top of his off stump.
De Villiers, normally a free-scoring batsman, had started his innings after tea on the fourth day, but was dismissed in the fifth over of the morning with the second new ball when he edged Ryan Harris to wicketkeeper Brad Haddin.
More From This Section
He had batted for 326 minutes and faced 228 balls in an effort reminiscent of De Villiers' innings in Adelaide when he batted for more than four hours and faced 220 balls in scoring 33.
Du Plessis was again resolute but shortly before tea he was leg before wicket to the first ball of a new spell by Smith, who pitched a ball on a good length on leg stump, turning slightly towards middle, with Du Plessis misjudging the line.
Du Plessis sought a review but it only confirmed that umpire Kumar Dharmasena had made the right decision. Du Plessis batted for 157 minutes and faced 109 balls in making 47.