The West Indies were 184 for six at tea, still 368 runs behind South Africa's first innings total of 552 for five declared.
A double strike by Philander ended a promising opening partnership by the West Indies, with both opening batsmen falling to the accurate fast-medium bowler.
His third wicket was the crucial one, that of West Indian veteran Shivnarine Chanderpaul, and he dismissed Jermaine Blackwood in the last over before tea.
Kraigg Brathwaite and Devon Smith saw off South Africa's new ball attack, putting on 72 for the first wicket before Philander dismissed both in successive overs.
Also Read
Smith was given not out by umpire Billy Bowden. South Africa sought a review. It showed the ball clipping the outside of leg stump and the decision stayed with the umpire.
Smith went on to make 35 before being caught behind off Philander's fourth ball of a second spell.
He was given not out by umpire Bowden but Philander immediately asked for a review which showed a faint bottom edge as Smith played a cramped pull shot.
Brathwaite fell for 34 in Philander's next over, edging a low catch to Hashim Amla at first slip.
Philander took two for two in four overs in his second spell.
Kyle Abbott had Leon Johnson caught at cover for 33 after lunch and Morne Morkel took the important wicket of Marlon Samuels, who chopped the ball on to his stumps after making 33, the fourth successive batsman to be dismissed in the thirties.
Philander had taken four for 26 at tea.