The West Indies were 110 for two at lunch in reply to South Africa's 552 for five declared.
Kraigg Brathwaite and Devon Smith saw off South Africa's new ball attack, putting on 72 for the first wicket before Philander dismissed both openers in successive overs.
The right-handed Brathwaite looked solid and well organised from the start but the left-handed Smith survived a confident appeal for leg before wicket by Philander with his score on five in the second over.
Smith went on to make 35 before being caught behind off Philander's fourth ball of a second spell. He was given not out by umpire Bowden but Philander immediately asked for a review which showed a faint bottom edge as Smith played a cramped pull shot.
More From This Section
The catch was taken by AB de Villiers, who took over the gloves after regular wicketkeeper Quinton de Kock sprained his right ankle during the warm-up before play. Faf du Plessis was also not on the field, suffering from stomach cramps.
Philander took two for two in four overs in his second spell.