New Zealand need 105 to win after Southee took three for 19 in a nine-over spell of high-class seam bowling.
Only lusty hitting at the end by Shaminda Eranga and Suranga Lakmal, who added 59 for the last wicket, meant New Zealand faced a respectable target.
Under a cloudless sky and with an ageing ball, Southee extracted plenty of movement as Sri Lanka resumed the day at 293-5 requiring a further 10 runs to make New Zealand bat again.
Kaushal was first to fall for 12 with an edge to Mark Craig at second slip which brought Sri Lanka's last recognised batsman, Prasanna Jayawardene, to the middle.
Also Read
His attempt to mount a late stand with Mathews ended abruptly four overs later when the skipper pulled at a Southee bouncer and was caught down the leg side by wicketkeeper BJ Watling for 66.
Sri Lanka were 320 for seven, leaving their brittle tail exposed with a lead of only 17.
But it was a short-lived reprieve and two balls later he was gone as this time Taylor held the chest-high catch.
Southee featured in the final two dismissals, catching Jayawardene (23) off Craig and Lakmal (16) from the bowling of Trent Boult leaving Eranga undefeated on 45.
Southee finished with four for 91 and Boult, who removed the top order on Sunday, returned figures of four for 100.