Opener Dimuth Karunaratne top-scored with 79, Lahiru Thirimanne made 44 and and Dinesh Chandimal was last man out for 38, but the rest of the batsmen folded against Pakistan's incisive bowling in Galle.
The tourists, who led by 117 on the first innings, have the entire post-tea session to achieve the victory target of 90 runs and take the lead in the three-match series.
The 29-year-old Yasir ripped through the batting as Sri Lanka lost their last five wickets for 39 runs after being 167 for five at one stage.
The hosts lost nightwatchman Dilruwan Perera off the first ball of the day, clean-bowled by Yasir as he shouldered arms to a ball that dipped in after pitching.
More From This Section
The left-handed pair of Karunaratne and Thirimanne gave Sri Lanka the lead with a 69-run partnership for the fourth wicket.
But with the hosts just 15 runs ahead, left-arm seamer Wahab Riaz had Thirimanne caught low at first slip by Younis Khan to keep the interest alive during the lunch break.
The New Zealander upheld on-field umpire Richard Illingworth, who had ruled that the batsman was caught at short-leg off Yasir even though replays proved inconclusive on whether the ball had come off the bat.
Mathews, who had called for a review as soon as the umpire raised his finger, was visibly furious as he returned to the pavilion.
Neither hot-spot or snickometer technology is part of the Decision Review System for the series.