Herath recorded figures of 5 for 89 -- his first five-wicket haul against Zimbabwe -- to bowl the home side out for 272 and give Sri Lanka a 232-run lead in the first innings.
But after the tourists opted not to enforce the follow-on, fast bowler Carl Mumba picked up three wickets to reduce Sri Lanka to 84 for four.
Although Herath had come into the series with 26 five-wicket hauls in Test cricket, he had never played against Zimbabwe in a 17-year career.
On Tuesday he joined countryman Muttiah Muralitharan and South Africa's Dale Steyn among an elite group of bowlers with five-wicket hauls against all of the other nine Test-playing nations.
More From This Section
"If you take my career until 2009, I had only taken about 40 wickets with no five-wicket hauls," said Herath.
Although Zimbabwe reached 134 for two on the third morning in reply to Sri Lanka's first innings total of 504, they ultimately succumbed to spin, with offspinner Dilruwan Perera picking up 3 for 51 as the hosts lost their last five wickets for just 19 runs.
However, it was fast bowler Suranga Lakmal who set Sri Lanka on their way in the morning session, which saw Zimbabwe resume on 126 for two, when he had Craig Ervine caught at slip for 64 in the eighth over of the day.
Zimbabwe had been rescued by their lower order in the first Test, but after the interval Perera ensured there would be no such recovery as he grabbed three wickets in three overs, including Sean Williams for 58 and Peter Moor for 33.
All that was left was for the stand-in captain to pick up a record fifth wicket, as he trapped Carl Mumba lbw.
Sri Lanka then tumbled to 44 for three in their second innings, as Zimbabwe's bowlers were rewarded for greater consistency.
First-innings centurion Dhananjaya de Silva attempted to rebuild with Karunaratne, but scooped a low full toss straight to extra cover in the final half-hour of the day.
Nevertheless, Zimbabwe faced a serious deficit as they look to level the two-match series.
"Let's be honest, if they put another 100 on it will be a lot," said batting coach Lance Klusener.
"But at the same time we'd like to finish this Test match on our terms.