"Officers at the site saw the terrorist holding the knife and shot him," near the Old City's Damascus Gate, police said in a statement, adding that the two victims were ultra-Orthodox Jews.
Israel's Magen David Adom emergency service said the perpetrator was shot and killed.
The medics said the two victims were aged 62 and 65, and were both taken to hospital "conscious and in stable condition." One suffered light injuries and the other moderate.