The 60-year-old character actor had picked up almost every supporting actor award of the season, including the Golden Globe, the Critics' Choice, the SAG and the BAFTA.
The actor was presented with the first award of the ceremony by last year's best supporting actress winner Lupita Nyong'o, who paid tribute to Robin Williams in her brief speech.
Simmons thanked his wife and children while accepting his trophy.
"Wow, thank you Academy and everyone involved with 'Whiplash'. I am grateful everyday for the most wonderful person I know, my wife...," Simmons said.
Also Read
Simmons created one of the most compelling villains of the year in a category that often rewards such performances. This is his first Oscar win.
The Damien Chazelle-directed indie favourite premiered at the Sundance Film Festival in January last year.
Arguably without his performance, "Whiplash" would not have been the critical and awards success that it's today.
Simmons, who has been an in-the-trenches working actor for years, made the movie sing with his role of a teacher from hell.