All-rounder Shakib Al Hasan took 5-85 to help the hosts dismiss Australia for 244 shortly into the second session on the fourth day.
David Warner top-scored with 112 for the tourists, who were chasing a victory target of 265 after dismissing the hosts for 221 in their second innings.
Shakib lived up to his billing as the world's number one Test all-rounder, finishing with a match haul of 10 wickets with his left-arm spin and making 84 in Bangladesh's first-innings total of 260.
They appeared to be coasting towards their fifth consecutive Test win over the minnows when Warner and Smith resumed the day on 109 for two.
More From This Section
Warner, who was unbeaten on 75 overnight, reached his 19th Test century off 121 balls with two runs off spinner Taijul Islam.
But Shakib brought Bangladesh back into the game, dismissing both Warner and Smith (37) to put pressure on the tourists.
Pat Cummins hit a valiant 33 not out after Shakib bowled Glenn Maxwell (14) off the first ball of the second session, leaving Australia tottering on 199-8.
Hazlewood, who was already nursing an injury, has been ruled out of the second and final Test of the series.
- 'A great feeling' -
=====================
Prime Minister Sheikh Hasina, who was watching from the stands, waved the nation's flag as the jubilant players embraced and celebrated their historic win.
"It's a great feeling beating Australia, and I think it was a great effort by the boys, especially Shakib," said captain Mushfiqur Rahim of his all-rounder, who was named man of the match.
"That's the belief we had and also our spinners did a great job," he added, saying Bangladesh would now be looking to seal the series.
A dejected Smith said his men could have performed better with the bat in the first innings when they scored 217, since the bounce and spin proved especially devilish the second time around.
"It was a great Test match. Credit to Bangladesh, the way they played," he said.
"We probably let ourselves down a bit in the first innings with the bat. We probably needed to make a few more there."
Australia have not played a Test in Bangladesh since Ricky Ponting's team visited in 2006, six years after Bangladesh were granted Test status.
They were due to play two Tests in the country in October 2015 but the tour was cancelled amid security fears after a wave of attacks by Islamist extremists in the Muslim- majority nation.