The hosts, lying at the bottom of the Test table in 10th place, piled up 501 in reply to New Zealand's 469 before being all out soon after lunch on the fourth day with a lead of 32 runs.
The Black Caps, ranked eighth above only Zimbabwe and Bangladesh, went to tea at 71-1 in their second innings, with Peter Fulton on 23 and Kane Williamson on 10.
Hamish Rutherford was removed after making a fluent 32, leg-before in Nasir Hossain's first over.
Gazi, a 22-year-old off-spinner whose highest score in six previous Tests was 32, remained unbeaten on 101 after coming in to bat at number eight. He hit 10 boundaries and three sixes.
More From This Section
Gazi and Robiul Islam exposed the limitations of the Black Caps' attack by adding 105 runs for the ninth-wicket to help Bangladesh grab the first-innings lead.
Robiul hit a career-best 33 before he was dismissed soon after lunch, caught off seamer Doug Bracewell, who returned as New Zealand's most successful bowler with three wickets for 96 runs.
Bangladesh added seven runs to their overnight score of 380-7 when they lost Abdur Razzak for seven, leg-before to seamer Trent Boult.
Bangladesh are looking for their first Test win against New Zealand, having lost eight and drawn one of their nine matches against the Black Caps.
The second Test will be played in Dhaka from October 21 to 25, followed by three one-day internationals and one Twenty20 match.