He breathed his last at the Lari Cardiology Centre here where he was being treated.
He was 80 and is survived by his wife and a son.
Rai was the first elected Mayor of Lucknow and served for two consecutive terms, from 1995 to 2005.
He was close to former Prime Minister Atal Bihari Vajpayee.
Uttar Pradesh Governor Ram Naik, Chief Minister Akhilesh Yadav, SP supremo Mulayam Singh Yadav and BJP state president Keshav Prasad Maurya condoled the demise of Rai.
In his condolence message, Naik said Rai was a popular doctor and dedicated to public service.