The blaze was noticed at around 4 PM, police said.
The fire has been brought under control, a senior official of the Fire and Rescue Department said.
The three storeyed building, housing the godown of Pothy's, a well known textiles showroom in south India, is located very near the temple.
The fire did not spread to neighbouring buildings, the official said, adding, the cause of the blaze is being investigated.
Panic had gripped the area when the fire broke out as there are several buildings nearby. At least 15 fire tenders had rushed to the spot to put out the blaze.