Two others who were trapped under the debris were rescued and had been admitted to hospital with injuries, they said.
The workers were demolishing the structure at Tiruvoodal village when the mishap occurred.
Preliminary investigation showed that adequate precautionary measures had not been taken before starting the demolition work, police said.
Further investigations are on.