"Three bodies were retrieved from the scene and two others succumbed to their wounds later. The injured 39 are being treated at three hospitals," a fire official said in Bagerhat.
He added that 39 people were rescued alive while searches were underway to retrieve the trapped people as "we have been told that at least 100 people were inside the structure when it collapsed".
Officials and witnesses said the roof collapsed when around 150 people were working on the extended part of the under-construction building of Mongla Cement Factory, run by an army welfare trust.
"The coastguard and navy soldiers joined hands with fire service in the salvage campaign," he said.
In April 2013, a nine-storey garment factory complex collapsed just outside the capital Dhaka leaving at least 1,138 people dead, the world's deadliest building collapse.