The roof collapsed in the afternoon when around 150 workers were working on the extended part of the under-construction building of Mongla Cement Factory, run by an army welfare trust.
Seven bodies have been retrieved from the rubble while 53 workers were injured, including six whose conditions are critical, a fire official said in Bagerhat.
"Searches were underway to retrieve the trapped people as we have been told that over 100 people were inside the structure when it collapsed," he said.
"The coastguard and navy soldiers joined hands with fire service in the salvage campaign," he said.
In April 2013, a nine-storey garment factory complex collapsed just outside the capital Dhaka leaving at least 1,138 people dead, the world's deadliest building collapse.