The fire broke out in a raised loft where 11 people were sleeping, said fire inspector Stefano Giannelli.
The blaze, which partially collapsed the factory's roof, was under control by midday, but the cause was not yet known, Giannelli said. Two people remained hospitalised, while two were released.
The Tuscan city of Prato has become a centre for Chinese-run factories producing garments for international retailers. Many of the workers are not declared to authorities, sleeping, eating and living in the factories where they work.
"This is on all of our consciences. We need to see the situation for what it is. This is the largest concentration of black-market labour in Italy's central north, probably all of Europe," Rossi said on Sky TG 24.
An ex-military police officer passing by said he notified authorities of the fire.
Leonardo Tuci told the Italian news agency ANSA that he saw smoke coming from the factory and Chinese workers screaming for help. One man was trying to put out the flames with a fire extinguisher.