The incident took place around 1.30 AM when the workers were sleeping on the factory premises, a police officer said.
The victims were identified as Vijay (55), Raja (28) and Babul (30), all residents of Agra.
The fire was noticed at about 1.30 AM following which the fire brigade was informed.
Fire fighters broke the wall of the factory and managed to control the fire, police said.
The charred bodies of Vijay and Raja were recovered from the ground floor while the body of Babul was found on the first floor of the factory, police said.
Shoes and other material worth lakhs of rupees were also destroyed in the fire, police said.
A case has been registered under various sections of the Indian Penal Code (IPC) against Rajesh, the owner of the factory.