Residents living within 100 metres of the factory were ordered to evacuate as 17 fire engines and nearly 60 firefighters battled the blaze.
Plumes of thick black smoke billowed into the sky from the factory belonging to Arakawa Chemical Industries, which manufactures chemicals for the paper industry.
"A 64-year-old worker, who had earlier been missing, was found dead," Takahiro Suzuki, a fire service official from Fuji City, told AFP.
The casualties were all male and aged in their 20s to 60s.
More From This Section
Fuji city is located in the foothills of the world-famous Mount Fuji, around 100 kilometres west of the capital Tokyo.
Officials said it was not immediately clear exactly what was burning but the factory operator denied risks of potential contamination or pollution.
"There is no concern" that chemicals in the plant could pollute the environment in the neighbourhood, said a company official who declined to give his name.
The factory produces resin for printer inks and chemicals used for making paper, added the official.