The landslide happened late last night in Yaxi Township in the outlying Liandu District of Lishui City, the city government said in a statement.
Torrents of mud and rocks buried nearly 20 homes in the village, it said.
Rescue work began immediately after the landslide. The four bodies were recovered by this morning and the only survivor rescued was admitted to an area hospital, state-run Xinhua news agency reported.
Lishui reported heavy rain over the past two days. It was still raining when the landslide happened, the local weather bureau said.