A policeman was among those killed, they said.
The car slid off the Tonia Bridge and rolled down into a 40-foot-deep gorge, a police official said.
The bodies have been recovered from the gorge.
The victims have been identified as head constable Mohmmad Rasheed (48), Razia Begum (55), Mohmmad Zabir (35) and his wife Parvez Akhtar (30).
They all hailed from Rajouri district, police added.