The incident happened at a place under Barun Police Station in the district along National Highway 2.
The deceased have been identified as Sunil Singh (32) and his wife Sandhya Devi (28).
Barun Station House Officer (SHO) Sanjay Kumar Sinha said, "Singh was driving the car, when he lost control over the vehicle. It swerved away from the road and fell into Dengra canal running close by. The family was going from Dhanbad in Jharkhand to their village in Bhojpur district here."
The children are admitted in the hospital and the police is trying to contact their relatives.