The deceased, hailing from Beed village, were identified as Natwarlal Bhavsar (45) and his wife Kamla Bhavsar (38), Additional SP Devendra Patidar told reporters.
The couple might have taken the extreme step out of frustration, he said.
According to villagers, the couple left home early in the morning to take bath in Narmada river but later plunged into the reservoir from the dam's viewpoint.
Police said they have fished out the bodies and were investigating the case to find out if it was a suicide or an accident.