The police said bodies of Tapasi Sardar, Abhijit Mondal and Mihir De were recovered at different points along the river.
A two-year-old boy Kakan Mondal is still missing.
The police said the boat with 19 pilgrims was on way from Sukhchar in North 24-Parganas district to Koonnagar where they were to offer obeisance at Shakuntala Kalibari.
Sources said boat capsizes often took place along the stretch of the river in Hooghly district for non-maintenance of the ferry ghats.
Most of them are leased to private operators by either the district administration or the zilla parishad and the leaseholders pay scant regard to hoisting danger signals when the river is in spate, they point out.
They give examples of Telenipara Ghat at Bhadreswar and those at Chunchura and Chandernagore.
