The wooden boat was believed to be smuggling 44 people including women and children from Malaysia's southern state of Johor to Indonesia's Batam island. It sank in rough seas late Thursday and four men were rescued the following day.
Asharudin Che Ami, an official with the Malaysian Maritime Enforcement Agency, said authorities may call off the search Tuesday after just one corpse was recovered early Monday and three on Sunday.
The boat capsized in high waves about 13 nautical miles off the coast with maritime authorities saying it was "not seaworthy".
The passengers are believed to have been illegal migrants who came to Malaysia to work but wanted to return to their country to celebrate Thursday's Eid al-Fitr, the end of the Muslim fasting month of Ramadan, without passing border controls.
More From This Section
One of the survivors found on Friday said he paid some 1,300 ringgit (USD 400) for what was to be his first trip home in 13 years.
In mid-July an Indonesian woman died and seven people went missing after their wooden boat capsized off Johor.
Twenty-seven Indonesians, heading to Batam without valid travel documents, were rescued.