Most residents were sleeping when the landslide hit the municipality of Salgar around 3:00 am, burying a large area in mud and debris.
The rush of mud and water "tore down everything in its path," Salgar Mayor Olga Osorio told RCN Radio.
The small town of Santa Margarita was practically "wiped off the map," she said.
The rescue operation, which was being carried out by 166 first responders, was suspended overnight Monday and will resume again early in the morning.
Also Read
Residents shoveled out piles of mud in the afternoon, trying to salvage what belongings they could, as dump trucks arrived to begin hauling away the mess.
President Juan Manuel Santos flew over the affected area and met with local officials.
"No one can bring the deceased back to you, that is something we deeply regret. But we have to get through this disaster and look to the future with bravery and strength," he said.
Several children were orphaned and are being cared for by the Colombian Institute for Family Wellbeing, he said.
Ex-president turned opposition leader Alvaro Uribe also visited the area, which is near where he grew up.
"I met a woman who was holding her three-day-old grandson. His parents are lost," he told radio station RCN.
"It's very painful what we saw."
Santa Margarita, the hardest-hit town, is one of four that make up Salgar, a municipality of 17,000 people in the department of Antioquia.
It was cut off from the rest of the municipality because of damage to an access road and a bridge, local media said.
Extra emergency teams, rescue dogs and humanitarian aid have been sent, said Red Cross spokeswoman Ana Carolina Gutierrez.
The area had been hit by several days of heavy rain.
Colombia's tropical climate and mountainous landscapes make it prone to landslides.
In 2010-2011, heavy rains caused flooding and landslides that killed 1,374 people and destroyed more than 100,000 homes.