Officials said four buildings collapsed in the quake which struck the island in the early hours of the morning, but rescue efforts are centring on the tower block that tumbled onto its side.
Firefighters pulled survivors from the twisted concrete, trying to access apartments through windows and scaling the rubble with ladders.
The baby, a man and a woman were pulled dead from the block, officials said, with 29 residents hospitalised.
"The search and rescue work continues there, home by home."
Also Read
More than 100 residents have now been freed from the rubble, with local media reporting the tower contained more than 200 homes.
Officials were unable to give an estimate of how many were still trapped as they scoured the building.
At least 30 people have been freed from another residential seven-storey building with reports that over 200 have been rescued across the city and more than 100 sent to hospital.
"I used a hammer to break the door of my home which was twisted and locked, and managed to climb out," she told local channel SET TV, weeping as she spoke.
The shallow quake struck at a depth of 10 kilometres around 0130 IST today, according to the US Geological Survey, 39 kilometres northeast of Kaohsiung, the second-largest city on the island and an important port.
The quake was initially reported as having a magnitude of 6.7, but was downgraded to 6.4.
Taiwan lies near the junction of two tectonic plates and is regularly hit by earthquakes.
A strong 6.3-magnitude quake that hit central Taiwan in June 2013 killed four people and caused widespread landslides.
A 7.6-magnitude quake struck the island in September 1999 and killed around 2,400 people.