In a brief resignation speech, Jiang Yi-huah said that he took "political responsibility" for the heavy losses suffered by the Kuomintang (KMT) party.
The polls are seen as an important test ahead of the 2016 presidential race, with China policy a key issue.
Official results showed the ruling KMT had lost in five out of Taiwan's six large municipalities -- the most hotly contested seats. Prior to the vote they had held four of them.
Under the KMT, previously frosty ties between Beijing and Taipei have warmed, but there is public anxiety over the closer relationship.
More From This Section
A proposed trade pact with the mainland sparked mass student-led protests and a three-week occupation of Taiwan's parliament earlier this year, reflecting frustration over what were seen as under-the-table negotiations and a lack of benefits for the general public from the deal.
The government has also been struggling with a slowing economy and a string of food scandals.
"It sent a very clear warning signal to the KMT that if it does not make fundamental policy adjustments to be more transparent and more responsive, it could face defeat in 2016."
Embattled President Ma Ying-jeou, who came to power in 2008 on a Beijing-friendly platform, must step down at the end of his second four-year term.
The opposition Democratic Progressive Party (DPP) has traditionally been sceptical over closer ties with Beijing and has criticised the KMT for lack of transparency over trade deals with China.
The president was "lacking caution" in his handling of growing ties with China, said Ding Shuh-fan, politics professor at the National Chengchi University in Taipei.
"This is a setback for Ma's China policy," he added.