Ma resigned as head of the Kuomintang (KMT) following the party's humiliation in local elections Saturday - seen as a key barometer ahead of the 2016 presidential race.
In a sombre meeting earlier in the day President Ma bowed deeply as a sign of apology as he formally presented his resignation to senior party officials, which was televised live.
"The KMT suffered an unprecedented defeat. I apologise to the supporters as the chairman... I feel shamed," he said.
Ma will retain the presidency until 2016 when he must step down after serving two terms, but observers say his influence within the party will now be severely diminished.
Also Read
His resignation as chairman, however, allows the KMT to begin the difficult task of reshaping itself, which began with the announcement of a new premier today.
Previous vice-premier Mao Chi-kuo will replace Jiang Yi-huah who resigned on Saturday night, hours after the party's election humiliation.
"We will remember that the country belongs to all people and the future belongs to the new generations," Mao said today.
But public sentiment has turned against the Beijing-friendly approach as voters say trade deals have been agreed in secret and not benefited ordinary Taiwanese people.
Taiwan and China split in 1949 at the end of a civil war, but Beijing still claims the island as part of its territory awaiting reunification - by force if necessary.
The continued stagnation of the island's economy and a string of food scandals have added to the KMT's woes.
Ma, who had announced his imminent resignation yesterday, urged the party to remember the "painful lesson" of the polls result.