Amla, who had been facing criticism of his leadership after losing the Indian series 0-3 and then the first match against England, did answer back with a double century (201) at Newlands but decided to step down as skipper.
ODI captain AB de Villiers will lead the Proteas side in the remaining two matches.
Amla was named as Graeme Smith's replacement as captain in mid-2014 and this was his sixth series in charge. After winning his first three rubber's against Sri Lanka, Zimbabwe and West Indies, he oversaw a rain-affected drawn series against Bangladesh and then his team's first overseas defeat in nine years in India.
"I was deeply honoured when the position was given to me. Throughout my time as captain, I have always had the support of my team-mates and coaching staff . I remain committed to the Proteas across all formats and I will fully support whoever is shown as my successor."
More From This Section
South Africa are currently 1-0 down in the four-match series, after defeat in Durban and a draw in Cape Town.
"The decision has been purely based on that I think somebody else can do a better job. It was just something that crossed my mind and was working on my mind. My leadership style, and we have a very young team, maybe somebody else would have done a better job."
Cricket South Africa chief executive Haroon Lorgat said: "We respect Hashim's decision and the manner in which he thought about it and then communicated with me. It was consistent with his well respected personality
(Reopens FGN 41)
Amla captained South Africa in 14 Tests, winning four, losing four and drawing six.
The penultimate Test of the England series is slated to start in Johannesburg on January 14.
Meanwhile, de Villiers said he is proud to be given the responsibility of leading the national cricket side.
"I've said this before that it is an incredible honour to captain South Africa in any format. The captaincy has obviously come at short notice and is the realisation of a lifelong dream," de Villiers said.
Proteas coach Russell Domingo also said that De Villiers is the front runner for the job in the longer run.