"If the US withdraws, this has a much bigger political impact. I'm afraid that any leadership vacuum could be filled by others, and this is not what we want to see," Ban said today in an interview with The Associated Press.
He called the choice made by the US president "misguided" and said that it responded to "a short-term vision."
He also called on the public to raise its voice and said that commitment to the agreement from governors, mayors, business leaders and civil society in the US was "very encouraging."
Nearly 200 countries, including the US, signed the agreement in December 2015.
More From This Section
"I sincerely hope that President Trump stands on the right side of history," Ban said in Madrid.
As the world's largest economy, the United States is also the planet's number one polluter, accounting for 14 per cent of global emissions.
"The decision has implications and effects politically and psychologically much larger than 14 percent, because smaller countries have traditionally followed Washington's leadership" said Ban, calling on the US leader to show "global vision."
They also called on other signatories of the Paris agreement to "show greater urgency and commitment in the fight against global warming."
The leaders represented about half of the members of the Club de Madrid alliance, which Ban joined recently as an honorary member.
Among the signatories were former Chilean President and former UN envoy for climate change Ricardo Lagos, Nobel Peace winner and former East Timor leader Jose Manuel Ramos-Horta and former New Zealand Prime Minister Helen Clark, who also headed the UN's development agency.
The former South Korean foreign minister said the testing of ballistic technology is putting Kim Jong Un's regime "on the verge" of being able to develop a nuclear-tipped long- range missile.
"Never before since the Korean War has the level of tension been so high," Ban said, calling on North Korea "to stop all provocations" in order for the international community to restart a dialogue.
He also said that the leaders of the US and South Korea should come up with solutions to reduce the tension in the Korean Peninsula during a summit in Washington in late June.