Speaking in Cape Town yesterday, Annan said "quite a few leaders are resisting and fighting" the ICC but cautioned on a vote against the tribunal without an alternative.
"If they fight the ICC, vote against the ICC, withdraw their cases, it will be a badge of shame for each and every one of them and for their countries if they do that," he said.
Annan denied claims by African leaders that The Hague-based tribunal was targeting the continent and accused them of protectionism.
Amid growing opposition to the ICC, the African Union is set to debate the continent's relationship with the world's first permanent court to try genocide, war crimes and crimes against humanity at a special summit on Friday and Saturday.
Also Read
The ICC was a place of last resort that would not be needed if local courts were competent to try the cases, said Annan.
The meeting follows the 54-member bloc's accusations that the court singles out Africans for prosecution, pointing to the high-profile trials of former Liberian president Charles Taylor and top leaders of Kenya.
But Annan said this was "not the case", adding he had been proud that so many African countries with weak judicial systems had given strong backing to the ICC.
"I am therefore concerned by recent efforts to portray the Court as targeting Africa. I know this is not the case," said the Ghanaian diplomat.
The trial against Kenyan Deputy President William Ruto started in September, and President Uhuru Kenyatta's case will begin in November.
Both are charged with stoking violence after a disputed 2007 presidential election.
The AU has called for the ICC to drop the crimes against humanity trials of the leaders in Kenya, where lawmakers have voted to withdraw from the 10-year-old court.
African countries account for 34 of the 122 parties to have ratified the Rome Statute, the court's founding treaty, which took effect on July 1, 2002.