"Madiba's life was the mirror image of the continent for the liberation of which he so relentlessly fought," AU Chairman and Ethiopian Prime Minister Hailemariam Desalegn said, referring to Mandela.
Photos of Mandela flanked the AU headquarters in the Ethiopian capital, while video footage of the former South African president played throughout the memorial service.
Mandela died on Thursday at the age of 95.
Hailemariam urged Africa to draw lessons from Mandela's life in order to build a prosperous future.
Also Read
Religious leaders opened the ceremony with various prayers before a portrait of Mandela surrounded by candles and flowers.
AU chief, Nkosazana Dlamini-Zuma, who served as Health Minister under Mandela, hailed him for unifying the continent in the face of racism.
"The unflinching pan-Africanness of Mandela and his peers across Africa inspired generations," said Dlamini-Zuma.
"Dada you have lived your life well, you have bequeathed us with a better world than the one into which you were born, you have now joined the galaxy of heroes, heroines, and the foundation of the pan-African movement," she said.