It was his first public speech since Mugabe fired him on November 6 over a succession tussle with the former first lady, a move that prompted the military's intervention to force Mugabe from power, leading to his resignation yesterday.
Mnangagwa, who has close ties to the army and the security establishment, will be sworn in as president at an inauguration ceremony on Friday, officials said.
"Today we are witnessing the beginning of a new and unfolding full democracy in our country," he said in front of hundreds of supporters, some wearing shirts emblazoned with images of the 75-year-old leader.
He was surrounded by a large security detail and arrived at the headquarters of the ruling ZANU-PF party in a presidential-style motorcade.
More From This Section
Two young men held a stuffed crocodile above their heads, a reference to Mnangagwas's nickname, earned for his reputation for stealth and ruthlessness.
He had flown in earlier to Harare's Manyame airbase from South Africa, and met key ZANU-PF officials before heading to the State House, the nerve centre of Zimbabwe's political establishment, for a briefing.
"Great speech all round, can't describe how I felt seeing him after what he went through. All I want is job creation," said Remigio Mutero, 30, an unemployed IT graduate.
He had last been seen in public on Friday and had given a televised address on Sunday, but neither he nor his wife Grace have been seen since, with their whereabouts unknown.
On the streets, the news that his long and often brutal leadership was over sparked wild celebrations which lasted late into the night, with crowds dancing and cheering amid a cacophony of car horns.
A former key Mugabe ally, Mnangagwa had fled the country after his dismissal, saying he would not return without guarantees of his safety.
In a highly symbolic scene shortly after his resignation, a man took down a portrait of Mugabe from a wall inside the building where MPs had assembled for the extraordinary session to impeach the defiant president.
Another person replaced it with an image of the ousted vice president.
But critics describe Mnangagwa as a ruthless hard-liner who was behind years of state-sponsored violence, warning that he could prove just as authoritarian as his mentor.
Mugabe's resignation capped a chaotic week in which the military seized control and tens of thousands of Zimbabweans took to the streets in an unprecedented show of dissent against Mugabe, who left behind an economy in ruins.
"We hope to be able to access our money from the bank come December and the US dollar must come back," said Talent Chamunorwa, 37, a brick seller.
He was referring to Zimbabwe's chronic shortage of cash and a mistrusted scheme for "bond notes" whose value is supposed to be linked to the US currency, but which trade at a much lower rate in reality.
But efforts to position his 52-year-old wife Grace as his successor were his undoing.
Although Mugabe's fate remains unknown, the ZANU-PF has said he deserves to be treated with respect after leading the country for nearly four decades.
Last week's military takeover had all the hallmarks of a coup, but the generals stopped short of forcing Mugabe out, and Mnangagwa thanked army chief General Constantino Chiwenga during his speech today.
As the crisis grew, the ZANU-PF, an instrument of Mugabe's brutal reign, removed him as party leader and began parliamentary proceedings to have him impeached.
The government of China, a major political and economic backer of Zimbabwe, described Mugabe as a "good friend of the Chinese people".