The 93-year-old leader's remarks exposed tensions in the ruling ZANU-PF party over who stands to take power after him, an event only expected when Mugabe dies but a generational change likely to prompt bitter battles.
Vice President Emmerson Mnangagwa, nicknamed "the Crocodile", is one of the top candidates.
But Grace Mugabe -- 41 years younger than her husband -- has become increasingly active in public life in what many say is a political grooming process to help her eventually take the top job.
Mnangagwa, 75, was appointed vice-president in 2014 and has been widely touted as the obvious successor to Mugabe, who has led the country for 37 years.
More From This Section
But a visibly incensed Mugabe unleashed wrath at his vice-president, sitting nearby, when speaking in Shona in the opposition stronghold city of Bulawayo.
"We are denigrated and insulted in the name of Mnangagwa. Did I make a mistake in appointing him as my deputy?" Mugabe asked.
"If it has come to this, it is time we make a final decision," he added, telling Mnangagwa and his supporters they were free to leave ZANU-PF.
"You can go ahead and form your party because we honestly cannot have this. We cannot be insulted on a daily basis."
Some rallygoers heckled Grace Mugabe, chanting at her "you know nothing" and "you are too junior" as the first lady spoke before a section of the crowd that broke into a popular local song "oyenzayo siyaizonda" which translates to "we hate what you do."
She then repeated her ambition to be appointed her husband's deputy.
"Even if I become vice president, is there anything wrong with that?"
Mnangagwa was last month stripped of his role as justice minister, in a cabinet reshuffle widely considered part of a campaign to reduce his powers and quash opposition within the government.