"Nkala died at the Avenues Clinic, Harare, this morning after his admission at the hospital on August 7," the News Day newspaper reported.
He was 81 years old. The cause of death is still unclear.
Nkala co-founded the Zimbabwe African National Union (ZANU) party in 1963 to fight against British colonial rule.
ZANU came to power after independence in 1980 and later joined the Zimbabwe African People's Union (ZAPU) to form ZANU-PF, which remains in power.
Also Read
As defence minister he oversaw a state-ordered crackdown by North Korea-trained forces on dissidents in the Matabeleland and Midlands provinces between 1984 and 1987.
Around 20,000 people, including women and children, were killed during the massacre named "Gukurahundi", which means "the wind that sweeps away the chaff before the spring rains" in the local Shona language.
Nkala sought to exonerate himself, saying Mugabe directed the deployment of the troops.
In the late 1990s Mugabe described the killings as "a moment of madness" but stopped short of apologising for them.
He later became a staunch critic of Mugabe's rule, but last year patched up his relationship with the 89-year-old ruler.
Mugabe is set to be inaugurated for a new five-year term tomorrow after winning a disputed election last month to extend his more than three-decade rule.