The announcement comes a month after the capture by Iraqi forces of second city Mosul further east in a major blow to the jihadists.
In a televised speech, Abadi, dressed in military uniform and standing in front of an Iraqi flag and map of the country, announced "the start of an operation to free Tal Afar".
"I am saying to Daesh that there's no choice other than to leave or be killed," he said, using an alternative name for IS.
Tal Afar is located 70 kilometres (43 miles) west of Mosul, where US-backed government forces ended jihadist rule in July after a months-long battle.
More From This Section
In June 2014, IS jihadists overran Tal Afar, a Shiite enclave in the predominantly Sunni province of Nineveh, on the road between Mosul and Syria.
At the time it had a population of around 200,000, but local officials said it was now impossible to know the exact number still living inside the city as most are cut off from the outside world.
Abadi said that Iraq's paramilitary Hashed al-Shaabi forces would help various army, police and counter-terrorism units in Tal Afar.
The umbrella organisation, which is dominated by Iran- backed Shiite militias, has already been fighting to retake a number of other Iraqi cities from the Islamic State.
"In the early hours, the guns and flags turned towards their targets," said Ahmed al-Assadi, a spokesman for Hashad.
"Victory is near" in Tal Afar, an "Iraqi city taken hostage and humiliated for years by attacks from these barbarians," he said.
Once Tal Afar is retaken, Iraqi authorities intend to launch a fight to retake jihadist-held Hawija, in the province of Kirkuk, 300 kilometres north of Baghdad.
Jihadists still hold areas in Anbar, a western province that faces major security challenges.
IS, which declared a cross-border "caliphate" encompassing swathes of Iraq and Syria three years ago, has also suffered major setbacks in Syria, where around half of IS's de facto Syrian capital Raqa has been retaken by US- backed fighters.