"I think Baghdadi's alive... And I'll believe otherwise when we know we've killed him," Mattis told Pentagon reporters.
"We are going after him, but we assume he is alive."
There have been persistent rumors that Baghdadi has died in recent months.
The Britain-based Syrian Observatory for Human Rights, a longtime conflict monitor, last week said it had heard from senior IS leaders in Syria's Deir Ezzor province that Baghdadi was dead.
Russia's army said in mid-June that it was seeking to verify whether it had killed the IS chief in a May air strike in Syria.
The Iraqi native has not been seen since making his only known public appearance as "caliph" in 2014 at the Grand Mosque of Al-Nuri in Mosul, which was destroyed in the battle for Iraq's second city.