The 'Her' star was in talks for the role in mid-July but they could not come to a deal.
Scott Derrickson is directing the Marvel feature, and sources say the director and studio are already on the hunt for a new leading man, said the Hollywood Reporter.
Names that were being considered for the role before Phoenix started discussing the role included Benedict Cumberbatch, Jared Leto and Tom Hardy.
The film's story centers on the former neurosurgeon who serves as the Sorcerer Supreme - the primary protector of Earth against magical and mystical threats.