German conservative Mueller, 69, who served a five-year posting as head of the powerful department responsible for church doctrine, the Congregation for the Doctrine of the Faith (CDF), had clashed with the pope over key reform issues.
He was one of several cardinals who questioned Francis's determination for the Catholic Church to take a softer line on people traditionally seen as "sinners", including remarried divorced people who want to take Communion.
The Vatican said Mueller's five-year term would not be renewed and he would be replaced by CDF Secretary Archbishop Luis Francisco Ladaria Ferrer, a 73-year-old Spaniard.