Randy, who has been on the panel of the show since it began, will instead become a mentor as producers still want him involved, reported TMZ online.
"There's some concern among the brass that Jackson is not well-suited for the mentor role... But they feel they need him because he's the glue that helps keep the show together," a source said.
Singer Mariah Carey has already signed on to be a judge on the show - after Jennifer Lopez and Steven Tyler quit and Nicki Minaj is expected to be another new addition.
Currently, it is hoped there will be four judges this year, with the third seat taken by a country music star and the fourth unknown.
Carey is not so keen to have Minaj on board and wants Lenny Kravitz instead.
"Mariah would love to have her dear friend Lenny Kravitz sit next to her on the 'American Idol' judge panel. She thinks he would be the perfect choice because he is extremely talented and knows music inside and out," the source added.
