Cowell, 53, has asked Sharon Osbourne to team up again with him and Louis Walsh, the sparring threesome who launched the show in 2004, reported Sun online.
"The first three series relied solely on genuine British talent and a chemistry between the judges that simply hasn't been there since Sharon left. By series three they had produced an international star with Leona Lewis and the show had real credibility.
"Now it's just a circus freak show with the ringmaster trying to pull the strings on the other side of the Atlantic. If Simon can get the original judges and format back he will truly have pulled victory from the jaws of defeat," a source said.
Next year is the last in X Factor's contract with ITV1. Current judges Gary Barlow and Tulisa have heavy schedules already and executives have told Cowell if ratings keep falling the series will be canned.