The 55-year-old music mogul and his new judging line-up of Rita Ora, Nick Grimshaw and Cheryl Fernandez-Versini, were supposed to start the first set of auditions in Manchester on Monday, reported the Daily Mirror.
"They were due to start tomorrow and carry on into Tuesday and Wednesday. Tomorrow's have been cancelled and we have not made any decisions beyond that," confirmed a spokesman for Syco Entertainment, the production behind "X Factor".
"Everyone at 'X Factor' is devastated for Simon and there was no suggestion that filming would go ahead," a source said.