The decision to quit the talent show means Simon Cowell will be the only returning judge from the four strong panel last year, reported Daily Mirror.
Ora, 25, has music commitments and has a part in the second "Fifty Shades Of Grey" movie.
She decided last week that doing "The X Factor" live shows and auditions would be too much.
"Rita will not be taking part in the show, 2016 is going to be all about her music and her career and that doesn't leave enough time for The X Factor and new talent," a source said.