The 'Toxic' hitmaker has just recently finished filming auditions for the second series of the pop star search show, and boss Cowell, 52, said that she and Demi Lovato are not afraid to hold back on contestants who are not up to standard, Aceshowbiz reported.
"Britney is quite mean, which I think you will discover," said Cowell.
However, Lovato disagreed with his statement and claimed, "She's tough, she's not mean. She's a sweetheart who is very honest," to which Cowell replied, "As sweet as a lemon."
"They are harder to please than me or L A (Reid). They are very happy giving people a 'no.' They definitely have their own opinions," Cowell added.
Spears, 30, who got her first break as a singer aged just 15, had said earlier that she is tough because she wants to give people a realistic taste of the music business.
'The X Factor' will return to screens in the autumn.