Walsh, 63, who will judge its latest season after a year off, said the two had turned the "X Factor" into a fashion show and knew nothing about talent, reported Contactmusic.
Grimshaw, 31, and the "Body on Me" hitmaker co-judged the reality show last year.
"Sharon and I are old, but that means we're far more experienced, unlike the young whippersnappers on last year's panel who didn't know a thing about talent," he said.
He criticized the last year's panel for being lenient over selecting the contestants and even called them unexperienced.
Grimshaw and Ora have officially left the show.