The 41-year-old singer is said to be embarrassed as a judge after his act Carolynne Poole was booted off the show in favour of controversial performer Rylan Clarke.
Barlow is said to be preparing to reunite with his bandmates for a new album and tour so he won't have time to work on next year's series of the show, reported Sun online.
"The events of the last week have really confirmed in Gary's mind that he needs to get back to what he loves best," a source said.
"Gary is going to give this series his all and then it will be full steam ahead for Take That. The lads are enjoying a bit of freedom at the moment, but they are all united in their desire to get Take That back on the road. A new album and tour have been discussed and all five of the boys are making really positive noises," the source added.
Barlowe is said to have met up with his bandmates Mark Owen, Howard Donald and Jason Orange over the past week to discuss plans for their next assault on the charts.
The group were also reportedly joined by Robbie Williams, who reunited with the group for what was supposed to be a one off album and tour in 2010.