The 36-year-old singer said that the boyband - which was signed by Simon Cowell - would not have stayed together for much longer if he hadn't left when he did as the other members - Shane Filan, Nicky Byrne, Kian Egan and Markus Feehily - "hated it", reported Femalefirst.
"If we had been given a break, it might have been different but when I look back now everything happens for a reason," McFadden said.
"None of the other guys were enjoying it at that stage. They hated it. But when I left, they all got scared with this feeling, how long is it going to last?
Westlife lasted for eight more years after Brian quit, eventually disbanding in 2012.