The 74-year-old television personality, who was on the jury panel of the BBC One show since its inception in 2004, says she was never told the cause of her being let go from the show in 2009, reported ContactMusic.
"I've never really been given a real reason. I don't think I will ever know the answer," Phillips says in an interview with "Good Housekeeping".
Her departure sparked an ageism row as she was replaced by "Strictly" series 5 champion Alesha Dixon.
Philips adds her exit from the TV show came at tough time.
"Certainly after 'Strictly', I was in a very vulnerable situation. Prior to finding out that I was not going to be on the panel, I had lost my manager of 30 years to cancer.
"I couldn't think about fighting, I couldn't think about losing a job. I was in despair and grief-stricken... I didn't stand up or make my voice heard. I wasn't in a place to be able to do that," she says.