According to a new YouGov poll commissioned by the Press Association (PA) news agency, only 36 per cent of the British public think the heir to the throne has been beneficial to the monarchy.
Four years ago, that figure stood at 60 per cent.
"We've moved on 20 years. Charles's reputation, I thought, was completely rehabilitated," royal commentator Penny Junor told PA.
"He gets a hard time because his marriage failed. He's never been forgiven for that -- because Diana blamed him and he never ever said a word," she added.
More From This Section
Since then, the monarchy's image has recovered and the once out-of-touch Buckingham Palace press office has overhauled its strategy to focus on distributing positive information.
According to Junor, in the past two decades Charles has become a "much happier, more relaxed and more confident Prince of Wales".
"And I thought much more popular as a result," she said.
But Charles has once gain found himself in a negative spotlight in the lead-up to the 20th anniversary of Diana's death.
But new revelations have surfaced, including tapes showing the late princess of Wales speaking frankly to her voice coach about her loveless marriage to the heir to the British throne and about his affair with Camilla Parker Bowles, now his wife.
Camilla's popularity has also suffered as only 18 per cent of people think she has made a positive contribution to the royal family, down from 28 per cent four years ago.
The princes have recently given candid interviews about their struggles following their mother's death and have stepped up their royal engagements.