Haddin, the vice-captain, has been overlooked in favour of the younger Peter Nevill, who impressed on his international debut in the second Test at Lord's after the 37-year-old pulled out to be with his sick daughter Mia.
Former captain Ponting, who played 168 Tests, said the decision did not sit well with him.
"I am disappointed with the decision to leave Brad Haddin out of the Test team for Edgbaston," he said in a column for The Australian newspaper, adding that he was "the heart and soul of the team".
"I know it would have been a hard call for him, he has been a single-minded and determined servant of Australian cricket, but he rightfully put his family first and has paid the price for it.
More From This Section
"I know he won't complain about being dropped. He is a tough character but it doesn't sit right with me."
The decision was made by on-tour selector Rod Marsh, himself one of Australia's greatest wicketkeepers, and coach Darren Lehmann, according to captain Michael Clarke.
Ponting added that he feared Test cricket was now over for Haddin and "I am a little concerned that the team may miss him more than they expect. I hope I am wrong".