The 19-year-old singer has told friends that he wishes he never went out with Swift, 23, after she took a swipe at him in her new music video 22, reported Sun online.
"Harry might have taken Taylor's swipes with an air of grace, but he now wishes he never ever dated her. I mean, the fact she's aiming a jab at him in the 22 video has got Harry wondering what's coming next! He just doesn't know what to expect," a source said.
Styles was previously said to have demanded that all footage of his ex-girlfriend be removed from the upcoming One Direction film.
The film, which is called '1D3D' and is due to be released in August, will show the group's meteoric rise to global stardom, their backstage antics and life on the road.