Conor, 18, who is four years younger to the singer was not ready for anything too serious, reported Radar online.
"Conor's just an 18-year-old kid and wasn't ready for anything super serious. But Taylor is looking for her soulmate and it kind of freaked him out with how strong she came on," a source said.
Conor, son of Robert F Kennedy Jr and late Mary Richardson Kennedy and belongs to the famous political family of the Kennedy's and the singer was thrilled to be dating a member of it.
"She was more obsessed with the idea of dating a Kennedy, than the actual Kennedy she was dating," the source added.
The couple dated for three months. Swift has previously dated actors Taylor Lautner, Jake Gyllenhaal, John Mayer and Cory Monteith.