Conor, son of Robert F Kennedy Jr and late Mary Nancy Richardson, belongs to the famous Kennedy political family.
Swift, 22, shattered her 18-year-old beau's heart when she left him, reported Daily Star.
"Conor's just devastated that Taylor chose career over love. She's making it appear Conor did the dumping because she didn't want to come out looking like the villain. But the bottom line is the poor kid is heartsick over the breakup. He believed they were headed to the altar," a source told the newspaper.
"He's still reeling from the suicide of his mom Mary in May, and the Kennedys are very concerned his broken heart will take a long time to heal. Taylor took advantage of Conor at a time when he was most vulnerable. Now that they've broken up, Conor is a wreck," a source said.