The 22-year-old 'Love Story' hitmaker and 18-year-old Conor were caught on camera getting cozy while enjoying a weekend together at the Kennedy family compound in Massachusetts, reported Ace Showbiz.
"They stayed close to each other, went on a stroll for two, and jumped on a trampoline as well," a source said.
The rumoured couple was also spotted having another outing on a beach side.
"Conor wrapped his arm around Swift's waist and she returned the gesture by placing her hand on top of his. They later hopped on a boat and sailed with a couple of friends," it added.
Swift is a self-proclaimed fan of the Kennedys. She was invited by Rory Kennedy to the premiere of her TV movie "Ethel" at the Sundance Film Festival after meeting Conor's grandmother sometime before.
Swift was previously linked to Joe Jonas, John Mayer, Taylor Lautner and Jake Gyllenhaal.