The 22-year-old was seen coming out of Swift's London hotel room sparking rumours that the duo are dating, reported Sun Online.
"You don't have to bang a girl just because she's your friend. It's not a necessity. So that's why I'd never go there with her. I wouldn't want to spoil things," he said.
Swift recently split from One Direction member Harry Styles.