Cameron's friend Charlie Brooks said they were in the middle of a game at Chequers when Obama phoned.
The PM opted to reject the call and play the third set, Brooks claimed, according to BBC.
But officials have said they cannot find any record of the incident.
Brooks, also a racehorse trainer, was at Eton school with the prime minister.
Brooks told the Racing Post: "I played tennis with him (Cameron) at Chequers one day. I won the first set easily, then he won the second set, and then someone came up to him and said, 'Er, Mr Obama is on the phone for you, prime minister.'
"I thought, 'OK, we'll have to leave it there.' But he said, 'I think we've got time for a third set - tell Mr Obama I'll ring him back.' He obviously thought he had me on the ropes. And I beat him two sets to one."
The source said Downing Street officials are bemused by Brooks' remarks.
They also questioned Brooks' memory of the story, insisting the pair only played tennis once in the three times Brooks came to Chequers since the Coalition was formed. They have also checked the log for calls between Obama and Cameron - which show no overlap with Brooks' visit.
The tennis match, on a Saturday morning in August 2010, did not coincide with a phone call from Obama, sources said.
One Cameron aide said, "We are scratching our heads" over Brooks' recollection of the match.
