Tokyo won the right to host the Games after seeing off Madrid and Istanbul in votes held yesterday in Buenos Aires.
But Nadal, the world number two tennis player and a tireless campaigner for Madrid, said he was shocked to see Madrid, who were trying for the third time to win the Games, eliminated in the first round of voting.
"It's very hard and tiring for all of us, because the country and the city of Madrid worked a lot to have the chance so many times," said the 27-year-old.
"I felt that after a lot of years working, I don't know if it's 100% fair that we go out at the first qualification when our opponents haven't been in that position as many times."
Nadal was speaking after booking a place in Monday's US Open final, where he will take on world number one Novak Djokovic.