The man, whose name was not identified, was taken into custody.
US President Barack Obama was not inside the building at the time of the incident.
According to NBC news, the man was seen holding a stuffed Pokemon "Pikachu" doll and wearing clothing related to the animated show.
One witness said the man refused to heed agent's commands to "get down."
The agents had their guns drawn and eventually the man sat down with his knees to his chest, the news channel said.