Benjamin Smith, 44, a resident of upstate New York, threatened to "kill, kidnap and inflict bodily harm" on Bush, according to a federal court complaint.
Smith also professed a love interest in 32-year-old Barbara Pierce Bush, one of the former president's twin daughters, according to the complaint filed on Friday in US District Court in Manhattan.
"Bush will get his," Smith allegedly screamed as he was being arrested, said the complaint.
Smith also allegedly wrote: "I have to slay a dragon and then Barbara Bush is mine... America is finished. Obama, Etc."
More From This Section
Smith's defense lawyer, Peggy Cross-Goldenberg, did not immediately return a call seeking comment, CNN reported.
Smith's mother reported that a rifle was missing from the house and that she did not know where her son was.
Using cell phone tracking, the Secret Service located Smith, who they say had driven to Manhattan.
During questioning, Smith told Secret Service agents, "I'm divorced and not currently seeing anyone, but I am working on a relationship with Barbara Bush," the complaint said.
Threats against presidents of the United States are a federal crime. Bush, now 67, served as the 43rd President of the United States of America from 2001 to 2009.