The suspect was carrying a rucksack containing knives when he was detained "following a stop and search as part of an ongoing operation", police said.
Whitehall, the political heart of London which houses Downing Street and other government buildings culminating in Parliament Square, and Parliament Street were closed to traffic as Metropolitan Police officers and forensics officers continued their investigation.
Prime Minister Theresa May was not in Downing Street during the police operation in Whitehall, her spokesman said.
The Metropolitan Police confirmed that the man in his 20s has been arrested under the UK's Terrorism Act and is being questioned at a south London police station.
More From This Section
"The man was arrested in Parliament Street, junction with Parliament Square, by armed officers from the Met's Specialist Firearms Command. The man, aged in his late 20s, was arrested on suspicion of possession of an offensive weapon and on suspicion of the commission, preparation and instigation of acts of terrorism," Met Police said in a statement.
"Knives have been recovered from him," it added.
According to reports, the arrest was the result of an ongoing anti-terrorism operation and that the arrested man was on the radar of intelligence officers.
Images from the site of the arrest showed two officers handcuffing a man, whose hands looked wrapped with a white bandage.
Eyewitnesses described seeing a back-pack lying on the ground near the suspect along with two knives - one referred to as a large bread knife.
Westminster area of London remains on high alert since the terror attack on Parliament on March 22, which claimed five lives including terror suspect Khalid Masood.
The gates of the Palace of Westminster, which houses the Parliament building, were quickly closed down as today's arrest unfolded.