Witness Justin Knock captured images of the base jump from the 87-storey building and said the "dangerous" move was "skilfully" done with the jumper landing on his feet.
The base jumper, whose identity remains a mystery, landed at St Thomas's Street Saturday morning, police said, adding that no-one has been arrested.
The Shard building at London Bridge stands at 1,016ft (310m).
A spokeswoman for The Shard said: "We can confirm that we are investigating an incident involving a member of the public this morning. All parts of the building remain open."
Also Read
"It was obviously a dangerous thing to do but he pulled it off skilfully. He was coming down at speed and if he caught a draft and hit the building it would be game over really."
He and another man bundled up the parachute within seconds of landing and left the scene. Police and staff from the building arrived at the scene soon after, Knock said.
The landmark tower is one of the tallest buildings in the European Union but has no external viewing platform, which raises questions as to how the Base jumper was able to leap from the top.