The man fell from 25 Bank Street, a 33-floor building in Canary Wharf, the capital's financial hub, around 8:00 am (0800 GMT), landing on a ninth floor roof.
Paramedics and air ambulance staff attended but he was pronounced dead at the scene shortly afterwards.
His death is being treated as "non-suspicious" at this stage, Scotland Yard said.
"Police were called at approximately 8:02 am (0802 GMT) on Tuesday 28 January to reports of a man having fallen from a building at 25 Bank Street, E14, and landing on a ninth floor roof," police said in a statement.
"No arrests have been made and the incident is being treated as non-suspicious at this early stage."
A spokesman for JP Morgan told AFP: "We are reviewing a very sad incident at 25 Bank Street this morning.