London Fire Brigade said there was no evidence of a blaze in the glass-clad, 1,016-foot tower, which pierces the British capital's skyline.
"As a precaution, the building was evacuated. It's thought around 900 people were in the building at the time," the LFB said.
The smoke dissipated and people were allowed back inside the building.
"No injuries were reported and the cause is not known at this stage," said LFB.
They said there were several possible causes for smoke without fire, such as an electrical fault.
More From This Section
Glen Cave, who was inside, told the BBC: "We had to walk to floors all the way down and it took about half an hour, but (there were) no dramas."
The Shard hosts restaurants, offices and the newly-opened five-star Shangri-La Hotel, whose guests were among those evacuated.
"We were going to be charged USD 990 a night, but now we are just standing out here in the cold now. We are finding another hotel," Wolfgang Grotzner, 62, from Hanover in northern Germany, told the Evening Standard newspaper.
However, it was soon overtaken by Moscow's 339-metre high Mercury City Tower, which was completed in November 2012.