Nearly 200 others were hurt and panic broke out as the massive crane crashed down on the mosque, one of Islam's holiest sites, as worshippers gathered for weekly Friday prayers.
Hundreds of thousands of pilgrims have already arrived in Mecca for the hajj, one of the world's largest religious festivals which last year drew two million worshippers.
A Saudi official said this year's hajj would proceed despite the tragedy.
Indonesia, the world's largest Muslim-populated nation, said two of its nationals were killed, while Malaysia and Iran said its citizens were among those injured.
More From This Section
As world leaders offered condolences, the governor of Mecca region, Prince Khaled al-Faisal, ordered an investigation into the incident.
Abdel Aziz Naqoor, who said he works at the mosque, told AFP he saw the massive construction crane fall after being hit by the storm.
"If it weren't for Al-Tawaf bridge the injuries and deaths would have been worse," he said, referring to a covered walkway that surrounds the holy Kaaba, which broke the crane's fall.
Local journalist Kamal Idris told AFP that Saudis and foreigners lined up last night to give blood in response to the tragedy.
Outside one hospital, more than 100 people waited in the street for their turn to donate, Idris said.
Pictures of the incident on Twitter showed bloodied bodies strewn across a courtyard where the top part of the crane, which appeared to have bent or snapped, had crashed into the building which is several storeys high.
A video on YouTube showed people screaming and rushing around right after a massive crash was heard.
Several other cranes towered over the site.
On Fridays, the Muslim weekly day of prayer, the Grand Mosque is usually at its most crowded.