The rally, entitled "We are together", which began with Russian pop star Oleg Gazmanov singing patriotic songs, took place just outside the walls of the Kremlin.
Crimean parliament speaker Vladimir Konstantinov, who earlier met top lawmakers in Moscow, offered greetings "from Crimea" as he took to the stage amid cheers.
"Yesterday we took a historic decision," he said, referring to a request by the autonomous republic's parliament to join Russia, a move they want voters to ratify in a March 16 referendum.
"On March 16, all Crimeans, all Russians must come out and say just three words: Sevastopol, Crimea, Russia," Konstantinov said, referring to Russia's military base in the Crimean Black Sea port of Sevastopol.
Also Read
Television footage showed people waving Russian flags and holding printed placards with slogans including "Crimea, we are with you!" and "We believe Putin!"
Such rallies in support of Kremlin policies are carefully orchestrated by the authorities. State employees are pressured to take part and are often bussed in.
"We are supporting Crimea with our souls," one middle-aged woman told Russia 24 state television, which carried the rally live.
"We will not accept Bandera on their territory," said a young man waving a Russian flag, referring to wartime Ukrainian nationalist leader Stepan Bandera.
"I have a lot of relatives in Ukraine and so I want Ukraine to join the rest of Russia," another man said.
Rights activists opposed to intervention in Ukraine were set to hold a sanctioned rally in central Moscow later in the day. Authorities have limited attendance to a maximum of 500 participants.