Russia President Vladimir Putin vowed to "respect" the outcome of the vote in a region that is now under the de facto control of Russian forces despite an international outcry.
Russian flags were being flown everywhere from city buses to convoys of bikers roaming the streets as thousands of people went to the polls in the strategic Black Sea peninsula.
"This is a historic moment," Sergiy Aksyonov, the local pro-Moscow prime minister, told reporters after casting his ballot in the regional capital Simferopol.
However, in the flashpoint eastern Ukrainian city of Donetsk, pro-Russian groups in favour of holding a similar referendum stormed the local headquarters of Ukraine's SBU security services and the prosecutor's office demanding the release of their self-appointed "governor," an AFP reporter said.
Also Read
Ukraine's new government and most of the international community except Russia have said they will not recognise a result expected to be overwhelmingly in favour of Crimea's immediate secession.
Ukraine's interim President Oleksandr Turchynov, who last month replaced the ousted Viktor Yanukovych, in turn accused Russia of fanning tensions in eastern Ukraine as a way of justifying an invasion.
"The result has been pre-planned by the Kremlin as a formal justification to send in its troops and start a war that will destroy people's lives and the economic prospects for Crimea," he said.