Cossacks and pro-Moscow militias were seen patrolling at some polling stations and Russian flags were being flown everywhere from city buses to convoys of bikers roaming the streets.
Ukraine's new government and most of the international community except Russia have said they will not recognise a result expected to be overwhelmingly in favour of immediate secession.
"This is a historic moment, everyone will live happily," Sergiy Aksyonov, the local pro-Moscow prime minister, told reporters after casting his ballot in the regional capital Simferopol.
The Black Sea peninsula is inhabited mostly by ethnic Russians and has been seized by Russian forces over the past month after the ouster of Ukraine's pro-Kremlin leader in February, plunging US-Russia ties to their lowest point since the fall of the Berlin Wall in 1989.
Also Read
Ukraine's interim President Oleksandr Turchynov, who last month replaced the ousted Viktor Yanukovych, called for a boycott of the ballot, accusing Moscow of preparing an invasion.
Some Crimeans who requested anonymity said they were planning to spoil their ballots in protest and there was a call on social media for people to stay at home and cook vareniki -- Ukrainian dumplings -- instead of going out to vote.
Accredited journalists including AFP were prevented from entering some polling stations in the port city of Sevastopol and the regional capital Simferopol, and several people were seen voting in Sevastopol even before the polls opened.
Voters can choose to become part of Russia or retain more autonomy but stay in Ukraine -- a vote for the status quo is not an option.
Preliminary results were expected to be announced soon after polls close at 8:00 pm (1800 GMT). The referendum committee said turnout was at 44 per cent a third of the way through voting.