With more than half the ballots counted, preliminary results showed 95.5 per cent of voters were in favour of leaving Ukraine and rejoining former political master Russia.
The vote would mark the most radical redrawing of the map of Europe since Kosovo's 2008 declaration of independence from Serbia, and follows the ouster of Ukraine's pro-Moscow leader.
"Our country does not approve the result" of the referendum, Chief Cabinet Secretary Yoshihide Suga, the Japanese government's top spokesman, told a regular news conference.
The comments came as US President Barack Obama hinted at possible additional sanctions on Russia, warning his counterpart Vladimir Putin that the United States and its allies would "never" recognise Crimea's breakaway vote held yesterday.
Suga said Japan would cooperate with other Group of Seven world powers in dealing with the Crimea issue.
Tokyo has started to study possible economic sanctions against Russia, the leading Nikkei business daily reported yesterday, citing unnamed government sources.