A clearly exasperated Foreign Minister Fumio Kishida told reporters the government was trying to prevent Yukio Hatoyama from making the trip, amid fears he might muddy the diplomatic waters and legitimise Moscow's contentious move.
"We understand former prime minister Hatoyama has left for Russia. We will continue our efforts" to stop him, Kishida said.
Hatoyama's plan to travel to the peninsula first surfaced on Friday, as it emerged Japanese officials were urging him to abandon the trip.
"I want to see with my own eyes how people in Crimea are feeling" about the annexation, Hatoyama told them on Monday.
More From This Section
Media say the former premier is expected to fly to Crimea later Tuesday and stay there until Thursday.
Russian television this week revealed how President Vladimir Putin gave the secret order for his troops to move into Crimea in February last year.
The Ukrainian province was formally annexed by Moscow on March 18, triggering international condemnation.
Tehran cited Hatoyama as saying Iran was the victim of "double standards" by the international nuclear watchdog over what it says is a peaceful atomic power programme.
The US and its allies say Tehran is trying to build a nuclear bomb.
The comments, which Hatoyama later denied, earned him a ticking-off from Tokyo when he returned home.
Hatoyama became prime minister in 2009 at the head of the Democratic Party of Japan, but his chaotic premiership ended just nine months later after a series of policy flip-flops and blunders.
His off-the-wall image was aided by his wife Miyuki, who famously said her soul once visited Venus on a triangular spaceship and that she met Hollywood star Tom Cruise in a previous life.
In January, the former prime minister donned women's clothes to star in a musical called "Waist Size Story."
Wearing a bright pink dress, he played an elderly woman named Rosario who served as the first female president of the United States.