Prime Minister Shinzo Abe accepted Masahiro Imamura's resignation today.
Imamura was replaced by Masayoshi Yoshino, former deputy environment minister from Fukushima, which was also hit by radiation leaks from a tsunami-hit nuclear power plant.
Nobody died from radiation, but overall, the tsunami and the quake killed more than 18,000 people across northern Japan.
Imamura's resignation comes a day after he made the remark in a speech at a ruling party reception, which Abe also attended.
More From This Section
Imamura immediately retracted the comment and apologized, but Abe's face reportedly froze.
Imamura came under fire earlier this month over his outburst and a suggestion during a news conference that those who left voluntarily following the meltdown at the Fukushima nuclear plant should fend for themselves.
His blunder is the latest in a string of remarks and scandals that have plagued Abe's government in recent months, prompting opposition lawmakers to step up their effort to weaken Abe's grip on power.
Imamura's predecessor faced allegation he stole female underwear. Also, earlier this month, a trade vice minister was forced to quit over an adultery scandal.