Organisers said 7,500 people gathered at a park in the city centre, including disaster victims and celebrities such as Nobel literature laureate Kenzaburo Oe.
Protesters later marched through the capital, holding anti-nuclear banners including one which read: "No Nukes! Unevolved Apes Want Nukes!"
They also demonstrated outside the headquarters of Tokyo Electric Power Co, operator of the Fukushima Daiichi nuclear power plant which was crippled by meltdowns after the March 2011 tsunami.
Abe, whose Liberal Democratic Party has close ties with the nation's powerful business circles, has repeatedly said he would allow reactor restarts if their safety could be ensured.
Radiation from the plant, 220 kilometres northeast of Tokyo, spread over a wide area after the worst nuclear accident since the 1986 Chernobyl disaster.