The symbol, from ancient Sanskrit, means happiness and prosperity. It has been used for centuries by Hindus and Buddhists, and has turned up in archaeological digs in Europe.
But many Western tourists associate it with anti-Semitism and the Holocaust because the emblem was adopted by Nazi Germany to try to enhance a sense of ancient lineage.
At Sensoji Temple, a top tourist destination in Tokyo, a big gold "manji" emblem appears on a pair of lotus-shaped bronze ornaments, while smaller, more subtle ones decorate roof tiles. It's even an official emblem for Hirosaki, a city in northern Japan.
In a report released last month, a government panel at the Geospatial Information Authority proposed a three-tiered pagoda symbol to replace the swastika.
Also Read
It is one of 18 suggested icons for landmarks like hospitals and convenience stores for foreign-language maps, part of a broader push to create user-friendly maps for the growing number of foreign tourists, which jumped more than 40 per cent last year to a record 19.7 million.
Japan's main Buddhist group is nonchalant because the change doesn't affect domestic maps and therefore likely won't alter perceptions at home.
"We are aware that some people say the 'manji' symbol could remind them of the 'hakenkreuz' symbol, which was created much later in history," said Ryoka Nishino, a spokesman for Japan Buddhist Federation, referring to the "hooked cross" term often used to denote the Nazi emblem.
"Even though we have more foreign visitors, our symbol that decorates each temple will stay," he said.
Public opinion seems divided on Twitter and other social networks.
Instead, they say, the symbol should be kept as a way to teach people about the ancient history behind it. Others point out that the "manji" symbol turns the opposite way from the Nazi symbol, so it is different.