More than 14,000 people gathered to commemorate the victims in the western port city of Kobe, where a 7.2-magnitude quake hit at 5:46 am on January 17, 1995.
The quake, which killed 6,434 people, levelled much of the city and sparked a major review of quake preparedness in the island-nation that suffers about one fifth of the world's most powerful tremors.
The warning was brought into reality four years ago when a 9.0 magnitude quake struck in March 2011, triggering a huge tsunami that smashed into the country's northeast coast, killing around 18,000 people and creating the world's worst nuclear emergency in a generation.
Later in the day, Emperor Akihito and Empress Michiko attended a memorial ceremony in the city and offered a one-minute prayer with some 500 other participants.
More From This Section
"By learning lessons from the disaster we will do our best to build a community where we can live in safety," Eriko Yamatani, state minister for disaster management, told the ceremony.
The quake buried residents in flattened buildings and uprooted highway overpasses and train tracks, while fires raged through collapsed timber houses and acrid smoke darkened the sky.
Japanese broadcasters and newspapers reminded citizens of the high probability of another big quake, with geologists saying a major event is overdue.
"We want to speed up preparations for building a strong country against disasters," the Asahi Shimbun said in an editorial.
"We cannot eliminate damage completely, but we can start making efforts any day now to reduce damage," the daily said.