After stepping off a plane from the Brazilian host city, Tokyo governor Yuriko Koike carried the flag at a ceremony at the city's Haneda airport.
"I strongly feel a heavy responsiblity" for the next Olympics, Koike told the crowd.
"I'm very happy that we're able to bring the flag back after more than 50 years."
Tokyo last hosted the summer Olympics in 1964, underscoring Japan's post-war coming out party as it grew into a global economic powerhouse.
Also Read
Japan's Prime Minister Shinzo Abe made a comical cameo as Nintendo video game character Super Mario as Tokyo set a light-hearted tone for its hosting of the Games in four years.
Abe came out from a pipe after a video showed plumber Mario drilling down from Tokyo into the earth to reach Brazil.
"I wanted to show Japan's soft power to the world with the help of Japanese characters," Abe told reporters.
Abe pledged in Rio that he will work hard to host the best Games yet but Tokyo's Olympic preparations have suffered high-profile setbacks including soaring costs and having to redesign the Games logo after accusations of plagiarism.
French prosecutors have also launched an investigation into alleged bribes linked to Tokyo's winning Olympic bid, which organisers have denied.
Koike, who was elected in July as Tokyo's first female governor, has ordered officials to rein in ballooning costs and pledged a formal review, as concerns grow over soaring costs which could potentially double or even triple from the reported original forecast of $7.14 billion.
Tokyo's metropolitan conurbation is the world's largest with more than 35 million people, but streets are safe, trains run on time and the air is clean.
And with strict gun control and a public honesty visitors find disarming, few people ever experience serious crime.
The games in Brazil -- which is embroiled in a political crisis over the impeachment of suspended president Dilma Rousseff -- suffered its own series of setbacks.
Disclaimer: No Business Standard Journalist was involved in creation of this content