The Nikkei 225 index added 1.13 per cent, or 224.81 points, to finish at 20,133.90. It last closed above the psychologically important level in April 2000.
The broader Topix index of all first-section shares climbed 0.80 percent, or 12.91 points, to 1,621.79.
The last time the Nikkei was at that level Sony's groundbreaking PlayStation 2 was in stores, the Dot-com bubble was collapsing and Bill Clinton still occupied the White House.
Attention is now focusing on the start of Japanese earnings season with many firms set to report bumper fiscal-year profits.
More From This Section
"Where the market will go from here depends on how company earnings fare."
A weak yen also provided support while a pending trade deal could push the market higher, he added, as cash from domestic and foreign pension funds flows in.
Tokyo and Washington emerged Tuesday from marathon talks on a Pacific-wide free-trade area saying they were close to an agreement.
Japanese Prime Minister Shinzo Abe is hoping the Trans-Pacific Partnership (TPP) -- which includes a dozen nations accounting for about 40 percent of the global economy -- will help him push through reforms to the protected agricultural sector, a key element in his bid to stimulate growth at home.
Adding to buying sentiment, Japan on Wednesday posted its first trade surplus in nearly three years in March as lower oil prices reduced import costs and the value of exports to North America soared.