The Nikkei ended 241.12 points lower at 15,880.33 after climbing 1.9% on Wednesday to snap losses on the first two trading days of 2014. Tokyo stock markets began this year's trading on Monday.
The US nonfarm payrolls report on Friday will further indicate how the world's largest economy is faring - and therefore, how fast the Federal Reserve will scale back stimulus. Economists polled by Reuters have forecast 196,000 jobs were added to the US economy in December.
Sony Corp, the second-most traded stock on the main board, surged 3.8%, its biggest one-day gain since June 18.
"Sony is being bought because it's been sold off recently and while other tech stocks like Panasonic are being bought back, Sony was still cheap," said Mitsushige Akino, chief fund manager at Ichiyoshi Asset Management.
"The market realises it was oversold and that the company will eventually do restructuring of its own."
The broader Topix index was down 0.7% at 1,296.75, with 3.02 billion shares changing hands, modestly down from a near three-week high of 3.04 billion set on Wednesday.
More From This Section
The JPX-Nikkei Index 400, which started trading on Monday, dropped 0.8% to 11,710.87.
BNP Paribas said it was bullish on Japanese equities in the first half of 2014, forecasting the Nikkei to reach 18,000 by June, but was cautious in the second half of the year.
"In the second part of the year, the domestic economy should slow after the rush in demand induced by the consumption tax hike; at the same time, fiscal stimulus effects should fade," BNP Paribas analysts wrote in a report, adding that the Nikkei would likely trade in a range of 15,000 to 18,000.
HSBC, on the other hand, said it remained 'underweight' Japan, saying that almost 35% of Japanese production was now overseas, lessening the positive impact of a weak yen on corporate earnings as some had expected.
"Our work suggests that the yen has much less impact on earnings than 10 years ago. This indicates that earnings may disappoint analysts' optimistic forecasts," it said in a report.