Among the nearly one-in-nine 14 and 15-year-olds who use their handheld device for at least four hours daily, grade scores suffer an average 14 percentage points across all subjects.
The deficit rises to more than 18 points in mathematics, figures from Japan's education ministry showed.
Nearly half of all third-year junior high school students questioned spend more than an hour a day on their phones, browsing websites, sending e-mails and playing games. Less than a quarter of those in the age group do not have a mobile.
Fifteen percent of them spend at least one hour on their device every day.
The results of the survey, which is the first of its kind by the education ministry, have sparked fears that schoolchildren are neglecting their books for the allure of the small screen.
Kazuo Takeuchi, who has studied the way youngsters use mobiles, told Jiji Press that children with the devices tend to lack confidence in their academic ability, and urged parents to set limits on their usage.