In the incident shortly after noon (0300 GMT), the light aircraft rammed onto the roofs of two houses in a densely populated neighbourhood near Narita airport east of Tokyo, local police said.
Television footage showed the white fuselage of the glider broken into several pieces and spread over the tile roofs of two residences.
The two men aboard, 69-year-old Shuji Fujioka and another man around the same age but not yet identified, were taken to hospital and later confirmed dead, officials said.
Last year three people died when a small single-engine propeller plane crashed into a Tokyo suburb shortly after takeoff.