"The court has judged the election of Miitig at the General National Congress (the interim parliament) as unconstitutional," a judge at the court said after a short hearing.
Outgoing premier Abdullah al-Thani had said he would await the judiciary's ruling before deciding whether to hand over power.
The ruling will be examined by the administrative court today, lawyers said.
Thani had announced his resignation earlier this year after an armed attack on his family, but he insisted that his successor should be chosen by a new parliament rather than its contested predecessor and refused to recognise Miitig's cabinet.
The GNC was elected in July 2012, in Libya's first ever free polls, almost one year after the revolution which ousted the regime of longtime dictator Moamer Kadhafi.
Its legitimacy was challenged after the GNC prolonged its mandate, due to expire last February, until December 2014.