The lawyers withdrew because Morsi has been confined in a soundproofed glass cage in recent courtroom appearances to prevent him from shouting and disrupting proceedings.
The 62-year-old Islamist leader is facing four separate trials, three of which have now opened.
Morsi was brought to Cairo's police academy this morning by helicopter from the Burj al-Arab prison where he is held.
He and 35 others are accused of working with Lebanese and Palestinian groups to carry out attacks in Egypt.
Also Read
The court said it would appoint a new defence team, the BBC reported.
Morsi, Egypt's first democratically elected president, was overthrown by the powerful military last July following mass street protests against his rule that was plagued by political uncertainty and violence in a deeply polarised country that ultimately led to his ouster.
Since Morsi's exit, there has been a severe crackdown on his Muslim Brotherhood group, as well as on other activists seen as hostile to the military-backed government. The Brotherhood has been declared a terrorist organisation and authorities have punished any public show of support for it.
In this latest trial, Morsi is accused of collaborating with the Palestinian movement Hamas, Lebanon's Hezbollah and Iran's Revolutionary Guards. If convicted he could receive the death penalty. Proceedings in two other trials are underway.
The first opened in November on charges of inciting the killing of protesters near the presidential palace when Morsi was in office in 2012.
In January, another trial opened concerning his escape from prison in a jailbreak in 2011, during which police officers were killed. The fourth trial will be on charges of insulting the judiciary.
In his previous court appearances Morsi has struck a defiant tone and refused to recognise the legitimacy of the court, insisting that he is still Egypt's rightful president.
The Arab nation has been gripped by political turmoil since the overthrow of long-time dictator Hosni Mubarak in the Egyptian Revolution in 2011 and later under Morsi.