The US Court of Appeals for the District of Columbia Circuit ruled 2-1 that the conspiracy case against Ali Hamza al-Bahlul was legally flawed because conspiracy is not a war crime.
The system of military commissions was created by the administration of President George W Bush after the September 11, 2001 terrorist attacks.
In the al-Bahlul case, the Obama administration argued that Congress acted within its authority in making conspiracy a crime that can be tried by military commission.
Writing for the court, Judge Judith Rogers said trial by military commissions "does not extend to the trial of domestic crimes" such as conspiracy.
Also Read
Judge Karen LeCraft Henderson issued a vigorous 85-page dissent, warning that the court's decision has "alarming" consequences for the future of bringing terrorists to justice.
"My colleagues' opinion means that, in future conflicts, the government cannot use military commissions to try enemy combatants for any law-of-war offense the international community has not element-by-element condoned," Henderson said. "Their timing could not be worse."
The decision should not affect military commission cases against those accused of plotting the September 11 attack, who are charged with war crimes recognized under international law, he said.
After the September 11 attacks, al-Bahlul was arrested by local authorities in Pakistan, turned over to the US military and transferred to Guantanamo Bay.
From 2004 to 2006, two military commissions were convened to try al-Bahlul for conspiracy.
The Pentagon said Bin Laden personally tasked al-Bahlul to create a videotape glorifying the al-Qaeda bomb attack against the USS Cole Navy destroyer in a Yemeni port in 2000, an attack that killed 17 American sailors.