The 16-story building, long criticised for openly flouting Dhaka's strict construction laws, must be destroyed within six months at a cost borne by its occupants, the Supreme Court ruled.
"If they fail to carry out the order, the (government's) capital development authority will demolish the building," Attorney General Mahbubey Alam told AFP.
The Bangladesh Garment Manufacturers and Exporters Association (BGMEA), which occupied the building, said they would vacate as "soon as possible".
A court first challenged its legality seven years ago after it was revealed the building was constructed illegally on a state-owned floodplain.
More From This Section
Lawyers declared the hard-fought verdict a "landmark" in Bangladesh's judicial history.
"It's challenging the culture of impunity that prevails in our society," Rizwana Hassan from Bangladesh Environmental Lawyers' Association told AFP.
Prime Minister Sheikh Hasina laid the foundation stone for the controversial BGMEA building in 1998, while a former premier now opposition leader formally opened it for business in 2006, underscoring the industry's ties to politics.
The garment industry employs nearly five million Bangladeshis, making it the single largest job creator in the impoverished nation.
A top union leader described the building as a "symbol of conspiracy and corruption".
"They (BGMEA) thought they were above the law. They wanted to flex their muscles with no respect for the law," union leader Babul Akhter told AFP.