Home Secretary Theresa May wants to wait for the results of a separate inquest into the 2006 death of Litvinenko, a former agent in Russia's FSB agency, who was poisoned with radioactive polonium-210 while drinking tea at a London hotel.
But three judges at England's High Court ruled that May must reconsider her decision after a challenge from his widow, Marina Litvinenko.
Speaking on behalf of the three judges, Lord Justice Richards said that "deficiencies" in May's 'wait-and-see' approach were "so substantial that the decision cannot stand."
However he stressed that his judgement that a public inquiry was necessary "does not of itself mandate any particular outcome."
Also Read
Speaking outside court, Marina Litvinenko, who believes the Kremlin was involved in her husband's death, said she was "very glad" the judges had ruled in her favour.
She called on May to "accept this decision".
British police have asked for the arrest of two Russian nationals in relation to the death, Andrei Lugovoi and Dmitri Kovtun, but Moscow has refused to hand them over.
A Home Office spokesman said: "We are carefully considering the judgement. The government continues to fully co-operate with the coroner's inquest into Mr Litvinenko's death.