The couple had argued that they were waiting for an official Hindu naming ceremony as per their religious beliefs but Justice Parker said the decision not to give the boy a name could be "emotionally harmful".
On Friday, the Hertfordshire-based couple were told they could challenge the decision in the Court of Appeal.
Justice Ryder ruled that the baby and his two-year-old brother should not be placed for adoption until the couple had aired their grievances to three appeal judges.
Justice Parker, who sits in the Family Division of the High Court, had highlighted concerns about the father's hostility to social workers.
Also Read
She also said she was concerned about the vulnerability of the mother, who had been diagnosed with a learning difficulty and said she was troubled by the couple's decision not to name the baby.
The father appeared to have become frustrated by what he saw as an "invasive" approach by social services staff, and the judge said she thought he could be "dangerous".
The couple's two-year-old son was taken into care after another judge raised similar fears and described the father's behaviour towards social services staff as "dangerous".
The judge said the mother also had an older third child - by another man - who had gone into foster care as a toddler and lived with a relative.
"I don't think I have learning disability and the judge hasn't given me a chance to prove that.
"And my partner only gets angry with social workers - no one else. The judge hasn't taken account of that. Anyone would be angry. We love our children and we're only human. And he has apologised.