Jamna Joshi and Moksha Jai Joshi, known as Jai, died in the house fire in the city of Liverpool on May 8 and police has confirmed that an accelerant had been used and there was "no evidence of any third-party involvement".
"It was a case of if she can't have him, I can't," Paresh Patel, Jai's father, told the BBC.
Patel had won custody of Jai and agreed that his mother could have him during holidays.
A serious case review is underway into social services' role in the case.
More From This Section
Patel claimed the 40-year-old pharmacy assistant had "some psychological issues" and knew she had no option but to return Jai under the terms of a court order.
He claimed her last thought would have been that "she had won", because he had lost his son.
"The crux is no one can tell you of the [details] or enforceability of a court order. After battling for four-and-a-half years to keep him safe...I feel it has all been taken away from me. You never expect your son to die before you," Patel said.
Lancashire Police said the full circumstances of what happened are being investigated and will be considered at both the inquest on July 7 at Liverpool Coroner's Court and Lancashire County Council's Safeguarding Children Board in June.
Lancashire County Council said it could not comment if any risks to the little boy were known.