Suhail Durani, 38, has been fighting to stay in Perth since he was released from jail after serving over 18 months for sexually abusing the 19-year-old girl at Royal Perth Hospital in 2010.
Durani has maintained his innocence and sought leave to appeal on seven grounds, including that his defence lawyer had acted incompetently.
A three-judge bench of the Appeal Court had dismissed Durani's application, refusing six of the seven grounds.
A Federal Court judge today dismissed his appeal against the Appeal Court's decision, reported Radio Australia.
More From This Section
Last year, the federal government cancelled the visa of Durani, who has a wife and a young son.
His lawyer Shahid Shakur said they intend to take the case to the full bench of the Federal Court. Durani has 21 days to lodge an appeal.
"He's utterly disappointed of course. He has been in detention now for nearly six months and at the time of the judgement obviously we expected a different decision," he said.
"We respect the decision. It's obviously appealable but it will depend on the counsel's advice."
"While we respect the judgement of his Honour today, as soon as we have the counsel advice and we have read the reasoning for the decision, we will be in a decision to take further steps.
"We have not been able to read the judgement as yet."
Shakur said Durani is not yet facing deportation and will remain in detention until the appeal process is resolved.
"If he doesn't appeal then he faces deportation under the law, but if he appeals against the decision he does not obviously," he said. "As we understand, appeals can take up to six months.