In an interview with National Public Radio that aired today, Hagel also acknowledged that he had taken a cautious approach to approving the transfer of some detainees out of the prison, which had reportedly irritated some officials in the White House.
Obama has repeatedly vowed to close the prison for terror suspects at the US naval base in Guantanamo Bay, Cuba. But Congress has banned the inmates from being sent to the United States and put up other obstacles, and US diplomats have struggled to find countries willing to take them in.
Asked if Obama will manage to fulfil his promise to shut down Guantanamo before leaving office, Hagel said: "It's going to be very difficult ... Especially if the Congress further restricts where these last 122 detainees go."
"This isn't a simple, easy matter of 'Oh, let's just move 122 detainees,'" he said. "These people are there for a reason. And as you draw down into the last numbers there, these are the most difficult cases."
The Pentagon chief has to formally certify each transfer that there is no imminent risk of the detainee returning to the battlefield, and Hagel said he took that responsibility seriously.
"Has there been a slowing of that (transfer process), which hasn't always made me popular in some quarters? Yes," Hagel said.