In remarks to the American Bar Association, Attorney General Eric Holder called minimum jail terms "counterproductive" while noting the need to stay strict but be smarter about tackling crime.
And he warned that, while the total US population has increased by about a third since 1980, the prison population has soared by 800 percent.
The United States accounts for five percent of the world population but nearly a quarter of all people imprisoned, he said.
And of the more than 219,000 people jailed in federal as opposed to state-run prisons, nearly half were convicted of drug-related offences.
Also Read
Altogether, inmates in local, state and federal prisons cost the government USD 80 billion dollars in 2010 alone, he added, saying it was time for reform.
"We will start by fundamentally rethinking the notion of mandatory minimum sentences for drug-related crimes. Some statutes that mandate inflexible sentences -- regardless of the individual conduct at issue in a particular case -- reduce the discretion available to prosecutors, judges, and juries," Holder said.
"And, applied inappropriately, they are ultimately counterproductive."
The mandatory minimum sentences were included in the penal code by Congress in 1986 and 1988.
Holder said he hoped Congress would pass new legislation but in the meantime he has mandated a modification of the Justice Department's charging policies.
Holder said that, under the changes, certain low-level, non-violent drug offenders who have no ties to large-scale organisations, gangs or cartels will no longer be charged with offences that impose draconian mandatory minimum sentences.