The lender said in a statement that the 4.2-billion pounds penalty was a "heavy price" to settle US mis-selling claims, which occurred in the run-up to the notorious global financial crisis and subsequent worldwide recession.
"The Royal Bank of Scotland... Has reached a settlement with the Federal Housing Finance Agency," the lender said in a statement today, adding that FHFA "litigation against RBS will be withdrawn".
The net cost will be USD 4.75 billion due to special indemnity agreements.
The agreement settles an FHFA lawsuit alleging that RBS sold faulty mortgage bonds to Fannie Mae and Freddie Mac between 2005 and 2007.
More From This Section
RBS will pay approximately USD 4.525 billion to Freddie Mac and approximately USD 975 million to Fannie Mae, the FHFA added.
The announcement resolves one of two major US probes into mis-selling allegations; RBS has yet to reach a deal with the Department of Justice.
"This settlement is a stark reminder of what happened to this bank before the financial crisis, and the heavy price paid for its pursuit of global ambitions," RBS chief executive Ross McEwan said in the statement.
Edinburgh-based RBS remains 70-percent state-owned after receiving a vast government rescue at the height of the 2008 crisis in the world's biggest banking bailout.
The FHFA fine is largely covered by money already set aside by the bank, which has long been plagued by legacy costs arising from its past conduct.
McEwan cautioned the bank may need to set aside more money to settle outstanding claims.
RBS chief finance officer Ewen Stevenson added the FHFA settlement was "in the region of what we'd been anticipating", but analysts indicated that it was higher than forecast.
Equities analyst Joseph Dickerson, at US bank Jefferies, said the net USD 4.75-billion figure was USD 1.0 billion higher than market expectations.
"This settlement clears a major hurdle for the bank, though there remain further significant related costs, such as the DOJ," he said.
Dickerson forecast that RBS would need to set aside another USD 2.5 billion to cover the DoJ deal in the fourth quarter.
The system-wide failure of complex securities derived from residential mortgages caused a cascading wave of bankruptcies and crises that sparked a global recession, leading to tens of millions of job losses around the world.
"Ten years on from the financial crisis, RBS and the UK taxpayer are still counting the cost of the bank's former misdemeanours," said Laith Khalaf, senior analyst at stockbroker Hargreaves Lansdown.
"The coming year is not going to be pretty for the bank, as it works through the costs of outstanding US litigation for mortgage-backed securities sold in the run up to the credit crunch.
"The elephant in the room is the US Department of Justice fine, which is likely to be sizeable, and is subject to a high degree of uncertainty."