Bobby Wood, who plays for Germany's third division side Erzgebirge Aue, scored the winner just before the final whistle to seal the Americans' deserved victory.
It capped a dream few days for the 22-year-old who also hit the winner when the USA fought back to stun the Netherlands 4-3 in Amsterdam last Friday.
Again the USA fought back from a goal down to cap a strong display as Germany suffered their third defeat since winning the World Cup last year.
"My players are very happy and the way things developed in the second half, the victory was well deserved.
Also Read
"This success will give American football even more attention."
The defeat was not what Germany needed before their Euro 2016 qualifier on Saturday when they face bottom side Gibraltar away in Faro, Portugal, as they look to claim top spot in their group.
But after their Dutch win, it was a confidence booster for the USA as they prepare to host the CONCACAF Gold Cup next month.
"If we had taken our chances at the start of the game, the USA would certainly not have come back from two or three-nil down."
Mario Goetze's first-half goal for Germany in Cologne was cancelled out by Mikkel Diskerud's equaliser just before the half-time break before Wood struck the decisive blow late on.
"That was inconvenient, we must play better football on Saturday, then we won't get into trouble again," said Goetze.
It was Klinsmann's second win over his former side Germany, coached by his former assistant Loew, after America's friendly win in Washington in 2013.