Goals from AC Milan's Carlos Bacca and Real Madrid star James Rodriguez secured a 2-1 win for Colombia over Paraguay in front of a vocal crowd at the Pasadena Rose Bowl in California.
Victor Ayala gave Paraguay hope after pulling a goal back on 71 minutes but Colombia held on for three points which means they are assured of a top two finish in Group A and a place in the last eight.
In Tuesday's other game, US striker Clint Dempsey scored his 50th international goal as the host nation bounced back from their 2-0 loss to Colombia on Friday with a convincing rout of Costa Rica.
Dempsey opened the scoring from the penalty spot after nine minutes before Jermaine Jones and Bobby Wood struck just before half-time to effectively settle the contest.
More From This Section
Substitute Graham Zusi added a fourth goal three minutes from time, which could prove crucial if goal difference comes into play.
Klinsmann meanwhile showed he was in no mood to cede
ground to his critics, picking the same starting line-up to face Costa Rica that had disappointed in the 2-0 defeat to Colombia. Klinsmann's faith was rewarded with a superb first half display from his team.
"I think it was, overall, a very pleasant performance," Klinsmann said.
"It confirms the spirit of the group. They are all there for each other, the whole bench is jumping up and down when we score. Everybody wants the others to do well. It's a good group of guys that we have together.
Klinsmann's team now faces Paraguay in its final Group A game in Philadelphia on Saturday, where a draw will probably be enough to advance to the knockout rounds.
"We wanted to put ourselves in the driving seat against Paraguay and that's what we did," Klinsmann said.
"We are right there to get through a difficult group, and this is the most difficult group in the Copa America. It's encouraging to see this group of players has the extreme hunger to make things happen."
In Wednesday's other game, five-time world champions Brazil face an intriguing matchup against Caribbean minnows Haiti in Orlando, Florida.
"If we win it will be the greatest miracle that God has produced in football," Haiti midfielder Jean-Marc Alexandre said Tuesday.