In sensational scenes after the final whistle, Neymar was shown a red card after an ugly melee involving players of both sides in which he appeared to aim a headbutt at Colombia goalscorer Jeison Murillo.
Colombia striker Carlos Bacca was also shown red following the fracas at the end of a frenetic Group C battle.
Neymar had already ensured he would miss Brazil's final first round game against Venezuela on Sunday by collecting his second yellow card of the tournament for a deliberate handball in the first half.
Neymar's Brazil and Barcelona team-mate Dani Alves lashed out at Chilean referee Enrique Osses following the defeat.
More From This Section
"Referees have to stop thinking that they are the stars, the stars are not them -- they are there to control the game," Alves fumed.
"We are used to this in South America -- everyone here is against Brazil."
Alves also accused Colombia's players of seeking to provoke Neymar, 12 months after the striker suffered a broken vertebra during the two teams' stormy World Cup last eight battle.
"They tried to provoke him and make him nervous."
Victory was sweet for Colombia, who were subjected to rough-house tactics from Brazil last year during the World Cup in a 2-1 quarter-final exit, a match scarred by a whopping 54 fouls.
"We had a great gameplan and it worked today," said Colombia's Real Madrid star James Rodriguez.
"Four days ago we lost to Venezuela when we played well, tonight we played well and beat Brazil, sometimes it happens like that."
A pulsating opening 45 minutes saw Colombia gradually get on top after a frantic start dominated by defence.