The incident happened during the 11th stage of the Tour de France, a 33-kilometre individual time-trial between Avranches and Mont-Saint-Michel in Normandy.
It came a day after he was blamed for a collision that sent Argos rider Tom Veelers crashing to the ground in Saint-Malo.
Cavendish did not talk to the media after the race, but Froome, speaking after holding on to the yellow jersey, said what happened "ruined" what has otherwise been a wonderful atmosphere throughout the 100th staging of the sport's greatest race.
"But to do something disrespectful like that, that's really sad. It ruins the whole atmosphere."
Earlier in the day, Omega Pharma team CEO Patrick Lefevere confirmed to AFP that Cavendish had been insulted by spectators and attacked by one individual on the race route.