Chanting "expulsion", supporters of the xenophobic PEGIDA movement waved banners and signs bearing slogans like "Rapefugees not welcome".
They also took aim at Chancellor Angela Merkel, accusing her of allowing migrants to run rampage in Germany through her liberal stance towards those fleeing war.
"Merkel has become a danger to our country. Merkel must go," a member of PEGIDA told the crowd, which repeated the call.
Witnesses have described the perpetrators of the assaults in Cologne on the night of December 31 as people with "North African or Arab" appearance.
Also Read
"These women who fell victim will have to live with it for a long time. I feel like my freedom has been robbed from me," a mother of four introduced as Christiane told the rally.
"That's impossible. Frau Merkel, Frau Reker, you are women! Where is your solidarity? What are women worth in this society?" she said.
Henriette Reker runs refugee services in Cologne
"There is nothing right about Nazi propaganda," said a slogan on the sign held up by one protester, while another read "Fascism is not an opinion, it is a crime".
"We are there to tell them to shut up. It is unacceptable for PEGIDA to exploit this horrible sexual violence perpetrated here on New Year's day and to spread their racist nonsense," said Emily Michels, 28.
PEGIDA started life over a year ago as a xenophobic Facebook group, initially drawing just a few hundred protesters to demonstrations in the eastern city of Dresden before gaining strength, peaking with rallies of 25,000 people.
But PEGIDA has seen a revival with the record influx of migrants to Germany.