Joey Gibson, leader of the group Patriot Prayer, said yesterday that members had been in contact with police and decided tomorrow's event "seems like a set up."
"It doesn't look safe, a lot of lives are going to be in danger," he said in a Facebook video, adding "In our opinion, it seems it would have been a huge riot."
He blamed local politicians and the media for drumming up fears by "saying that we are white supremacists."
However, the city's mayor Ed Lee said on Twitter that no permits had been requested or granted for that location.
More From This Section
"Public safety is always our top priority. We are prepared for contingencies and spontaneous events," Lee said.
Another right-wing rally set for Sunday -- a "No to Marxism in America" rally in neighbouring Berkeley -- was up in the air, with organiser Amber Cummings asking people not to come.
Police are nevertheless on watch in both locales, with counter-protests still possible.
That rally saw a woman killed after an avowed white supremacist rammed his car into a group of anti-racism counter-protesters.