Naidoo has sold millions of albums in Germany but songs such as 2012's "Wo Sind" (Where Are) have been widely criticised.
Anti-racism groups complained after his selection, on Thursday, for the Stockholm contest, BBC reported yesterday.
Public broadcaster ARD denied the "brilliant" singer was racist.
Executive Thomas Schreiber said: "It was clear that his nomination would polarise opinions, but we were surprised about the negative response."
The discussion on Naidoo could harm the image of the contest, he added.
Also Read
"This is why Naidoo will not represent Germany. We will quickly decide now, how the German entry for the 2016 Eurovision Song Contest will be found," he said.
In 2014, when Germany marked the 25th anniversary of reunification, Naidoo was criticised for appearing at a rally of the controversial Reichsbuerger group, which wants the re-establishment of Germany as a two-border state.
Anti-racism group the Amadeu Antonio Foundation also described the choice as "problematic".
In response, 44-year-old Naidoo said on Facebook in his native language that it was "OK for me" and that ARD had urged him to compete in the first place.
He also said he represented a Germany that was "open to the world" and tolerant of different religions and lifestyles.
Germany, which came last in this year's competition with zero points, would name a new contender as soon as possible, Schreiber said.