A key aide of former UKIP leader Nigel Farage, 30-year-old Kassam said he was not "satisfied about the integrity of the process".
"It is a decision I have not taken lightly, but following meetings this weekend I realised the path to victory is too narrow," he said in a statement.
Kassam, son of Tanzanian immigrant parents of Gujarati origin,urged his supporters to back Peter Whittle for party leader.
Kassam said those at the top of UKIP were treating the contest "like a coronation", adding that he was "not satisfied about the integrity of the process".
More From This Section
He accused some Members of the European Parliament of using party databases to "effectively campaign against me - ostensibly against the rules".
Kassam said he had not raised enough money to run "more than a digital campaign from SW1" and added that not to "feature at events all around the country would have made me a hypocrite, given how much I criticise the establishment for ignoring the country".
Kassam launched his official campaign last Friday, portraying himself as the "Farage-ist" candidate and pledging to increase UKIP's membership to more than 100,000.
After pulling out of the race, he said he would continue as editor-in-chief of the Breitbart London news website and would be travelling to the US to cover the conclusion of the presidential election campaign.
Raised in the Ismaili sect of Shia Islam, Kassam has often described himself as non-practising Muslim opposed to all sort of religious fundamentalism, and has questioned foreign funding of religious institutions.
Another candidate, London Assembly member David Kurten, withdrew yesterday. Nominations for the UKIP contest closed on Monday, with the result to be announced on November 28.
Farage, for whom Kassam worked as chief of staff, returned to the leader's role on an interim basis after his successor Diane James resigned only 18 days into the job.