"The politicians are using Muslims as a punch bag," said Abdul Rashid, head of the city's main mosque.
"The politicians look for cheap and easy ways to popularity... And at the moment the cheap and easy way of gaining popularity is immigration and Islamophobia," the 73-year-old told AFP.
Birmingham is Britain's second biggest city with 1.2 million residents -- almost half of them from ethnic minorities -- and is often singled out when fears are expressed about a growing Islamist threat.
He since apologised for the comments, which hurt many inhabitants in the former industrial hub.
Also Read
The first Muslim immigrants -- Yemenis -- arrived in Birmingham in the early 20th century to work in the metal industry. There have since been waves of arrivals, many of them from the Indian subcontinent but also from Afghanistan, Bosnia and Somalia.
The city is known now for a mostly peaceful coexistence between communities, although it was hit by a nationwide spate of riots in 2011 following the police killing of a mixed-race man in London.
Ally is in constant contact with the city's dozens of ethnic communities to avoid any disputes or violence.
"Although we are said to be the second most vulnerable city to terrorism after London, we are fairly safe because the community is part of the solution," he said, hailing cooperation between the city council, Muslim communities and the police.
Philippe Trzebiatowski, a young Frenchman working on a Birmingham travel guide for the company Expedia, said that "people mix quite naturally here".