Noting that it was the fifth such incident in the national capital, he alleged that the attack "reflected failure of the government as it encouraged Hindutva".
St Alphonsa's church was vandalised by some unidentified persons in South Delhi's Vasant Kunj area, sparking outrage among the Christian community.
Singh was speaking to reporters at Paradeshar ashram of the Nath tradition, about 75 km from here, which he visited to receive an honour.
Calling himself "a god-fearing person", he said he has belief in the Nath tradition of guru Gorakhnath. But present Mahant Adityanand, who is also an MP of BJP, was "spitting venom" against Muslims, he alleged.