Tejpal was invited to talk about "The Tyranny of Power".
The withdrawal of the invitation follows a flurry of tweets against Tejpal and the event organisers.
The Litfest Directors Bachi Karkaria and Namita Devidayal said in a statement "we have regretfully withdrawn our invitation to Tarun Tejpal to participate in the Litfest, Dec 5-7, 2014.
"While there was never a question of a sub-judice case being discussed at any of its forums, the reaction to his inclusion suggests that our Litfest was in danger of being overwhelmed by an extraneous issue", they said.
Also Read
"All these years, we have managed to be about discussion not noise and we'd like to keep it that way for the sake of our acclaimed participants and engaged audiences," they said.
Tejpal, who was arrested November last year, has been charged with sexual harassment and rape of a colleague in Tehelka magazine during an event in Goa. He has denied the allegations against him.
After spending some months in jail, he was granted bail in July this year.