Attrey was arrested last week for allegedly raping a woman for about a year.
"Delhi BJP president Vijender Gupta has expelled Yogesh Attrey, who is facing serious charges, from BJP with immediate effect," a party statement said.
Gupta said BJP will not tolerate any such acts by its members.
"BJP will adopt zero tolerance against such people. They do not have any place in the party. The party will adopt stern attitude against them so that no body could charge BJP with protecting any criminal element," he said.
Attray, who lost the elections to Delhi Minister Raj Kumar Chauhan, was arrested after a woman filed a complaint on Thursday.
Police said the case was registered at Vijay Vihar Police Station in Outer Delhi and that Attray was booked under sections 376 (rape), 506 (criminal intimidation), 509 (intending to insult the modesty of any woman) and 313 (causing miscarriage without woman's consent).
More From This Section
They also said a statement of the victim, whose identity has not been revealed, has been recorded.
The incident comes in the wake of national outrage against the gang-rape of a 23-year-old paramedical student.