"The counting began this morning and the BJP candidate is leading by 4,704 votes after two rounds," the state election commission said.
The Congress candidate Banwari Lal Sharma was trailing in the initial trend.
The result of the bypoll will be announced today after counting of votes.
The by-election was necessitated following disqualification of BSP MLA B L Kushwaha in a murder case and the BJP surprised observers by pitting his wife Shobha Rani as its candidate against Congress veteran Banwari Lal Sharma.