"I will not resign," he told reporters at Sriganganagar district when his attention was drawn to street protests by Congress and women outfits demanding his removal from the ministry.
Replying to further questions on the issue, the 43-year-old Minister of State for Chemicals and Fertilizers, said "no comments."
Nihalchand and 16 others were named in an FIR lodged by a 24-year-old married woman from Jaipur. In the FIR filed in 2011, she alleged that he had sexually exploited her.
Nihalchand and the other accused have now been asked by a Jaipur court to appear before it in connection with the case after a review petition was filed by the complainant.
Nihalchand is the only MP from Rajasthan in Modi's Cabinet.