His remarks came a day after the Deputy CM's office announced that the venue of the December 3 wedding had been changed to Veterinary College ground (near Patna airport) from Shakha Maidan in Rajendra Nagar due to "security reasons".
"I am not a criminal or terrorist. Whatever I said at a public meeting should not have been taken literally by Modi. He should go ahead with the wedding without any worry or fear," Yadav said.
More From This Section
However, he said even if there was not "any untoward incident involving me or my party, their (Sushil Modi/BJP) own men could create a ruckus to defame me".
A video had surfaced last week purportedly showing Yadav threatening to disrupt the wedding.
In the video aired by news channels, Yadav was seen delivering a speech in Aurangabad and saying: "I have been invited by Sushil Modi to attend the wedding of his son. If I go there I will expose him in public."
The former Bihar health minister was also heard threatening to indulge in violence at the wedding venue.
Modi has levelled a number of allegations against Prasad's family acquiring benami properties.
The RJD has trashed all the allegations and accused the senior BJP leader of suffering from "Lalu phobia".