Nisar was referring to the criticism he faced from Pakistan Peoples Party (PPP) for his meeting with Maulana Ahmad Ludhianvi, chief of the proscribed anti-Shiite Ahle Sunnat Wal Jamaat (ASWJ).
The party had also accused him of maintaining friendly ties with Ludhianvi.
"How is it fair to link everything to Maulana Ludhianvi?" Khan told reporters in Rawalpindi in response to a question.
"Which PPP leader did not meet leaders of proscribed organisations in their time?" he asked.
More From This Section
The minister created another controversy last week when he told Senate that outlawed sectarian organisations should not be equated with terrorist outfits.
Responding to criticism, Khan said that he was suggesting that there should be a separate law to deal with groups banned on sectarian basis from the militants groups.
He said there was nothing wrong in saying that the Shia-Sunni conflict dated back 1300 years and is a part of the Islamic history.
He said that efforts were being made to recover the five civil society activists who went missing in the past week.
"We are making efforts to recover all missing men so they could return to their families," Khan said.
The minister also said that a meeting was held to discuss the future of special military courts to try 'hard-core' terrorists but no decision has been taken so far.