"It is a matter of national security, political colours should not be given," he told reporters here.
Rijiju was responding to a question on criticism faced by the government from several political parties over the decision to allow the Pakistani team to visit India in connection with their probe into Pathankot terror attack.
Congress said the Modi government has set a "wrong precedent" and wondered why 'biryani' was being served to the Pakistani Joint Investigation Team (JIT), which also includes an ISI representative.
Shiv Sena leader Sanjay Raut while terming the government's decision as "wrong" said rather an Indian team needs to be sent to Pakistan to probe those behind the terror strikes.
Raut also reminded the Centre that not India, but Pakistan is an accused in the Pathankot strike.
Waving black flags and placards, scores of Congress and Aam Aadmi Party (AAP) workers stood outside the IAF base in Pathankot yesterday protesting against the visit and shouted anti-Pakistan slogans.