In a veiled message to the newly-nominated BJP MP, he said the nation will not benefit from "such craving for publicity" and added "if anybody considers himself above the system then it is wrong".
In an interview to Times Now, Modi also spoke about relations with Pakistan in the wake of the terror attack on a CRPF convoy in Kashmir and said while engagement for peace was on, forces have been given "full freedom to answer back" in any manner they have to.
The Prime Minister's comments assume significance in the context of Finance Minister Arun Jaitley and the BJP distancing themselves from Swamy's recent attacks on Rajan, CEA Arvind Subramananian and Economic Affairs Secretary Shaktikanta Das.
Swamy also had made some carping comments about Jaitley without taking his name.
Also Read
Even after that there have been criticisms made against very senior government officials, the questioner said recalling Modi's recent counsel to party leaders in Allahabad to maintain balance and restraint in their speech and conduct.
"My message is very clear. I have no confusion about it," Modi said when he was asked whether his message on the issue is clear.
Modi then went on to praise Rajan, who has said no to a second term in RBI, saying his patriotism was no less and was sure that he would continue to serve India irrespective of whether he was in some position or not.