The protesters raised slogans against Swamy, police said.
As the party office was locked, the students could not enter the office building. But they damaged the party flag post and notice board in the front portion, police said.
Deputy Commissioner of Police Tirunavukarasu visited the Janata Party office. Twenty armed policemen had been posted.
The meeting between Swamy and Rajapakse caused a flutter in Tamil Nadu as it came amid demands for India to support the US-sponsored resolution against Sri Lanka at UNHCR over alleged war crimes.