The trespassers were thrashed by the party workers and handed over to police, party sources said.
Their identity has not been ascertained yet.
The men shouted slogans of 'Hindu Sena zindabad' and 'CPI(M) murdabad'.
Incidentally, party general secretary Sitaram Yechury was about to enter the party's central committee meeting hall on the first floor of the office to brief the media.
He was to brief the press on the conclusion of the CPI(M) politburo's two-day meeting here.
Vijayan has been facing flak from some right-wing organisations over his criticism of the Centre's recent notification relating to cattle slaughter.