"People who didn't contribute anything in the Independence struggle are now not even sparing the image of Gandhi," the CPI-M leader said in a Facebook post.
He said replacing Gandhi's image with that of Modi is an act of "sheer pettiness".
"A Prime Minister should not have stooped to such a level," Vijayan said, adding, "the image of Gandhi spinning wheel is one that remains in the heart of the people".
Vijayan said all secular-minded people in the country should unite to face the "challenges posed by RSS".
More From This Section
A controversy had erupted last week when KVIC published their calendar and diary with a photo of Modi weaving khadi on a charkha (spinning wheel).
Vijayan also came down on BJP leaders in the state for their criticisms of Malayalam film director Kamal and said no one has the right to demand another person leave the country.
Sangh Parivar affiliated outfits had slammed the director over his alleged remarks in connection with the playing of national anthem during the screening of films at the IFFK last month.
Vijayan also criticised some BJP leaders' statements against noted writer M T Vasudevan Nair for the latter's opposition to demonetisation.