"Only if my party asks, will I resign", he told reporters at nearby Kunchithanni, a day after opposition parties and women organisations demanded his resignation and apology.
Mani's remarks allegedly questioning the character of women activists of a plantation workers' outfit in the district invited strong condemnation from various quarters, including from some of his own party colleagues.
Chief Minister Pinarayi Vijayan had said, "If any offensive statement was made against them (the women), it was improper."
He had also claimed his remarks were wrongly interpreted by the media and that he had not made any offensive remarks against the women workers.
Also Read
Mani also said that Vijayan and party state Secretary Kodiyeri Balakrishnan had called him over phone and sought his explanation.
However Mani, known for making controversial remarks, refused to apologise, saying, "Let them sit there and agitate."
He said the chapter wasclosed and he had nothing more to add while and would not visit the agitating workers.
Meanwhile, vehicles kept off roads and shops remained closed in the hilly district in view of a dawn-to-dusk hartal called by the NDA.
The opposition Congress led UDF is also lending support to the women's agitation by observing a 'black day'.
Mani had also accused an IAS officer of plotting with the RSS to pull down a cross on an encroached piece of land in Munnar.
The CPI(M) leader had stirred up a controversy few years ago by openly stating that the party had eliminated political rivals in his home district, Idukki.
Disclaimer: No Business Standard Journalist was involved in creation of this content