"We will no longer keep quiet" the former ministers, all women, including current IMF chief Christine Lagarde, ex- health minister Roselyne Bachelot and ex housing minister Cecile Duflot said in a statement published by the Journal du Dimanche weekly paper.
They promised to "systematically denounce all sexist remarks, inappropriate gestures, inappropriate behaviour."
Their cross-party stand comes after French judges on Tuesday opened an investigation into multiple claims of sexual harassment against former deputy parliamentary speaker Denis Baupin, a move which experts welcomed as an end to the "omerta" around alleged abuse by politicians.
The same day Finance Minister Michel Sapin became ensnared in a sexual harassment scandal after admitting that he acted "inappropriately" towards a female journalist after twice denying any improper conduct.
More From This Section
The ex-ministers in their joint comment today said: "Like all women who have entered into previously exclusively male environments, we have had to either submit to or fight against sexism.
"It's not for women to adapt to these environments. It's the behaviour of certain men that need to change," they added.
The former ministers encouraged all victims of sexual harassment and aggression to speak up and lodge complaints.
"Today the judicial arsenal exists but the laws are not sufficiently applied".
Labour laws protect employees but are not respected, with few women lodging complaints and very few of those complaints leading to convictions, they complained.
Women's rights minister Laurence Rossignol, speaking on the France 3 television channel later, outlined two moves to tackle the problem.