The workers had died on March 17 after a huge rock fell on them.
"If action is not taken, I will switch over to the O Panneerselvam camp. I have no option as public have put faith on me," R Kanagaraj told reporters after meeting locals who were agitating for closing down the quarry.
The locals also alleged that police and revenue officials had registered false cases that the workers had died accidentally.
He said State minister Udumalai Radhakrishnan visited him this morning and asked him "not to take any hasty decision which will affect the government."
An official release later said the Collector cancelled the licence given to the quarry owner to mine in 1.65 hectares of land.
Disclaimer: No Business Standard Journalist was involved in creation of this content