They initially faced tough resistance from forces guarding the premier's office but some were able to muscle past and briefly enter the premises, an AFP photographer at the scene said.
Sadr followers have been protesting for weeks to demand reforms, a new government and an end to corruption.
Several people were wounded in the process and many others were also hurt outside the walls of the Green Zone, where security forces fired tear gas directly into the crowd to keep the bulk of the protesters at bay.
"We came in a peaceful protest but the cowards began shooting at us," said one protester, holding a handful of bullet casings.
More From This Section
The protesters gathered on Tahrir Square in central Baghdad but they removed barbed wire on one of the main bridges over the Tigris and converged the Green Zone.
After facing much tougher resistance than three weeks ago when they broke into the restricted area for the first time, some of the protesters eventually forced one of the gates.
"Don't be with the oppressor, be with the nation," they chanted as they confronted security forces guarding the entrance to Abadi's compound.
Some protesters eventually forced their way in, an AFP photographer reported.
Sadr supporters were posting pictures of the prime minister's office on social media, including from the cabinet meeting room.
The protesters eventually pulled out and security forces pinned the crowd back near one of the main bridges in central Baghdad as reinforcements were brought in.
It was not clear where Abadi himself was at the time of the fresh security breach.
Abadi had subsequently sacked the security chief for the Green Zone and beefed up measures around the restricted area.
The breach of the Green Zone -- which is also home to several major embassies, including that the United States -- further deepened the country's political crisis.