"The Ouagadougou centre is in our hands," the operations centre for forces loyal to Libya's Government of National Accord (GNA) said.
Reda Issa, a spokesman for the forces, said IS jihadists remained in three residential areas of the city and in a villa complex near the seafront.
"The announcement of the liberation (of Sirte) will only be made once the entire city is liberated," he told AFP.
Capturing the Ouagadougou centre has been the key goal of pro-GNA forces as they have battled for weeks to oust IS from Sirte, which the jihadists seized in June last year amid the chaos that followed the 2011 ouster of Moamer Kadhafi.
Loyalist forces stepped up their fight for the city in recent days, and the Pentagon announced last week it had begun carrying out air strikes on IS positions in Sirte at the GNA's request.