Arsenal would have provisionally supplanted leaders Chelsea on goals scored with victory and took a 20th-minute lead when Alexis Sanchez's free-kick deflected in off Williams.
But Seamus Coleman equalised with a 44th-minute header before Williams's first Everton goal ended his team's five-game winless run and sent Ronald Koeman's men up to seventh in the table.
"We showed two faces," Everton manager Koeman, who had defender Phil Jagielka sent off in stoppage time, told BT Sport.
"This was a good match though because we showed how we have to play -- with commitment and aggression. You see the reaction of the crowd when you go for it and that's what we did."
Also Read
Arsenal manager Wenger said the game had been "very physical".
"We faced many physical challenges and that disturbed our game, but we were a bit unlucky to lose as well," he said.
"They were a bit scared at first, but they got their confidence. We did fight, I cannot fault that."
Arsenal made only one change from their 3-1 win over Stoke City, fit-again right-back Hector Bellerin replacing injured centre-back Shkodran Mustafi, and took a 20th-minute lead.
The goal stemmed from a comedy of errors on the edge of Everton's box as Ross Barkley lost possession, Williams fouled his own team-mate Idrissa Gueye and Jagielka tripped Francis Coquelin.
Sanchez sent the free-kick curling towards Everton goalkeeper Maarten Stekelenburg and in seeking to clear the ball, Williams succeeded only in deflecting it into his own net.
Everton did not look like equalising, with Romelu Lukaku and Aaron Lennon slicing shots off target, but with half-time looming they drew level.
Leighton Baines curled a right-foot cross towards the back post from the left and Coleman nipped in between Laurent Koscielny and Nacho Monreal to glance a deft header past Petr Cech.
The noise level rose as the game wore on and with four
minutes to play Everton struck the winner, Williams flying in at the back post to send a thudding header past Cech from Barkley's corner.
It was Arsenal's first league defeat since their 4-3 loss to Liverpool on the season's opening day and means they will be six points off the pace if Chelsea win at bottom club Sunderland on Wednesday.
Leicester remain without back-to-back league wins this season after defeat at Bournemouth -- their eighth of the campaign -- kept them in 14th place, just four points above the relegation zone.
"The result is too severe against us because we created more chances than Bournemouth," said Leicester manager Claudio Ranieri. "If we continue to fight at this level, sooner or later we will get a good result."
Eddie Howe's Bournemouth leap four places to eighth, nine points clear of the bottom three.