Top-seeded Serena was frustrated by a swirling wind, her inability to find her A-game, and a tenacious opponent as she went down 6-4, 6-7 (3), 6-4 to 72nd-ranked American compatriot Madison Brengle.
Older sister Venus withdrew from the tournament with an arm injury ahead of her scheduled second-round match. Both players will now head to Melbourne to prepare for the first Grand Slam tournament of the season with less match play than they would have hoped.
She struggled with her ball toss in the windy conditions and made 88 unforced errors, including the double fault that ended the match and caused Brengle to clap her hand over her mouth in surprise.
Brengle, a 26-year-old Delaware native, had played Serena only once before, in 2015, losing 6-0, 6-1.
More From This Section
Williams, a 22-time Grand Slam tournament winner, played only eight events in 2016 - the most recent at the U.S. Open in August but still got off to a good start, taking a 4-1 lead before faltering.
She had four set points in Brengle's next service game but again her accuracy deserted her and she squandered every one. It finally took a fifth set point in the tiebreak for Williams to close out the set in 60 minutes.
Games went on serve in the third set, and Brengle's plan of taking the pace off the ball with her returns paid off as she broke Williams in the final game.
"It was difficult conditions with the wind and I just tried to right for every point. I tried to use the slice and get as many balls back as I could and it ended up working out.