Williams, trying to complete the first calendar Grand Slam singles sweep since Steffi Graf in 1988, defeated 110th-ranked Dutch qualifier Kiki Bertens 7-6 (7/5), 6-3 at Arthur Ashe Stadium despite 34 unforced errors and 10 double faults.
"I just kept fighting for each point, not for a lot but just one at a time," Williams said on Wednesday.
"I had been pretty relaxed. Today I was a little tight. I think it showed. Hopefully I can get back to where I was before."
Spanish eighth seed Nadal, a 14-time Grand Slam champion, ousted Argentina's Diego Schwartzman 7-6 (7/5), 6-3, 7-5, while top-ranked Novak Djokovic, on a last-eight collision course with Nadal, played later against Austrian Andreas Haider-Maurer.
More From This Section
Three-time defending champion Williams, the oldest women's Slam winner, completed her second "Serena Slam" of four major wins in a row by winning the Wimbledon crown in July.
Williams broke back to level at 5-5 in the first set but double faulted four times in the 11th game before holding, then fell behind 4-0 in the tie-break before rallying largely on Bertens' unforced errors.
But her form was so shocking that Williams ran to the practice courts to work on serves after the match.
Next up for Williams will be fellow American Bethanie Mattek-Sands, who beat compatriot CoCo Vandeweghe 6-2, 6-1.
"I'll have to play a little better if I want to win," Williams said.
- Nadal 'lucky' to advance -
============================
Nadal was down 5-4 in the first set tie-breaker before battling back against Schwartzman.
"Happy to be through," Nadal said. "Was a tough battle. I feel lucky to be through.... To take that first set was very important to me.