Shares of Target were down 8.3% at $67.50 after its quarterly sales missed expectations.
Dow-component Wal-Mart Stores, which reports results on Thursday, was also down 3% at $63.10. Costco Wholesale slipped 2.8% to $139.61.
The Fed minutes, scheduled to be released at 2 p.m. ET (1800 GMT), could give clues on the path of rate hikes. Dallas Fed President Robert Kaplan said on Tuesday he will advocate for a hike in June or July. Two other Fed officials said they expect up to three increases this year.
The central bank next meets on June 14-15 but chances of a hike in June are slim. Traders see a 58% probability of a rate hike after the Fed's November meeting, up from about 42% on Monday, according to the CME FedWatch tool.
"The market got jittery yesterday after comments from the Fed officials but what it's really looking for is comments from Yellen," said Randy Frederick, managing director of trading and derivatives for Charles Schwab in Austin, Texas.
"I think we could see two rate hikes but September and December are more likely rather than June."
More From This Section
April data on Tuesday showed the biggest rise in U.S. consumer prices in more than three years as gasoline prices and rents rose, while other data showed housing starts and industrial production rebounded strongly.
The strong data and a recovery in oil prices to seven-month highs have stoked expectations that inflation will rise further. The Fed has a 2% inflation target.
At 9:38 a.m. ET (1338 GMT), the Dow Jones industrial average was down 77.46 points, or 0.44%, at 17,452.52, the S&P 500 was down 6.15 points, or 0.3%, at 2,041.06 and the Nasdaq Composite was down 4.13 points, or 0.09%, at 4,711.61.
Nine of the 10 major S&P sectors were lower, with the consumer staples index's 1.02% fall leading the decliners.
Goldman Sachs on Tuesday downgraded equities to "neutral" over a 12-month time-frame, saying, "until we see sustained signals of growth recovery, we do not feel comfortable taking equity risk."
Staples fell 1% to $8.20 after the company said it does not expect top-line growth this year, while Lowe's was up 2.9% to $78.15 after the home improvement chain's quarterly sales exceeded expectations.
Tesla Motors was up 2.4% at $209.80 after Goldman Sachs upgraded the electric car maker's stock to "buy" from "neutral."
Declining issues outnumbered advancing ones on the NYSE by 1,727 to 872. On the Nasdaq, 1,309 issues fell and 883 advanced.
The S&P 500 index showed three new 52-week highs and five new lows, while the Nasdaq recorded four new highs and 23 new lows.