The sports ministry, reached shortly after Wagner spoke in Rio de Janeiro, said it could not confirm the resignation, illustrating the confusion engulfing Brazil's federal government as Rousseff tries to fight off impeachment charges.
Reports surfaced last week that Hilton had stepped aside, only for him to remain in office.
Hilton recently left a political party that was part of Rousseff's coalition government but is no longer - and changed parties.
Wagner said Ricardo Leyser, who has worked in the ministry, was likely to take over as sports minister.
Hilton has had a marginal role in organizing the Rio de Janeiro Olympics, which open on Aug. 5, and have been swamped with problems including severe budget cuts, water pollution and the mosquito-borne virus Zika.