The agreement took effect in January following its signing last year after years of international effort.
It calls on Tehran to curb its controversial nuclear programme in exchange for sanctions relief from the United States and other nations.
Iran has denied that it seeks to acquire a nuclear weapons capability.
"There are some significant downsides that should cause us great concern," Petraeus told the Manama Dialogue security forum in Bahrain of the accord.
More From This Section
Petraeus is on the shortlist to be secretary of state under Donald Trump, who will assume the US presidency at the end of Barack Obama's term in January.
Trump has promised to tear up the Iran nuclear agreement once in office, calling it the "worst deal ever negotiated".
Despite Petraeus's reservations, the former general said "there are actually some positive elements", as it helped curb Iran's path to a nuclear weapon.
"There's a pretty intrusive set of verification measures," he said.
He headed the Bahrain-based US Central Command and then the NATO force in Afghanistan in 2010, before retiring to lead the CIA.
He resigned from the spy agency in late 2012 after coming under investigation for giving his biographer and mistress, Paula Broadwell, access to classified information.
Petraeus eventually pleaded guilty to a misdemeanour charge of mishandling classified information.
Saudi Arabia and its Sunni Gulf allies fear the Iran nuclear pact will lead to more regional "interference" by predominantly Shiite Iran, which backs opposite sides in the Syria and Yemen conflicts.