The Saudi hajj ministry said that the delegation had "asked to go back home without signing the agreement on arrangements for the pilgrims" despite two days of extensive talks.
A statement said the ministry had offered "many solutions" to meet a string of demands made by the Iranians who had arrived on Tuesday.
Agreement had been reached in some areas, including to use electronic visas which could be printed out by Iranian pilgrims, as Saudi diplomatic missions remain shut in Iran, it said.
Earlier this month Iran had accused its regional rival of seeking to "sabotage" the hajj, saying "arrangements have not been put together" for its pilgrims.
More From This Section
Tehran said Riyadh had insisted that visas for Iranians be issued in a third country and would not allow pilgrims to be flown in aboard Iranian aircraft, which the Islamic republic rejected.
Today the Saudi hajj ministry said Riyadh had agreed to allow Iranians to obtain visas through the Swiss embassy in Tehran, which has looked after Saudi interests since ties were severed in January.
Talks on an agreement were the second attempt by both countries to reach a deal on organising this year's pilgrimage for Iranians after they held an unsuccessful first round last month in Saudi Arabia.
The Saudi ministry said the Iranian Hajj Organisation would be held responsible "in front of God and the people for the inability of its pilgrims to perform hajj this year."
The annual hajj is one of the five pillars of Islam which devout Muslims must perform at least once during the lifetime.