But the two sides said they were within striking distance of consensus, setting up a hectic negotiating rush ahead of next week's EU summit which must decide whether to broaden the talks to the topic of future relations.
On a day that see-sawed between hope and disappointment, the leaders fell short of reaching what the EU considers "sufficient progress" on the divorce issues: Britain's exit bill, and rights of citizens affected by Brexit, and the status of the currently invisible Irish border, the main sticking point
EU leaders want a deal on divorce terms in time for them to agree at a summit on Dec. 14-15 whether to move the negotiations on to the next stage of talks, including trade.
The lack of progress so far has raised concerns that Britain may not have a deal on key issues by the time it officially leaves on March 29, 2019.
More From This Section
"Despite our best efforts and the significant process we and our teams have made over the past days on the remaining withdrawal issues, it was not possible to reach an agreement," said EU Commission President Jean-Claude Juncker.
But she said talks would reconvene later this week "and I am also confident we will conclude this positively."
The EU and the UK are nearing agreement on some divorce terms, including the size of the bill that Britain must pay as it leaves and the rights of citizens affected by Brexit. But the border issue has proved more intractable.
After Britain leaves the bloc, the currently invisible 500-kilometer frontier will be the UK's only land border with an EU country.
Negotiators were discussing an agreement that would commit Britain to maintaining "regulatory alignment" between Northern Ireland and Ireland after Brexit. Both sides would promise to maintain compatible trading rules, keeping the border transparent for trade.
Irish and EU officials indicated that agreement was close. But then Northern Ireland's Democratic Unionist Party, which props up May's minority government, announced it wouldn't support any deal that made special rules for Northern Ireland. The pro-British Unionist party opposes any special status that could take Northern Ireland further from Britain and closer to the Republic of Ireland.