The Moria camp, which was visited by Hollywood star and UN refugee agency envoy Angelina Jolie last week, has now been sealed and migrants transferred there are "not allowed to go out," said a spokesman.
The UNHCR, which has a policy of not working in sealed camps, will scale back there and only monitor it to identify and assist the most vulnerable migrants, including unaccompanied minors.
Spokesman for the UN High Commissioner Refugee Agency Boris Cheshirkov said: "We have taken a principled decision to disengage from transportation operations from the shoreline and the port to the closed facility of Moria."
He added that the "majority" of migrants and refugees in Lesbos before Sunday "had left the island" to reception facilities on mainland Greece.
The agency has also raised fears that the EU-Turkey migrant agreement has been rushed into force. "The UNHCR is concerned about the premature implementation of the deal," said Cheshirkov.