"The UK would need to be treated as a friend, but not as family. Relations would be polite but not intimate. The UK would be respected but not trusted," Muscat said.
Britain is set to vote on whether to remain in the 28-member bloc on June 23, and Malta is set to take over the EU presidency shortly afterwards for the first half of 2017.
Speaking at the Chatham House think-tank in London yesterday, Muscat said that the process of revising relations with Britain would involve "acrimony and resentment".
"EU leaders would be keen to show to their national audiences which might warm to the idea of leaving the union that such a process would be very ugly, painful and costly."
More From This Section
Because Malta is part of the Commonwealth, a group of states which are mostly ex-British colonies, Maltese citizens resident in Britain are eligible to vote in the referendum.
There are 27,000 Maltese-born residents in the United Kingdom, according to data from 2014.
Polls in Britain have the "Remain" camp as slight favourites.