"We are no longer going to the EU-Africa. We disagreed on the composition of our delegation," said a source at the ministry, who asked not to be named.
Harare had earlier today urged the African Union to shun the summit for failing to invite all the Africa bloc's leaders and lift a ban on Zimbabwe's first lady.
But diplomats in Brussels were unfazed by the call for a boycott.
"We see no risk" of a boycott of the April 2-3 summit, which gathers 90 nations from both continents, including 65 heads of state and government, said a senior official speaking on condition of anonymity.
Another concern is "the issue of our first lady who was denied a visa," Bimha told AFP, referring to Mugabe's wife, Grace.
Mugabe and his wife remain targeted by an EU travel ban but the restriction can be suspended temporarily to allow the head of state to attend international forums.