Operation Sophia "does not in any meaningful way deter the flow of migrants, disrupt the smugglers' networks or impede the business of people smuggling on the central Mediterranean route", it said.
The European Union insisted that the British report "applauds the huge contribution of the EU operation" and said it had helped to reduce the incentive for people trafficking.
The report by the House of Lords' sub-committee on EU External Affairs noted that arrests so far have been of "low-level targets", and the destruction of vessels has caused smugglers to shift from using wooden boats to rubber dinghies, which are even more unsafe.
Committee chairman Lord Tugendhat said the mission to patrol an area six times larger than Italy "was always going to present an enormous challenge".
More From This Section
"Our report stresses that the operation is succeeding in carrying out its separate search and rescue obligations, which is to be commended," he said.
"However, a naval mission cannot disrupt the business model of people smuggling, and in this sense it is failing.
"The smuggling networks operate from Libya, and they extend through Africa. Without support from a stable Libyan government, the operation is unable to gather the intelligence it needs or tackle the smugglers onshore."
The spokeswoman said Operation Sophia had saved 13,740 lives including more than 850 children, contributed to the arrest of 69 suspected smugglers and neutralised 114 vessels.
"This is a substantial achievement in such short period of time," the spokeswoman said. "We are now looking at ways for the operation to be even more effective, building on these achievements.