The two-day meeting in Kuala Lumpur, which was confirmed by the South Korean and US governments, was the latest in a series of unofficial talks commonly referred to as Track 2 that are closely monitored in the absence of any official contact between Washington and Pyongyang.
In July, the North cut off its only remaining official channel of diplomatic communications with the United States in retaliation for American sanctions against its leader, Kim Jong-Un.
American participants at the talks in the Malaysian capital included Robert Gallucci, who had led the US negotiating team that brokered a 1994 deal with Pyongyang on freezing its nuclear weapons programme.
Among those on the North Korean side was vice foreign minister Han Song-Ryol, who previously served as deputy ambassador to the UN.
More From This Section
The meeting came after North Korea on Thursday test-fired a powerful new medium-range missile and Leon Sigal, an academic specialising in the Koreas who attended the talks, said the North's nuclear weapons programme had dominated the discussion.
The US side stressed that the moves to scrap the nuclear programme had to come first, said Sigal.
Under President Barack Obama, the United States has eschewed an official dialogue with the North, but with a looming change in the White House, there is growing speculation as to whether a new administration might adopt a different track.
Critics of the current policy say sanctions and non-engagement have done nothing to prevent the North's accelerated drive towards a credible nuclear deterrent that could directly threaten the US mainland.
"We are aware that the US government maintains a firm stance that rushing into dialogue in the absence of North Korea's will to denuclearise will only justify their wrong behaviour," a foreign ministry official told AFP.
The UN Security Council is currently discussing a new resolution to punish North Korea over its fifth nuclear test in September - having already imposed tough economic measures after a fourth test in January.
The Track 2 talks have been taking place sporadically for years, with meetings in Singapore, Berlin, Beijing and elsewhere.