But experts warn the ELN will be a tougher negotiating partner than the FARC, and say no deal is likely before President Juan Manuel Santos -- the man who has staked his presidency on ending the conflict -- leaves office next year.
Santos, who won the Nobel Peace Prize in October, was nevertheless full of optimism heading into the talks.
"This conflict is over," he said Thursday.
"The public phase of negotiations between the Colombian government and the ELN... Will enable us to achieve complete peace."
More From This Section
Colombia, South America's third economy and the world's biggest cocaine producer, has been torn since the 1960s by fighting that has drawn in multiple leftist rebel groups, right-wing paramilitaries, drug gangs and the army.
Last November's landmark peace accord with the oldest and largest rebel group, the Revolutionary Armed Forces of Colombia (FARC), after four years of talks leaves the National Liberation Army (ELN) as the last active guerrilla insurgency.
It has an estimated 1,500 fighters, mostly in the north and west.
A flurry of behind-the-scenes negotiations followed, leading to Sanchez's release on Thursday in exchange for two ELN prisoners.
In a further goodwill gesture yesterday, the ELN released a soldier they had captured two weeks earlier.
But there will be more bumps in the road, warned Frederic Masse, an expert on the conflict at the Universidad Externado in Bogota.
"The ELN has more fundamentalist demands than the FARC," he said.
A prominent ELN commander warned ahead of the talks that the rebels would not back down on the thorny question of land rights for the rural poor -- one of the main issues in the conflict.
"As long as the necessities that were at the root of this insurgency exist, we will have to keep fighting," Danilo Hernandez, commander of the Resistencia Cimarron guerrilla front said.