Heavily armed fighters "attempted to cross the bridge at Fotokol" in the extreme north of the country and opened fire on Cameroonian soldiers, the ministry of defence told state radio.
"Cameroon defence forces energetically reacted to this assault which lasted three hours," the ministry said, adding that one soldier was wounded by mortar shrapnel.
There was no independent confirmation of the battle.
Gamboru Ngala, the Nigerian town on the other side of the bridge, fell to the Islamist extremists last week after they reportedly overran the Nigerian garrison there.
Also Read
The extremists, who have waged a bloody insurgency for five years in northern Nigeria, seem to have changed tactics in recent months, going from spectacular kidnappings, massacres and suicide attacks to attempting to conquer territory.
Cameroon's Defence Ministry said that 246 Nigerian soldiers and customs officials who had fled Gamboru Ngala into Cameroon to escape the Boko Haram offensive "have left the Fotokol area under military escort" to rejoin their units in Banki in Nigeria.
The Nigerian army denied its troops had fled into Cameroon, instead calling the retreat a "tactical manoeuvre".