Members of the vigilante group made of civilians were trying to stop Islamists from attacking the town of Benishiek in Borno state, Boko Haram's historic stronghold, state government official Garba Ngamdu said.
"When the terrorists arrived early on Sunday morning, there was exchange of fire between them and the vigilante," he said, adding that the vigilante lost 13 of its members while five Boko Haram fighters were also killed.
According to him, they were ambushed while patrolling the town in search of the insurgents.
The oil rich country's President Goodluck Jonathan had declared a state of emergency in three states where the militants are very active.
Nigeria has 36 states and the country's population is more than 150 million.