The attackers opened fire at the place of worship in Etogi village during morning prayers on Saturday, Niger state police spokesman Bala Elkana said.
"The gunmen opened fire on the worshippers inside the mosque during morning prayers and killed all the 20 worshippers," Elkana said.
"They later shot sporadically on the village, injuring eight people."
The shooting was a revenge attack after farmers killed one herder in a clash days earlier, the spokesman added.
More From This Section
But the herders reneged on the agreement this year, refusing to pay the tax and instead claiming that the land belongs to them, according to the police.
"This led to a clash in which one herder was killed. The (latest) attack was to avenge the death," Elkana explained.
Three suspects have been arrested in connection with the violence.
Deadly clashes between herders and farmers in Nigeria over land and water rights are frequent, especially in the Middle Belt region, which is considered the country's agricultural heartland.
Sparse vegetation in the north along with the gradual encroachment of the desert has forced largely nomadic herders to move south in search of food for their cattle.
That has led to clashes with farming communities over competition for limited resources.
Efforts by authorities to end hostilities between the two sides have not been successful.
The Nigerian government last year announced a plan to create grazing reserves for herders across the country to relieve tensions.
But farming communities rejected the plan, forcing the government to put it on hold.