The Y chromosome is one of two human sex chromosomes. Unlike the X chromosome, the Y chromosome is passed exclusively from father to son.
Previous research has shown that the DNA of modern humans is from 2.5 to 4 per cent Neanderthal DNA, a legacy of breeding between modern humans and Neanderthals 50,000 years ago.
Researchers found that unlike other kinds of DNA, the Neanderthal Y chromosome DNA was apparently not passed to modern humans during this time.
"We have never observed the Neanderthal Y chromosome DNA in any human sample ever tested. That does not prove it is totally extinct, but it likely is," said Carlos Bustamante from Stanford University in the US.
Also Read
Another possibility is that Neanderthal Y chromosomes include genes that are incompatible with other human genes, researchers said.
One of the Y chromosome genes that differ in Neanderthals has previously been implicated in transplant rejection when males donate organs to women.
"The functional nature of the mutations we found suggests to us that Neanderthal Y chromosome sequences may have played a role in barriers to gene flow, but we need to do experiments to demonstrate this and are working to plan these now," said Bustamante.
Because these Neanderthal antigen genes are on the Y chromosome, they are specific to males.
A woman's immune system might attack a male foetus carrying Neanderthal H-Y genes. If women consistently miscarried male babies carrying Neanderthal Y chromosomes, that would explain its absence in modern humans, researchers said.
So far this is just a hypothesis, but the immune systems of modern women are known to sometimes react to male offspring when there is genetic incompatibility, they said.
After the final split from other apes, the human lineage branched into a series of different types of humans, including separate lineages for Neanderthals and what are now modern humans.
The findings were published in The American Journal of Human Genetics.