At just 28 weeks' gestation, the babies appeared to discriminate between different syllables like "ga" and "ba" as well as male and female voices, French researchers found.
They studied detailed brain scans of 12 infants born prematurely, the BBC News reported.
The team said it was unlikely the babies' experience outside the womb would have affected their findings.
Experts already know that babies are able to hear noises in the womb - the ear and the auditory part of the brain that allow this are formed by around 23 weeks' gestation.
Also Read
Writing in Proceedings of the National Academy of Sciences (PNAS), scientists said environmental factors are undoubtedly important, but based on their findings they believe linguistic processes are innate.
"Our results demonstrate that the human brain, at the very onset of the establishment of a cortical circuit for auditory perception, already discriminates subtle differences in speech syllables," Dr Fabrice Wallois and colleagues wrote.
However, this "does not challenge the fact that experience is also crucial for their fine tuning and for learning the specific properties of the native language", they added.