The research was conducted on the colony of storm petrel birds in a tiny island off the coast of Spain called Isla de Benidorm. Storm petrels are small nocturnal seabirds which emit a musky smell.
Lead researcher Francesco Bonadonna, from the Centre of Functional and Evolutionary Ecology in Montpellier, France, told 'BBC Nature' that the birds used smell to recognise and communicate their "genetic compatibility".
It is well known that mammals can smell a genetically suitable mate. Until recently, scientists thought that birds relied on vision and sound when choosing a partner.
According to Bonadonna, the fact that they use odours explains how these birds manage to return to their family colony to breed and avoid mating with a relative.
"These birds are theoretically faithful to one mate for life," the researcher explained to BBC Nature. "So a bad choice may have catastrophic consequences."
He said that smell or chemical communication was the most ancient and simplest form of communication in the animal kingdom, adding, "it makes sense that the birds would use it".
Bonadonna and his colleagues collected bird scents by "taking swabs" from a selection of birds in their study colony.
The scientists set a group of petrels a test, placing one cotton swab with the scent of a relative on one arm of a Y-shaped maze and a swab containing the scent of an unrelated bird on the other.
Almost all of the birds that performed the test chose to walk along the arm containing the scent of the unrelated bird.
European storm petrels chose to avoid the scent of a relative in favour of the smell of an unrelated bird as researchers think this prevents the birds from "accidentally inbreeding", the report said.
The study was published in the journal Animal Behaviour.
