The study used magnetic resonance imaging (MRI) and diffusion tensor imaging (DTI) to scan postmortem specimens of two Tasmanian tiger or thylacine brain specimens, both of which were about 100 years old.
The results, when compared to the Tasmanian tiger's closest living relative, the Tasmanian devil, suggest that the larger-brained thylacine had more cortex devoted to planning and decision-making.
"The natural behaviour of the thylacine was never scientifically documented," said Gregory Berns, neuroscientist at Emory University in the US.
The comparative study also supports theories of brain evolution suggesting that as brains grow larger they become more modular or divided into sections associated with discrete functions, said Berns.
More From This Section
"The technology for imaging the preserved brains of rare, extinct and endangered species is an exciting innovation in the study of brain evolution," said Kenneth Ashwell from the University of New South Wales in Australia.
The Tasmanian tiger looked like an amalgam of several animals. It is one of only a few marsupials to have a pouch in both sexes.
It was the size and shape of a medium-to-large size dog, but had tiger-like stripes running down its lower back and an abdominal pouch.
The fossil record shows that the Tasmanian tiger appeared about four million years ago in Australia. By the 20th century it was extinct, or extremely rare, on the mainland but was still found in Tasmania, the island state off Australia's southern coast.
The study was published in the journal PLOS ONE.