The fossil, discovered in 2014, has a poorly preserved head, but it seemed to have a flamingo-like beak.
Scientists found that "beak" is actually part of a hammerhead-shaped jaw apparatus, which it used to feed on plants on the ocean floor.
"It's a very strange animal. It's got a hammerhead, which is unique, it's the first time we've seen a reptile like this," said Olivier Rieppel, from The Field Museum in Chicago.
However, newly discovered fossils make it clearer how its "strange teeth" were actually configured. Its wide jaw was shaped like a hammerhead, and along the edge, it had peg-like teeth.
More From This Section
Then, further into its mouth, it had bunches of needle-like teeth, researchers said.
"To figure out how the jaw fit together and how the animal actually fed, we bought some children's clay and rebuilt it with toothpicks to represent the teeth," said Rieppel.
"We looked at how the upper and lower jaw locked together, and that's how we proceeded and described it," he said.
"It used the peg-like front teeth to scrape plants off of rocks on the sea floor, and then it opened its mouth and sucked in the bits of plant material," Rieppel said.
"Then, it used its needle-like teeth as a sieve, trapping the plants and letting the water back out, like how whales filter-feed with their baleen," he said.
Not only does this discovery solve the mystery of the strange-toothed animal, but it also provides us with an example of the first herbivorous marine reptile.
"The jaw structure is clearly that of an herbivore. It has similarities to other marine animals that ate plants with a filter-feeding system, but Atopodentatus is older than them by about eight million years," said Rieppel.
"Animals living the years surrounding the Permian-Triassic extinction help us see how life on earth reacted to that event," said Rieppel.
"The existence of specialised animals like Atopodentatus unicus shows us that life recovered and diversified more quickly than previously thought," he said.
The study was published in the journal Science Advances.