The remains of this large creature give clues about how mammals "took over" when dinosaurs died out.
Kimbetopsalis simmonsae, as the newly-discovered species has been named, was a plant-eating creature that resembled a beaver.
Carissa Raymond, a student in the team of Dr Stephen Brusatte from the University of Edinburgh found the fossil while prospecting at a site in New Mexico, US.
"We realised pretty quickly that this was a totally new type of mammal that no-one has seen before," Brusatte, lead researcher on the study, told BBC News.
More From This Section
They named the species after Kimbeto Wash, the area in the New Mexico badlands where it was found.
"The other part of the name - psalis - means 'cutting shears' and is in reference to [the] blade-like teeth," Brusatte said.
This group of now-extinct mammals, collectively known as multituberculates, originated alongside the dinosaurs, during the Jurassic and thrived for more than 100 million years until they were apparently superseded by rodents.
"[During the Jurassic] these animals were all pretty small. Then the asteroid hit, wiped out the dinosaurs and suddenly - in geological terms - this [group of animals] started to proliferate and get bigger," said Brusatte.