The discovery used observations made by NASA's Kepler satellite.
The star, named Kepler-444, hosts five planets smaller than Earth, with sizes varying between those of Mercury and Venus.
"We've never seen anything like this - it is such an old star and the large number of small planets make it very special," said Dr Daniel Huber from the University of Sydney's School of Physics.
"It is extraordinary that such an ancient system of terrestrial-sized planets formed when the universe was just starting out, at a fifth its current age. Kepler-444 is two and a half times older than our solar system, which is only a youthful 4.5 billion years old.
Also Read
"We now know that Earth-sized planets have formed throughout most of the universe's 13.8-billion-year history, which could provide scope for the existence of ancient life in the galaxy," Dr Tiago Campante, the research leader from the University of Birmingham said.
Together with their international colleagues the team used asteroseismology to determine the age of the star and planets.
This technique measures oscillations - the natural resonances of the host star caused by sound waves trapped within it.
Their closeness to their host star means they are uninhabitable because of the lack of liquid water and high levels of radiation, researchers said.
Discoveries like Kepler-444 provide important clues on whether a planet that is more truly comparable to Earth may exist.
"We're another step closer towards finding the astronomers' holy grail - an Earth-sized planet with a one year orbit around a star similar to our Sun," said researchers.
The study was published in the Astrophysical Journal.