The country's indigenous people are the nation's most disadvantaged, with a much shorter life expectancy than other Australians while suffering disproportionate levels of imprisonment and social problems such as unemployment.
Abbott said late Tuesday he supported a plan to close more than 100 remote Aboriginal communities across the vast Western Australia state if essential services could not be provided.
"It's the job of the taxpayer to provide reasonable services in a reasonable way, indeed to provide high quality services in a reasonable way," he told the Australian Broadcasting Corporation.
Aborigines have lived in Australia for at least 40,000 years and the comments drew stinging criticism, with Abbott's key indigenous advisor Warren Mundine saying Aborigines had a cultural connection to their land, and it was not simply a matter of going to "live in the bush".
Also Read
"These people are actually living on their homelands and it affects a lot of things, it affects their cultural activities, it affects their native title, it affects a number of areas," he said.
Aboriginal and Torres Strait Islander Social Justice Commissioner Mick Gooda said the comments were "baffling" and would cause offence in the indigenous community.
"We're going to make a situation that's pretty bad already in those places even worse," he said.
Abbott, a passionate supporter of recognition for first Australians in the nation's constitution, last year spent almost a week running the government from a remote Aboriginal community and will do the same this year.
But Rolf de Heer, an acclaimed filmmaker who has made indigenous movies including "Ten Canoes", said the comments were "so inappropriate that it's laughable".
Aboriginal leader Noel Pearson, who has heavily influenced conservative thought on indigenous policy, added that it was a "hopeless statement by the prime minister".