Police said the 41-year-old died at popular tourist spot Shelly Beach near Ballina, some 116 miles south of Brisbane.
A group of surfers helped get him out of the water and first aid was administered, but to no avail.
"We can confirm the man is a Japanese national, aged 41," Detective Inspector Cameron Lindsay told the local Northern Star newspaper, adding that efforts were being made to contact his family.
A local cafe owner, identified only as Karen, told reporters the shark came from nowhere.
More From This Section
"It just came up between a bunch of surfers," she said. "They weren't even that far out. He lost both his legs."
Police only described the man's injuries as "substantial".
"He had leg injuries and they were substantial, they were injuries that led to a significant loss of blood," said Lindsay.
Ballina Shire mayor David Wright said rescuers believed it was a great white shark.
"For a shark to take the board and the person sitting on it, it's got to be very big," he said.
He suffered a cut to his back and puncture wounds to his buttocks.
"Something just jumped out of the water and just grabbed me at my hip and dragged me off my board and took me under the water," he told the Australian Broadcasting Corporation.
"I just came back up and I paddle, paddle, (and) paddled about 60 metres back into the shore."
Experts say attacks by sharks are increasing as water sports become more popular.
Fatalities, however, remain rare although two teenagers died in shark attacks in December, one on the country's west coast and one on the east.