The defeat means that the tournament's top three seeds have been ousted inside six days, with top seed Serena Williams losing in the second round and second seed Li Na departing in the first.
Ranked 72nd in the world, 21-year-old Tomljanovic will play the winner of the tie between Taylor Townsend of the United States and 14th seed Carla Suarez Navarro of Spain for a place in the quarter-finals.
Fourth seed Simona Halep of Romania is the top seeded player left in the draw, having reached the third round on Thursday.
"I went out out there and I really felt that I could win. It showed and that was how I won.
More From This Section
"I have a fourth round next and obviously happy, but I do not want to get too happy about it either."
Tomljanovic ended 2013 ranked 78th in the world, improving from 495th - the biggest ranking improvement by any player in the top 100.
This year she lost in the third round at the Australian Open but more recently failed to qualify for the main draw at Madrid and Rome in the buildup to Roland Garros.
Last year she was a semi-finalist at Wimbledon, a quarter-finalist at the Australian and French Opens and made the fourth round at the US Open.
Watched by Croatian 1997 French Open winner Iva Majoli, Tomljanovic quickly took control of a match played in front of a sparse centre court crowd.
She jumped out into a 5-1 before a shell-shocked Radwanska managed to break serve and get back to 5-4.
The Croatian made no mistake on serve in the following game though and an early break in the second set sent her on the way to the biggest win of her career.