The 70-year-old was heard to say about Williams's pregnancy at Friday's press conference ahead of the World Group play-off: "Let's see what colour it (the baby) has. Chocolate with milk?"
Furious that his remarks were made public, Nastase's anger had not abated by the time the tie reached the second rubber on Saturday between British number one Johanna Konta and Sorana Cirstea.
According to reports, he used foul and abusive remarks courtside to pregnant British captain Anna Keovathong -- whose room number he'd asked for at the press conference -- and Konta.
Nastase had also been riled by the umpire appealing to the crowd for calm with the Romanian captain interjecting: "It's not the opera, what's your fucking problem?"
More From This Section
After a lengthy delay to proceedings, Nastase was banished to the stands before being finally kicked out of the tournament.
"His accreditation was removed and he will play no further part in the tie," said the International Tennis Federation (ITF) in a statement.
"The ITF has launched an investigation into this matter as well as previous comments made by Mr Nastase during the week," the ITF added, referring to the comments about Williams's pregnancy.