Toback, an Oscar nominated director, has been accused of sexual harassment by more than 200 women in Hollywood following an expose in the Los Angeles Times.
"I actually did have a situation with James Toback where I kind of told him to go (expletive) himself," Pompeo told USA Today.
"It was held in a public place and I brought a man my friend Tony. And he didn't like that at all. And I could tell that. I could tell instantly that he was, like, no good. The minute my friend left, he asked me if I would get naked in a movie. And I was like, 'Really, dude? My friend has been missing all of 30 seconds and now you say that? I kind of laughed in his face," she said.
Pompeo's "Grey's Anatomy" costar Caterina Scorsone also detailed the harassment she experienced at the hands of Toback during an audition.
Pompeo also met Harvey Weinstein but said she did not face any unwanted attention from the Hollywood producer, who stands accused of sexual harassment, assault and rape by numerous women. He has denied any allegation of sexual assault.
"I met with Harvey at the Peninsula and he didn't try anything on me. I was very lucky. I escaped those clutches," she said.