Both the stars were married at the time but Alley says sparks flew between her and Swayze almost immediately though they tried their best to keep it secret.
"Both of us were married," Alley said, clarifying that the pair did not technically "have an affair."
"But again, I think what we did was worse. Because I think when you fall in love with someone when you're married, you jeopardize your own marriage and their marriage. It's doubly bad," she told Entertainment Tonight.
Alley said she is good friends with Patrick's wife of 34 years Lisa Niemi Swayze and she was even asked to speak at his funeral after he died of pancreatic cancer in 2009. But she is not sure Lisa knew about their past connection.
Earlier this week, Alley called John Travolta "the greatest love of my life". The actress said she fought the urge to "run off and marry John" while the two worked together on the set of 1989's 'Look Who's Talking'. Travolta was single at that time while Alley was married to Parker Stevenson.