The couple announced the news on Saturday, reported the New York Post
"After a lot of thought, care and consideration, we have decided to end our marriage," the pair said in a statement.
"It's a sad decision that did not come easily, especially because we respect and love each other and our children. They remain our top priority. We will raise them and care for them together."
Spencer, 45, and real estate broker Haffenreffer wed in Southampton in 2000.
A source said the break-up is amicable. They are even going on vacation with their children Kate and Duff later this week.