Olivier, 42, who is the half-brother of former French President Nicolas Sarkozy, has been dating the 26-year-old fashion designer for around five months, reported New York Post online.
"Olivier and Mary-Kate fell in love with the 146-year-old property because they like that it is old," a source said.
The 4,200-square-foot, five-storey home was designed by architect James Renwick Jr and comes with nine fireplaces, a Juliet balcony in front and a large rear garden.
It isn't known how much renovation they plan to do, although the plumbing and kitchen need to be completely updated.
Mary-Kate dated artist Nate Lowman for over two years before splitting in 2010, while Olivier romanced actress Stella Schnabel for a year before their break-up in March.
Olivier was previously married to wife Charlotte and they have two children together.
Charlotte has since called the coupling grotesque and not right.