The 45-year-old actress was initially linked to the controversial film, which looks back at a self-diagnosed nymphomaniac's erotic experiences with a man who saved her after a beating, The Hollywood Reporter said.
While the reason for Kidman's departure has not been confirmed, it is thought she is not keen to star in the project since Trier revealed he is planning to use real sex scenes.
Fellow star Shia LaBeouf seems keener about the nature of the feature and even sent the director sex tapes of him and his girlfriend as part of his audition to be in the film.
Charlotte Gainsbourg is to play sex addict Joe in the movie, and Willem Dafoe, Uma Thurman, Christian Slater, and Stellan Skarsgard are also in line to appear in the motion picture.
Two versions of the film, hardcore and softcore, are to be released in 2013.
