The song titled 'Put it Up' was not included in the album's track list as Brown does not want to do anything with his on-again-off-again girlfriend, Ace Showbiz reported.
"Just saw the working tracklist for 'X' yesterday -- Rihanna is not on there. Record execs wanted that song on there, but Chris said he would rather not and wouldn't allow the record to be released unless they took it off," a source said.
The record company executives are still trying to convince Brown to reconsider and according to the source, they are even ready to pay him more to have the track back on the album.
Brown, 24, and Rihanna's relationship has been rocky throughout. The couple first broke up in 2009 when Brown infamously assaulted his singer girlfriend, which almost ended his career.
They got back together again last year only to split again.
If the song makes the cut, it would be their second collaboration to 2012 track 'Nobody's Business'.