Coulson also served as British Prime Minister David Cameron's director of communications until 2011, earning him the unofficial title of spin doctor.
He resigned from the post soon after allegations of phone-hacking at Britain's major tabloid offices hit the headlines.
He arrived at court hand-in-hand with wife Eloise Patrick before taking to the witness box.
He denied his affair with Brooks breached professional standards but agreed it was "wrong" and "shouldn't have happened" as he gave evidence.
Also Read
But father-of-three replied: "No, that did not happen. With the caveat, unless on very particular occasions there was a pre-determined deal when there was a share between the two papers ('The Sun' and 'News of the World')."
The 46-year-old denies conspiracy to hack phones and conspiracy to commit misconduct in a public office.
Coulson was the last of seven defendants, who deny all charges against them, to take the stand.
Coulson also told the court that shortly after he resigned, he and his family had spent a pre-arranged weekend with Cameron in April.
He said after that he had experienced "no further contact" with Cameron.
He was also asked what contact he had experienced with Murdoch, the owner of News International, after he had left the 'News of the World'.