The 48-year-old 'Iron Man 3' star said his childhood addiction to drugs like marijuana forced his career to take a back seat, reported Contactmusic.
"My attitude was always,I don't care. I don't care about fame. I don't care about my career. It's not important to me," he said.
The 'Sherlock Holmes' star credits his second wife, Susan Downey, whom he married in 2005 for getting his life back on track and encouraging him to create order in life and putting the future plans in place.
However, the veteran actor feels his life would have been easier if his father, Robert Downey Sr, had put firmer rules in place when he was a youngster.