The 44-year-old "Bourne Identity" star, who grew with Affleck said, "Daredevil" was the comic they read as kids, but he turned down the offer because he did not know the director, reported New York Daily.
"For us it was always 'Daredevil' - that's the comic we read when we were kids. But when that one came along (in 2003) I chickened out, because I couldn't tell. I hadn't seen the director's work and I didn't know. So I just said, 'No.'
"Ben was like, 'I gotta do it.' And the movie ended up doing very well, even though I don't think Ben was ultimately very proud of it," Damon said.
"Ben had that whole moment where he was in the crosshairs of (the tabloids) and couldn't really get a job. And he said to me at the time, 'I can sell magazines and not movie tickets, I'm in the worst place to be'...
"From there, he wrote and directed his way out of that and into a kind of place that neither of us had been - winning an Academy Award for Best Picture. So that's a good fallback. That mitigates some of the actor insecurity.