"We've been talking about it for a few years, funny enough, me and NBC," Cowell said that he and the network first discussed the job five years ago, but that it didn't pan out at the time.
"This year... Everything worked. So it was perfect," he added.
The reality TV personality, 56, is known for his unflinching honesty when it comes to critiquing performers. He previously judged "American Idol", from its premiere in 2002 until he left in 2011 after nine seasons.
Stern, 61, joined the popular NBC show in 2012 for Season 7, and stepped down after Season 10 concluded in September.
AGT is hosted by Nick Cannon, and judged by Howie Mandel, Heidi Klum, and Mel B, though none have been officially confirmed for Season 11 yet.