"There is never a perfect time for this type of transition, but now is the right time," Ballmer said in a statement.
"My original thoughts on timing would have had my retirement happen in the middle of our company's transformation to a devices and services company. We need a CEO who will be here longer term for this new direction."
Ballmer took over as CEO in 2000 when co-founder Bill Gates stepped down from day-to-day operations at the company. At the time, Microsoft was the undisputed tech sector leader, and the world's largest company in market value.
While its Windows software is used on the vast majority of personal computers, Microsoft has had little impact in the fast-growing segments of tablets and smartphones.
More From This Section
Microsoft shares leapt nearly nine percent in pre-market trading on the news.
Ballmer, 57, will continue as CEO "and will lead Microsoft through the next steps of its transformation to a devices and services company that empowers people for the activities they value most," said a statement from the Redmond, Washington, company.
The board of directors has appointed a special committee to direct the process chaired by led independent director John Thompson and including Gates, Chuck Noski and Steve Luczo.
"As a member of the succession planning committee, I'll work closely with the other members of the board to identify a great new CEO," said Gates.
"We're fortunate to have Steve in his role until the new CEO assumes these duties.