The Liberal Democrat candidate won with 20,510 votes, ahead of Goldsmith's 18,638 votes, local mayor and councillor David Linnette announced.
The race left six other candidates falling far behind Olney and Goldsmith, in a by-election which saw 41,367 votes cast with turnout at 53.6 percent.
The vote has been widely seen as a mini-referendum on Brexit in a pro-European Union heartland, with locals turning out to support the Lib Dem centrist party that wants a second referendum on EU membership.
The outcome will come as a shock for Goldsmith, who was re-elected in the 2015 general election with a 23,000 majority.
More From This Section
Following his resignation from the Conservatives the party announced it would not put forward a candidate to challenge the Richmond seat.
Olney, 39, is a newcomer to politics -- an accountant who joined the Lib Dems only in May 2015.
In the June referendum on EU membership, 52 percent nationwide voted to leave, but in Richmond, a well-heeled borough in southwest London, 69 percent voted to remain in the bloc.
"While some people feel very strongly about Heathrow expansion, lots more people feel much more strongly about Brexit," Olney told AFP during the campaign.
"That's really alarmed and upset people and they want to use this opportunity to send a message.