About 5,000 attended the event, held under the banner "Stand Up: Jew Hatred - Never Again!", which coincided with the first ever annual meeting of the World Jewish Congress (WJC) in the German capital.
"The fight against anti-Semitism is our state and civic duty," said Merkel in an address at Berlin's iconic Brandenburg Gate, close to the city's Holocaust Memorial.
"I will not accept it, and none of us here will accept it."
More seriously, four people were shot dead in May at the Jewish museum in Brussels, Belgium. The museum only reopened today.
More From This Section
The spate of ugly incidents that deeply unsettled Germany's resurgent 200,000-strong Jewish community also saw a petrol bomb hurled at the facade of a synagogue in the western city of Wuppertal.
The attacks came 75 years after the outbreak of World War II, during which Nazi Germany murdered six million Jews, a crime that remains a source of enduring shame in the country.
"Jewish life belong here and is part of our identity.... There must be no room for discrimination and marginalisation," she said.
Merkel -- who has won Israel's highest civilian honour for her efforts against anti-Semitism and Holocaust denial -- said that "Germany is aware of its eternal responsibility for the break of civilisation called the Shoa".
"Those who hurl abuse at people who wear a kippah or a Star of David on a chain, attack them or assault them, also hit out at and injure all of us," she said.
Germany, which was home to some half a million Jews before WWII, saw that number plummet to only around 15,000 after the war. The Jewish community has grown again, in part as Germany took in Jews from the former Soviet Union, and now numbers around 200,000.
WJC president Ronald S Lauder praised post-war Germany for being "one of the most responsible countries on earth" and "Israel's ally and friend," but added that "something has changed".
He cited several reasons for the recent resurgence of the "medieval stain of anti-Semitism".