A police statement said young Palestinians "threw stones and fireworks at police and border police forces," who responded with "riot dispersal means".
Calm returned to the compound later in the morning and most police had withdrawn but Israeli Arab activists remained inside, AFP correspondents reported.
Muslims have been alarmed by an increase in visits by Jews and fear rules governing the compound will be changed.
Al-Aqsa, the third holiest site in Islam, is also venerated by Jews as the Temple Mount and is considered the most sacred place in Judaism.
More From This Section
Visits by Jews were stopped on Sunday and age restrictions on Muslim men entering the compound lifted.
But recent weeks have seen a series of Jewish holidays during which there has an been an uptick in visits by Jews that have sparked repeated clashes.
The radical northern branch of the Islamic Movement in Israel and the Higher Arab Monitoring Committee (HAMC), which represents Arab communities in Israel, had urged Muslims to go to the compound to defend it.
AFP correspondents saw around 150 people at the compound sporting green Islamic Movement caps.
Several Arab members of the Israeli parliament were also present at the compound.
Islamic Movement official Sheikh Qairi Eskender said: "We're afraid they want to divide the Aqsa compound, because the Jewish extremists want to take control of Al-Aqsa.
"Our goal today is to prevent Al-Aqsa from being tarnished by their visits," Eskender told AFP.
"In the coming days, we'll have people staying here."
Israeli Prime Minister Benjamin Netanyahu has said repeatedly that there will be no change to the rules governing the compound despite the views of some hardliners within his governing coalition.