The clashes raged through the night and into the morning on Saturday in the border region of Qalamun, said the Syrian Observatory for Human Rights.
Among the dead were at least seven pro-regime fighters, including government soldiers and members of Hezbollah, the Lebanese Shiite militant group that backs Syria's President Bashar al-Assad.
Regime forces recaptured most of the Qalamun region in April, with many rebel fighters withdrawing from the strategic area or slipping across the border in Lebanon.
Though the Islamic State and Al-Nusra Front both have roots in Al-Qaeda, IS has formally broken with the group, while Nusra is its official branch in Syria.
Despite ideological similarities, the two groups are opposed and in conflict with each other in other parts of Syria, particularly in the north.
But in Qalamun, their fighters battled the regime and Hezbollah forces alongside each other, with support from some smaller Islamist rebel groups, the Observatory said.