The protesters reached the 8, Safdarjung Lane residence of Swaraj at around 1 PM and protested there for about 45 minutes.
Soon after the protest began, police rushed to the spot and removed the activists who were later released.
Youth Congress national president Rajiv Satav said they were protesting against the "double-speak" of Swaraj.
Swaraj had accused Gandhi of preventing her from speaking in Lok Sabha because the Congress chief had "instigated" Congress members to create disturbances.