A fifth suspect, a minor, was in detention in a juvenile centre pending further proceedings, the source told AFP.
Spanish and Moroccan authorities said the suspects were believed to have enlisted 12 people who were then sent to IS-run territories in Syria and Iraq.
Judge Santiago Pedraz at the National Court in Madrid "ordered unconditional detention for membership in a terrorist organisation" for the four adults, the source added.
The five were taken into custody on Tuesday in a series of raids against a suspected recruiting cell that police say contacted women online and radicalised them.
More From This Section
According to the source, one of the suspects refused to answer the judge's questions while the three others responded.
The Moroccan government said the suspects had links to members of the IS group planning attacks in Morocco.
Ten years after the March 2004 Al-Qaeda-inspired bombings killed 191 people in Madrid, police and security analysts say homegrown jihadists, already familiar in Britain and France, have started to appear in Spain.
In August police detained two females -- a 19-year-old from Melilla and a girl of 14 from Ceuta -- who tried to cross from Melilla into Morocco, suspected of heading for Syria.
The 14-year-old is also in detention in a juvenile centre.