The 2 million pounds campaign, promoted by Transport for London, includes children's books distributed in nurseries as well as an interactive website.
The stories are illustrated with characters from ethnically diverse backgrounds.
The Muslim girl, aged three or four, is called Razmi and always shown wearing a religious headscarf. Razmi is seen indoors in the home of a Chinese girl and that child's grandmother and on outings, The Times reported.
The Children's Traffic Club London, promoted by Transport for London (TfL), has recruited more than 66,000 children across the capital.
More From This Section
The books were introduced under his Conservative predecessor Boris Johnson, now the foreign secretary, in 2015.
Gina Khan, an advocate of Islamic women's equality, was quoted as saying, "You are sexualising a four-year-old girl. It is as simple as that. The reason a female is covered is so men don't look at her. How can you integrate in society if you have a four-year-old girl wearing a hijab?"
Shaista Gohir, chairwoman of the Muslim Women's Network UK charity, said, "It's like trying to get that child to try to grow up far too quickly. A child needs to be treated like a child."
"The hijab is a Saudi-isation of British Muslim identity," she said.
"If you are a Muslim girl and look at these images and see this girl is Muslim and she is wearing a hijab and you aren't, you will think there's something wrong with you. It is far too young. You are a child. What are you being modest for?" she added.
Dame Louise Casey, the government's integration tsar, intervened in a row in Birmingham this year when a Catholic school came under pressure to let a four-year-old girl wear a Muslim headscarf.
A TfL spokeswoman was quoted as saying, "We apologise for any offence caused by this content and we will not use these designs in future.