The 26-year-old actress has grown increasingly close to Hiddleston but insiders insist their romance is not serious, reported Us magazine.
"They have been hooking up. It's casual (but they) text when they're apart," a source said.
Olsen split from fiance Boyd Holbrook in January and is apparently not looking for more than "fun" in her personal life at the moment.
"She wants to make a new life for herself. She's having fun," the source said.
Meanwhile, Hiddleston, 34, was previously romantically linked to record producer Jane Arthy.