Unfancied Croatia took a shock lead when Marcelo put the ball into his own net early in the Group A match before Neymar levelled in front of a jubilant crowd in the Corinthians Arena.
But the European side were holding their own in the second half until Japanese ref Yuichi Nishimura pointed to the spot after Fred backed into Lovren and fell.
"I think this referee didn't deserve to be here," said the Southampton player. "I don't know for what he deserved to be here. It's really disappointing for us because I think we were playing a good game against a big team, Brazil.
"We're really disappointed but what can you do against 12 players? You can't do anything," he added.
Also Read
"I'm sad, that's it, because we played a good match and the referee -- it wasn't a mistake, it was a scandal. It's a scandal for everyone, it's a scandal for FIFA. I think this referee shouldn't be at this World Cup."
Frustrated Croatia coach Niko Kovac said the referee was "out of his depth" but his Brazilian counterpart Luiz Felipe Scolari was adamant Nishimura was correct to award the spot kick converted by Neymar to make the score 2-1.
"If you continue in this vein then there will be 100 penalties during this World Cup.