The 34-year-old singer secured full custody of his sons, Usher V, five, and Naviyd, four, after a lengthy battle with his ex-wife Tamika Foster.
Usher does not trust anyone else with his kids after their former nanny, Cecilia Duncan, filed a lawsuit against him, claiming she was unfairly fired for asking for overtime pay, reported Contactmusic.
"He's at the point where he doesn't trust anyone to take care of his boys. Besides, he didn't fight so hard for custody just to hand them over to a nanny," a source said.
"Blake told him that stars are always targets for people who think they can treat celebrities like cash registers. But Usher is really steamed about this lawsuit. He believes he's always been generous to people who work for him. If he has to leave 'The Voice' to free up more time to be with his kids, he's ready to do it," the source added.