Investigators retrieved a leg, a head, and part of a torso in the one-metre deep water, while a turtle was caught with an unspecified body part in its mouth, Nippon Television reported.
Police confirmed the discovery of some body parts but declined to give further details or speculate on how the remains ended up in the shallow pond.
"Whether or not it is a crime hasn't been confirmed yet," a police spokesman told AFP.
But police believe the dismembered remains belonged to an adult, Kyodo news agency said.
Also Read
No clothing or shoes were found near the pond, reports said. Police are searching for more body parts.
The small park in a residential neighbourhood in western Tokyo is accessible around the clock and has no security guards, they added.
Visitors can rent boats to row on the pond in the park, which also has baseball field and a gymnasium.
"This is a tranquil park where children come and play all the time," a long-time resident told NHK.