"It is clear that 20 individuals...," told their lawyer before the ICC "that they no longer wish to continue participating in the present proceedings," the court's judges said.
They added: "The decision to withdraw could have been motivated by a range of factors including security concerns."
The court's confirmation comes after it received a letter in early June by 93 victims who allegedly said they wanted to stop participation in the two Kenyan cases before the ICC.
Ruto went on trial in September, becoming the highest-ranking serving official to do so before the ICC.
Also Read
Kenyatta's trial was due to start on Tuesday, but has now been postponed to February.
In the letter dated June 5, the 93 victims of violence in Turbo town in western Kenya said they no longer wish to take part in the trial.
At least four people were murdered in Turbo as a result of the attacks in late 2007, the court heard previously.
Seven victims changed their minds after the letter was written and will now continue to be represented in the case.
Twenty other victims could not be contacted and were therefore considered to still be taking part, the judges added.
In all, some 628 victims are scheduled to have their case represented before the court by their Kenyan lawyer Wilfred Nderitu, said ICC spokesman Fadi El Abdallah.
Ruto, 46, and Kenyan radio boss Joshua arap Sang, 38, stand accused of stoking the worst violence in the east African country since independence in 1963.