Worshippers sang solemn hymns in a community hall in Qunu, calling on God to heal the revered leader "during this difficult time".
"We want him to recover from whatever pain he is experiencing. That is what we are here for and praying for," said Methodist Church minister Sonwabile Msotyana, as Mandela started a fourth week in hospital.
"This is a very important man that is lying there, not only for South Africa but the entire world," he said.
Mandela's oldest grandson Mandla and several other relatives also joined the inter-denominational service around 1,000 kilometres away in South Africa's rural hinterland.
More From This Section
Supporters of the ruling African National Congress (ANC) dressed in party colours also attended.
Mandela spent 27 years in jail for his opposition to white minority rule, and went on to become the nation's first black president after all-race elections in 1994.
"The new South Africa would not have been possible" without the sacrifices made by Mandela and other anti-apartheid leaders, Methodist Bishop Don Dabula said.