Abe, accompanied by his wife, spent 40 minutes at the Kelaniya Rajamaha temple which Buddhists believe was visited by the Buddha himself more than 2,500 years ago.
During his overnight official visit, Abe had paid tribute to a former Sri Lanka President Junius Jayewardene, who was cremated at the temple grounds in 1996.
Jayewardene had made an impassioned plea on behalf of Japan at the 1951 Peace Treaty signing in San Francisco which officially ended World War II, and demanded war reparations.
After the San Francisco treaty, Tokyo became a strong political and economic ally of the island and is still Sri Lanka's largest single donor of foreign aid.
More From This Section
In talks with President Mahinda Rajapakse on Sunday, the two leaders agreed to forge stronger maritime links between their countries in a move seen as countering China's influence in the region.
"It is my intention to increase cooperation (with Sri Lanka) in the maritime area for open and safe seas," Abe told a business forum in Colombo attended by leaders of several high profile Japanese companies.
Abe is the first Japanese premier to visit Sri Lanka in 24 years, and follows his grandfather and then prime minister, Nobusuke Kishi, who paid an official visit in 1957 to the island then known as Ceylon.
Abe's visit comes ahead of an official tour of India and Sri Lanka by Chinese President Xi Jinping expected next week.
China is increasingly asserting its influence in the Indian Ocean, with Sri Lanka a midway point on one of the world's busiest international shipping lanes.