Mattis landed at the Nur Khan Airbase in Rawalpindi where he was received by officials from Pakistan's defence and foreign ministries.
He will hold talks with civilian and military leaders including Pakistan Prime Minister Shahid Khaqan Abbasi and Army Chief General Qamar Javed Bajwa.
Relations between the US and Pakistan have plummeted after President Donald Trump announced his new Afghan and South Asia policy in August and directly blamed Islamabad for supporting the Afghan Taliban.
Visiting Islamabad about a week after the White House asked Pakistan to immediately charge and arrest 26/11 Mumbai attack mastermind Saeed, in the absence of which it warned of repercussions, Mattis said he needs to talk to leaders in the country and gain their understanding.
More From This Section
JuD chief Saeed, who has an American bounty of USD 10 million on his head, walked free on November 24 after a court ordered an end to his 10-month detention.
The Pakistan government decided against detaining him further in any other case, pushing back against India's efforts to bring the perpetrators of the 2008 Mumbai terror attack to justice.
Mattis is the first top American official to visit Pakistan after the completion of the first 100 days of the US' new South Asia strategy unveiled by Trump on August 21.
"We know we have some common ground. They have lost hundreds, thousands of their troops killed and wounded by terrorists. They have lost hundreds, thousands of their innocent people murdered and wounded by terrorists, so we know that there is common ground," he said, ahead of his visit.
"So we know there's common ground; it's how much more common ground can we find by listening to one another without being combative with one another, listening to others' perspective," Mattis said.
"But at the same time, as the General (Bajwa) has said, he wants no havens for terrorists anywhere, so we will work together and we'll find that common ground, if we have the will to. And then we'll work on how we address the problems where we can work together," he said.
"There is clearly an abundance of areas where we have to double down, and I am optimistic at this point that because of what our adversaries, our mutual enemies are doing, that we can find ways to work together," he said.
Mattis said both Pakistan and Afghanistan, the Pakistan military, the NATO-led militaries of 39 nations have all been in fights here. "We've got to find a way to work together".
He said there is "always an urgency to something" when 39 nations plus Afghanistan have their troops in the midst of a long war, where causalities are being taken.
Mattis said that it was essential in South Asia that all countries work together to restore the level of stability that allows for the economy to build.
"I mean, can you imagine a border between Pakistan and India where economic trade is not ongoing to the benefit of people on both sides of that border. We can not only imagine it, it's a reality right now," Mattis said.
"So we have got to work together on this, and there's a sense of urgency about that," he said.
"General Bajwa is a mature officer with a deep background in not just military matters, but South Asia security issues, and we have got to find a way to move beyond this constant fighting that's characterised the area," he said.
Media reports suggested that Mattis faces a difficult task ahead as Pakistan is not happy over repeated US allegations of safe havens for militants in the country.