The bilateral meeting took place shortly after Modi arrived here on a five-day visit to Australia to attend the G20 Summit beginning tomorrow.
At their meeting, Cameron told Modi that "relations with India are at the top of the priorities of the UK's foreign policy."
The British Premier also invited Modi to visit the UK to which the Indian Prime Minister said he would do so at the earliest, the Ministry of External Affairs spokesman Syed Akbaruddin said.
The UK Premier was one of the first world leaders to congratulate Modi following his victory in the general elections in May.
Earlier, Cameron's spokesperson had said that the Brisbane meeting was set to strengthen the broad and deep relationship that UK and India have, building on a partnership of equals based on mutual respect.
The Indian community in the UK is keen to have a visit by Prime Minister Modi to inaugurate Mahatma Gandhi's statue which is being raised outside the Palace of Westminster in London's Parliament Square. The statue will stand amid monuments to other statesmen, including Nelson Mandela, Winston Churchill and Abraham Lincoln.