The two stars collaborated for the 1995 drama "Restoration", which also starred Sam Neill, David Thewlis and Meg Ryan.
"He hated me. He took one look at me and wanted to kill me," Grant told People magazine.
However, the "Paddington 2" actor sounded positive about his other co-actors such as Emma Thompson, whom he called a "genius".
"She is not remotely sane. She's nuttier and nuttier as the years go," Grant said.