Historian and TV presenter David Olusoga has said that rightwing newspapers characterise him as an activist and critical race theorist to “delegitimise” his voice, despite there being no basis for these claims.

Olusoga, whose work has explored black Britishness and the legacy of empire and slavery, said that people “feel perfectly comfortable making these comments about me without being able to point to a single reference or footnote in my books”. He said that in reality he is “an old-fashioned empirical historian who fundamentally tells stories and tries to create empathy and a public understanding of history”.

He told an audience at Hay festival: “Why the need to describe me as a critical race theorist? Why the need to describe me as an activist rather than a historian? These are all about delegitimising people’s voices.”

Olusoga was speaking as part of a debate on “how cancel culture has become a blood sport”, but said that the phrase did not capture his experience, since it is usually attributed to students, who he thinks are falsely accused of fomenting “cancel culture”, when in reality it reflects “a growing intolerance” in rightwing newspapers.

He was also asked for his views on the response to historian David Starkey’s comments that slavery was not genocide. Starkey subsequently resigned from his post at the University of Cambridge’s Fitzwilliam College.

Olusoga said he was “conflicted” because while what Starkey had said was “appalling” and “inaccurate”, he felt “it’s sad that somebody who is a great historian was getting into those debates”.

He blamed Starkey’s tone on the influence of the Moral Maze, BBC Radio 4’s provocative show that has run since 1990, for “elevating opinion over expertise”. “It’s taken some who have great expertise away from that expertise and into that carnivalesque world of commentary,” he said.

The panel was hosted by the editor of Prospect magazine, Alan Rusbridger. The former Guardian editor said he had observed during his six years as principal of Oxford University’s Lady Margaret Hall how the increasingly polarised and vitriolic tone of public debate had resulted in a “growing fear” among academics that “one misplaced word or tweet” would lead to a “pile-on”.

Prof Jonathan Bate, who was also on the panel, said that how controversial issues are discussed varied in different countries. While in the UK there is “a great deal of heat” around trans issues, this is not so fiercely debated in the US, where he teaches at Arizona State University. “The hot button in the States, for entirely understandable reasons, is race,” he said.

Speaking generally, he worried that “history goes in cycles and an age of liberation is followed by an age of puritanism” and that the current mood of polarisation was “bringing us into a new puritanism”.

Fellow panellist and comedian Shazia Mirza said that when she started out in comedy 15 years ago, death threats were “considered serious” but that social media had made them so commonplace that they were considered “a badge of honour” among comedians.

“If you get a death threat or hate mail it’s considered that you’re doing well, it means you’re making people react, people are talking about you; you’re having an effect on people,” she said.

She added that although people say that comedians such as Jimmy Carr, Louis CK and Ricky Gervais have been cancelled, in reality “it’s a myth”. “They are definitely not unemployed, they are still working, earning millions.”