Boris Johnson is under growing pressure to apologise for remarks falsely linking Keir Starmer to the failure to prosecute Jimmy Savile after a senior cabinet minister and the Commons Speaker urged him to draw a line under the matter.

No 10 made clear the prime minister would not apologise for the remarks, which provoked renewed anger when protesters shouted abuse at the Labour leader on Monday. Surrounded near parliament, Starmer was branded a “paedophile protector” and one demonstrator carried a noose. He and his Labour colleague David Lammy were bundled into a police car.

Eleven Tory MPs have called for Johnson to retract and apologise for the claims he made in the Commons last week.

Michael Gove, the levelling-up secretary, did not defend the comments when asked about them on Tuesday. “It’s a uniquely sensitive issue and I think the sooner the debate moves on, the better,” he told reporters at Convention of the North in Liverpool.

The Commons Speaker, Sir Lindsay Hoyle, condemned the abuse of Starmer and the shadow home secretary and linked it to the prime minister’s comments last week, which he called “inappropriate”.

Hoyle told MPs: “Those sorts of comments only inflame opinions and generate disregard for the house and it is not acceptable. Our words have consequences. And we should always be mindful of that fact.”

Johnson’s official spokesperson said the prime minister had already “clarified” his remarks to make clear Starmer was not personally responsible for the Savile case because the comments were “capable of being misconstrued by a tiny minority”.

Asked whether that “tiny minority” included the crowd that surrounded Starmer, he said: “I’m not seeking to link anything. I wouldn’t seek to look into the motivations of individuals.”

Chris Philp, the digital minister, had earlier said the comment that Starmer had failed to prosecute the child sex offender was “not incorrect” but said Johnson had clarified he did not mean Starmer took the decision when he led the Crown Prosecution Service (CPS).

“The first comments in the house on the previous Monday were capable of being misconstrued and that is why it is important and right that a couple of days later that Boris Johnson … did clarify that he was not suggesting at any time that Keir Starmer had personal responsibility for the case,” Philp told BBC Breakfast. “But he obviously did have responsibility for the conduct of the CPS.

“I don’t think there is any way you can reasonably suggest that the comments on Keir Starmer’s overall responsibility for the CPS in any way provoked the very unseemly and totally unacceptable harassment we saw last night.”

A No 10 source said the prime minister did not intend to go further than his condemnation of the abuse of Starmer.

Keir Starmer is escorted to a police car after being heckled by protesters.

Johnson’s initial comments did not suggest Starmer had taken institutional responsibility but said as director of public prosecutions he had “spent more time prosecuting journalists and failing to prosecute Jimmy Savile”.

Tories including the former chief whip Julian Smith and the select committee chairs Simon Hoare and William Wragg suggested Johnson should apologise. Others who also voiced concern at the consequences of the prime minister’s rhetoric included the former cabinet ministers David Davis and Caroline Nokes and the ex-ministers Stephen Hammond and Tobias Ellwood, as well as three MPs elected in 2019, Anthony Mangnall, Aaron Bell and Rob Largan.

The husband of Jo Cox, the Labour MP murdered in 2016 by a far-right extremist, also warned about the consequences of politicians lending credence to far-right conspiracy theories.

Brendan Cox said the prime minister’s choice to “inject poison into politics” with his comment had “unintended consequences”.

“If it was a one-off, I think we could be more sanguine about it,” he told BBC Radio 4’s Today programme. “I think the people that are directly responsible for what happened yesterday were the people that did it.

“However, it’s also true that if you inject poison into politics that has a whole set of unintended consequences that people will react to in different ways and at times that can lead over into intimidation, it can lead over to violence, it can lead over into extremism.”