Boris Johnson has again sought to take credit for the departure of Matt Hancock, despite having backed the health secretary before he resigned, and prompted an angry response by dismissing the story as one only of interest to the “Westminster bubble”.

In a noisy prime minister’s questions, Keir Starmer called the prime minister’s narrative “ridiculous”, asking why Johnson had defended Hancock after the emergence of photos showing him in an embrace with his colleague and long-term friend Gina Coladangelo.

Johnson tried to claim credit for Hancock’s departure on Saturday, even though the health secretary resigned.

“I read the story in common with you and everybody else on Friday, and we had a new health secretary in place by Saturday,” Johnson told Starmer. “Which I think, given we have a pandemic, to move with that speed was fast.”

Starmer replied: “What a ridiculous answer. The prime minister must have been the only person in the country who looked at that photo on Friday morning and thought that the health secretary shouldn’t be sacked immediately.”

Pressed by the Labour leader on why he did not sack Hancock, and the potential conflict of interest of Coladangelo being made a non-executive director at Hancock’s department, Johnson accused Starmer of “focusing on stuff going on inside the Westminster bubble”.

Starmer stressed what he said was public anger about Hancock breaching Covid rules, contrasting this with the anguish of the friends and family of a 27-year-old man who died in early May.

Starmer said the family of Ollie Bibby from Essex, who died in hospital from leukaemia the day before Hancock was pictured with Coladangelo, were “livid” at barely being allowed to see him due to Covid rules, given Hancock’s actions.

“I know the prime minister is keen to sweep this under the carpet, but let me tell the prime minister why this matters,” Starmer said, saying he had spoken to Bibby’s mother before PMQs.

“She told me, prime minister, that she watched the press conferences every day they were on, and she hung on to every word the government ministers said, so she would know what her family could and couldn’t do. And then they followed the rules,” Starmer said.

“This is not the Westminster bubble. She told me that for her and her family this case isn’t closed. And she speaks for millions of people.”

Johnson declined to answer Starmer’s questions, or to withdraw the words about the “Westminster bubble”. Instead he repeatedly intimated he had played a role in Hancock’s departure.

Johnson said: “We all share the grief and the pain of Olly and his family, and millions of people up and down the country. And that is why we had a change of health secretary the day after the story appeared.”

Starmer criticised what he called the “pattern” of Johnson backing ministers and others who had broken rules, also citing the PM’s former chief adviser Dominic Cummings, Robert Jenrick, the communities secretary, and the home secretary, Priti Patel.

He said: “Every time it’s the same old story. Isn’t it the case that while the British people are doing everything asked of them, it’s one rule for them and another rule for everybody else?”