Four of Boris Johnson’s key staff have quit as the fallout from the Downing Street party scandal continued to shake his hold on government.

Johnson’s longstanding policy chief Munira Mirza was the first to go, using a stinging resignation letter to accuse the prime minister of “scurrilous” behaviour when he falsely linked Keir Starmer to the failure to bring paedophile Jimmy Savile to justice.

Within hours, three more of Johnson’s staff had gone – Dan Rosenfield, his chief of staff, Martin Reynolds, his principal private secretary, and his director of communications, Jack Doyle.

All three have been implicated in Downing Street’s response to the lockdown parties that are now being investigated by the Metropolitan police.

Reynolds sent a widely shared email urging staff to “bring your own booze” to one event on 20 May 2020.

While Johnson was expected to clear out many of his top team in the wake of Sue Gray’s report into the lockdown-breaking gatherings published earlier this week, Mirza’s resignation came out of the blue.

And in a sign of Johnson’s waning authority, this also provoked the chancellor to issue a rare rebuke, as he also criticised the prime minister for his jibe against the Labour leader.

Asked about the comments at a press conference on Thursday, Sunak said: “Being honest, I wouldn’t have said it.”

Pressed repeatedly on whether Johnson should apologise, he said that was a matter for the prime minister.

Sunak has previously declined to criticise Johnson directly throughout the weeks of revelations about lockdown-busting parties in Downing Street, though he has acknowledged that mistakes were made.

The flurry of resignations came at the end of an important day for the government as the chancellor sought to explain how he hopes to avert a cost of living crisis for millions of people affected by rising fuel bills and mortgage payments.

But the focus once again turned to the turmoil in Downing Street when Mirza resigned.

Johnson has repeatedly refused to apologise, after saying in a rowdy House of Commons session on Monday that, as director of public prosecutions from 2008 to 2013, Starmer had “spent most of his time prosecuting journalists and failing to prosecute Jimmy Savile”.

In fact, Starmer took no personal role in decisions connected to Savile’s case.

In a strongly worded resignation letter, Mirza told Johnson: “There was no fair or reasonable basis for that assertion. This was not the usual cut and thrust of politics; it was an inappropriate and partisan reference to a horrendous case of child sexual abuse.”

Mirza’s departure was regarded at Westminster as a serious blow to Johnson, who was under pressure to overhaul his Downing Street operation after Gray identified failures of leadership and judgment.

It was swiftly followed later on Thursday by the resignation of Doyle – though he insisted his departure was unconnected with Mirza’s. A No 10 spokesperson said Doyle had “made a huge contribution and the prime minister is immensely grateful for the work he has done”.

Downing Street later confirmed the departures of Rosenfield and Reynolds in a brief statement.

“Dan Rosenfield offered his resignation to the prime minister earlier today, which has been accepted,” it said.

“Martin Reynolds also informed the prime minister of his intention to stand down from his role as principal private secretary and the prime minister has agreed to this.

“He has thanked them both for their significant contribution to government and No 10, including work on the pandemic response and economic recovery.

“They will continue in their roles while successors are appointed, and recruitment for both posts is under way.”

The prime minister had earlier sought to row back from the Savile claims on Thursday, saying “a lot of people have got very hot under the collar”.

“Let’s be absolutely clear, I’m talking not about the leader of the opposition’s personal record when he was director of public prosecutions and I totally understand that he had nothing to do personally with those decisions,” Johnson said on a visit to Blackpool.

“I was making a point about his responsibility for the organisation as a whole. I really do want to clarify that because it is important.”

That was not enough to placate Mirza, however. In her resignation letter, she told him: “You tried to clarify your position today but, despite my urging, you did not apologise for the misleading impression you gave.”

She said Johnson was “a better man than many of your detractors will ever understand”, adding that it was “so desperately sad that you let yourself down by making a scurrilous accusation against the leader of the opposition”.

A spokesperson said: “We are very sorry Munira has left No 10 and are grateful for her service and contribution to government.”

Munira Mirza’s resignation letter.

Johnson moved quickly to replace Mirza, promoting Andrew Griffith to head up the No 10 policy unit as a minister in the Cabinet Office.

A wealthy former Sky executive, Griffith lent Johnson his £9.5m townhouse as a campaign base during his 2019 leadership bid. Now the MP for Arundel and South Downs, he has been serving as Johnson’s parliamentary private secretary – liaising with MPs.

Mirza, often seen as a proponent of Downing Street’s “war on woke”, had worked with Johnson for more than a decade, including at City Hall when he was mayor of London. He previously identified her as one of the five most inspiring women in his life. Her partner, Dougie Smith, is also a senior Tory adviser.

Johnson’s former adviser Dominic Cummings claimed Mirza’s departure was “an unmistakeable signal the bunker is collapsing and this PM is finished”. He urged ministers to show a similar “flicker of moral courage” and resign.

In another sign of the pressures on Johnson from within his own party, West Midlands mayor Andy Street, one of the most prominent Tories outside Westminster, said he was unsure whether he could continue to support the prime minister.

“What he’s done, very clearly, is wrong. It’s bad by any measure,” Street told BirminghamLive. “He has apologised. He appears to me to be a man who is genuine in that apology. You can see that from his demeanour. But what we’ve now got to see is that this apology follows through in terms of actions, how he leads and what is done within Downing Street to make it very, very different.

“The honest answer is that we are waiting to see if he follows through on what he’s now said.”

Three Conservative MPs went public on Wednesday with their intention to submit letters of no confidence in the prime minister. One senior Tory MP said the departure of Mirza smacked of “the last days of Rome,” suggesting the number of no confidence letters may now be approaching the threshold of 54 that would trigger a no confidence vote.

If Johnson lost such a vote, which could be held within days, his premiership would be over.

The Savile/Starmer smear has circulated on far-right blogs, and at prime minister’s questions on Wednesday, Starmer accused Johnson of “parroting the conspiracy theories of violent fascists” for political gain.

Several cabinet ministers, including Nadine Dorries and Michael Gove, have since defended Johnson, however, with the justice secretary, Dominic Raab, calling it part of the “cut and thrust” of parliamentary debate.

Before prime minister’s questions on Wednesday, the Speaker, Lindsay Hoyle, reminded MPs of the importance of being truthful in the House of Commons.