Senior Tories have warned that traditional supporters are abandoning them after Boris Johnson’s Partygate fine, as another MP broke cover to say the prime minister should be removed over his conduct.

Conservative MPs across the country said on Saturday they believed many people who had backed the party before were now raising concerns, with Downing Street braced for further fixed-penalty notices relating to parties in the coming days.

Writing in the Observer, former immigration minister Caroline Nokes said she was sticking with her decision to submit a letter of no confidence in the prime minister.

It makes her the latest MP to back a leadership contest since the lockdown breach fines were issued to the prime minister and the chancellor, Rishi Sunak.

She also highlighted correspondence suggesting previously secure voters were expressing concerns. She wrote: “There are those who say these emails are only from ‘the usual suspects’. It is true to say there have been a smattering of them, the political activists who send an automated email at the drop of a hat. But they are very much in the minority.

“The bulk of the emails I received last week are from people who are genuinely distressed at the family events they could not and did not attend, and many are from people I know who have long been Conservative supporters.

“I have not withdrawn the letter of no confidence in Boris Johnson that I wrote months ago to Sir Graham Brady, the chair of the [Tory backbench] 1922 Committee, because to do so would be letting down all of those people who spent the pandemic doing the right thing.”

Other Tory MPs, including serving ministers, said they believed some sections of their voters were being alienated. A number of them said that the party was trying so hard to hold on to newer, pro-Brexit voters that many traditional supporters were being put off, by Partygate and by the government’s plan to send asylum seekers to Rwanda.

“If you go to our newer working-class, blue-collar voters, I’m not sure they’re that bothered,” said one northern Tory MP. “But if you go to our traditional middle-class Tory voter, they’re angry. That’s how I would define it. I think he should’ve resigned.”

A minister said: “We’re going down a route that isolates the people in the middle. I don’t know whether there’s enough votes on the right and the core of the party to get us through. I just think it’s offensive and it’s doing real brand damage. I’m just appalled.”

One former minister said: “There’s a couple of what I call ‘barometer people’ have come to me and said: ‘Look, we’re still angry with him. We don’t think he should resign at the moment, because we’ve got the Ukraine crisis, but he shouldn’t lead us into the next election.’ These are people who I know are Conservative supporters.”

The news comes with Tory MPs demanding further contrition from Johnson when he meets them after parliament returns in the coming week. He has pledged to give a fuller explanation for his earlier denial that parties had taken place. There are also likely to be demands this week for a vote on referring the prime minister to the cross-party privileges committee over whether he misled MPs about Downing Street lockdown parties.

Labour has called Johnson’s involvement in the Partygate saga “indefensible” after further allegations emerged about the prime minister’s conduct.

Johnson’s official photographer captured photographs of the prime minister holding a beer at his birthday gathering in June 2020 and Sunak with a soft drink, the Sunday Times reported.

The Sunday Mirror said No 10 had refused to answer questions about the birthday gathering submitted through a freedom of information request because of national security reasons.

Labour’s deputy leader, Angela Rayner, said: “While the British public was making huge sacrifices, Boris Johnson was breaking the law.

“He has deliberately misled the British people at every turn.

“The prime minister has demeaned his office. The British people deserve better.”

Some MPs critical of Johnson believe they have only a few months to decide whether he is the right person to lead them into the next election. “As a party, we have to make a decision about Boris by summer recess,” said an MP. “If we haven’t done it by then, Boris will lead us into the next general election.”

The Labour leader, Keir Starmer, called on Tory MPs to remove Johnson. “He is unfit for office and every day he remains in Downing Street degrades his office further,” he said. “The cabinet, ministers and Tory MPs must realise this is not going to change and every time they defend Boris Johnson, they embroil themselves in his web of lies.

“Are they really prepared to stand by and do nothing as his behaviour brings their party into disrepute? Only Conservative MPs have the power to bring this shameful saga to a close. If the prime minister is not going to resign, then they must take action when parliament returns this week to send a clear message that honesty and integrity in public life still matters.”

A new Opinium poll for the Observer suggests that the fines for Johnson and Sunak have had a more immediate impact on the chancellor’s popularity, which has hit a record low. The proportion of voters approving of the chancellor was 24%, with 49% disapproving. His net approval rating of -25 is his lowest ever. Johnson remains on a net approval rating of -26.

The Metropolitan police are also coming under growing pressure to explain their approach and the timing of their decision to fine senior politicians for breaches of lockdown regulations. Unmesh Desai, a Labour member of the Greater London authority’s police and crime committee, said he and colleagues were concerned that the Met had chosen to reveal last week’s politically explosive Partygate findings at a time parliament was not sitting.

Desai, a former chair of the committee that examines the work of the Mayor’s Office for Policing and Crime, which in turn oversees the Met, said: “The timing and the manner of the announcement raise more questions than answers.”

Desai also wanted to know why a drip-feed policy on the force’s fining decision had been adopted by the Met, a strategy he said had no precedent. He said the committee would be asking Sir Stephen House – the acting head of the Met until a new commissioner is appointed – why the force had decided to release findings of the investigation in a piecemeal way.