Anger among Tory backbenchers is spreading amid fallout from the Sue Gray report, with one minister warning Boris Johnson is in “yellow card territory”.

Speaking on BBC Radio 4’s Any Questions, the Treasury minister John Glen said he had had a “very frank and very honest” meeting with Johnson to express his and his constituents’ misgivings.

“I think we are in yellow card territory but as a member of the government I went to see him and let him know how I felt and my constituents felt, but he asked me to get on with the job of driving reforms in financial services,” he said on Friday night.

The former cabinet minister David Davis said unease was spreading across Conservative ranks as MPs feared the Downing Street lockdown parties scandal could cost them their seats.

“Nobody in the world could have made it plainer, I don’t think, that I want the prime minister to go – I haven’t changed my mind about that,” he told BBC Radio 4’s Today programme.

Asked whether discontent was spreading in the Tory party, Davis said: “There is no doubt about that, for two reasons.

“Number one, frankly they see their own seats disappearing in many cases, they see themselves losing the next election on the back of this.

“Also, it has a bad effect on the country … it is a distraction on everything you do and it doesn’t help the reputation of the country.”

The former Brexit secretary said party leadership trouble traditionally took a long time to be sorted out, pointing to the length of time John Major and Theresa May stayed in No 10 despite experiencing backbench revolt.

He added: “I fear we’ll not resolve this until the latter part of the year.”

The warning by Davis of increasing mutiny came after polling company YouGov produced new modelling suggesting the Conservatives would lose all but three of 88 “battleground” constituencies if a general election were held on Saturday, putting the government’s Commons majority in jeopardy.

Under the predicted outcome, Johnson’s Uxbridge and South Ruislip seat would “likely fall” into Labour hands and “red wall” seats such as Blyth Valley and Stoke-on-Trent North would revert back to Labour, according to YouGov.

In order for a vote of no confidence to be triggered, the chair of the 1922 Committee, Graham Brady, must receive letters from at least 54 Conservative MPs – 15% of the parliamentary party.

Former health minister Steve Brine has added his name to the list of Tory MPs to have handed in letters of no confidence, it has emerged.

Brine said the Gray report had not altered his view that it was “inevitable the prime minister would face a vote of confidence. All I can do as a backbencher is seek to trigger that process and (some time ago actually) I have done that”, he said in a statement published on his website on Wednesday but only picked up by the media on Saturday.

“I have said throughout this sorry saga I cannot and will not defend the indefensible. Rule-makers cannot be law-breakers.”

Newton Abbot MP Anne Marie Morris is also among the letter writers, having confirmed that she has had the Tory whip restored after it was removed in January for her decision to support an opposition move to cut VAT on energy bills.

The veteran Conservative Sir Bob Neill, a qualified barrister and chair of the Commons justice committee, confirmed he had submitted a letter of no confidence on Friday.

And also on Friday, the Conservative MP for Rutland and Melton Alicia Kearns , a previous critic of Johnson, released a statement saying she still had no confidence in him.

Her constituency, which includes Melton Mowbray, sparked talk of a “pork pie putsch” when she and other Tory MPs discussed trying to oust the PM in January.

In a stinging Facebook post, Kearns said calls to move on “treat with contempt and disregard” the sacrifices other people made. She wrote: “I can only conclude that the prime minister’s account of events to parliament was misleading.”

Johnson announced changes to the ministerial code on Friday in a move his rivals said watered down punishments for ministers.

An update said ministers would not automatically lose their jobs if theybreached the standards code, and can instead apologise or possibly have their salary suspended instead.

Chris Bryant, the chair of the Commons standards committee, said the “loosening” of the ministerial code by Johnson was “bizarre” and showed why there should be an independent system in place for judging the conduct of ministers.

The Labour MP told BBC Radio 4’s Today programme he disagreed with recommendations made by the independent committee on standards in public life, which allow ministers to remain in their jobs for what could be deemed minor breaches of the code.

“Maybe this is what you would expect from people who have mostly been civil servants in the past – that’s how they end up on the committee on standards in public life – that they would support a strong government that is, broadly speaking, able to do what it likes,” he said.

He called for a “proper system whereby an independent figure, entirely without the prime minister’s involvement, decides whether or not to launch an investigation into a minister, and decides whether it is a very serious case or a less serious case, and then suggests the sanction”.

Bryant added: “That’s not what the prime minister has got, it still all lies in the prime minister’s hands and we know, don’t we, that the prime minister always finds himself innocent in the court of his own opinion.”