Boris Johnson’s lurch to the right after Partygate is fuelling even more anger among rebel Tory MPs, with momentum now building for a leadership challenge next week.

Conservative whips spent the first day of recess anxiously phoning round the parliamentary party to shore up support for the prime minister, as four more MPs called on him to resign, including Jeremy Wright, the former attorney general.

Several Tory MPs told the Guardian they believed the threshold of 54 letters withdrawing support for Johnson was close to being crossed – or may have been already. This would trigger a secret ballot on whether they still have confidence in the prime minister.

It is understood that Sir Graham Brady, the chair of the 1922 Committee, will have to use his own judgment about whether to announce the milestone being passed straight away if it occurs while parliament is off this week, or wait until Monday, when the House of Commons returns after the Queen’s jubilee celebrations.

One backbench critic of the PM said MPs from the 2019 intake were “gathering their courage” to put in letters before next Monday, but were worrying about repercussions if No 10 were to identify them after an unsuccessful coup. They said the opposition to Johnson was increasingly coordinated and determined to trigger a vote, with almost 30 MPs having publicly declared their opposition so far.

In his statement withdrawing support for the prime minister, Wright said Johnson had done “real and lasting damage” to the institution of government, and while he could not be sure that the prime minister had misled parliament, Johnson had been at best “negligent” in how he had approached the issue.

Elliot Colburn, a Tory MP with a small majority against the Lib Dems, said he had put in a letter “some time ago”, while Nickie Aiken, the Cities of London and Westminster MP whose council turned Labour this month, called on Johnson to bring an end to the situation by submitting a no-confidence letter in himself. Tory MP Andrew Bridgen also told constituents he had resubmitted his letter.

The dismay over Johnson’s premiership is worsening among Tories in so-called “Blue Wall” seats at risk of losing them to the Lib Dems, and “Red Wall” marginals where they have narrow majorities over Labour.

With Johnson’s future in the balance, No 10 has begun launching a number of rightwing, nationalistic policies in recent weeks. These include the return of imperial measures, plans to override the Northern Ireland protocol, a hint about expanding grammar schools, a review of fracking, and repeated promises to tear up more EU regulation.

A cabinet minister told the Guardian that Johnson appeared to be trying to stop the right of the party turning against him in the event of a leadership challenge, citing policies such as the review of fracking – which is electorally unpopular but appeals to a minority in parliament.

But Tory pollsters and some centrist MPs warned that this “core vote” direction was the wrong route to go down with public trust in Johnson so low among swing voters. Tobias Ellwood, a Tory former minister and chair of the defence committee, warned: “We will lose the next election on the current trajectory as reflected in recent elections.

“There is not only just a concern on the conduct of behaviour in No 10, because that has breached the trust with the British people, it is now concerns about No 10 thinking what our policies are.”

On the weights and measures policy, he told Sky News: “There will be some people in our party which will like this nostalgic policy in the hope that it’s enough to win the next election. But this is not the case. This is not one-nation Conservative thinking that is required to appeal beyond our base.”

One Tory cabinet source said the imperial measures policy was “absolutely bananas”, while another cabinet source said they had “no idea which muppet had come up with that idea”, as “this is not what the government’s overall strategy is about”.

Another Conservative MP said he represented a seat in the “heart of middle England” and about half of the core Conservative voters there had lost faith in the prime minister.

Some local government leaders also expressed a lack of confidence in Johnson. Rishi Sunak’s local council leader, Carl Les, the Conservative leader of North Yorkshire county council, said he thought it was time for a leadership election, blaming Johnson for heavy losses in the local elections.

“I am very disappointed that the strong majority we had in North Yorkshire has diminished down to a working majority, but only just, and a lot of the comment we were getting on the doorstep was about the impact of Partygate,” said Les.

The warnings from MPs and councillors were echoed by pollsters and political strategists, including former No 10 advisers James Johnson and Will Tanner. Both said Johnson was on course to lose the election by swinging to the right instead of focusing on delivering goals on schools, hospitals, housing and the cost of living.

Tanner, a former No 10 aide and the director of Conservative thinktank Onward, said: “My view is that while it’s understandable that the prime minister and Downing Street would want to demonstrate their commitment to rightwing policy issues, to satisfy some of his backbenchers at a moment where clearly the prime minister is worried about his future, those issues are not going to win the Conservative party the next election.”

He said he had “never sat in a focus group or conducted a poll where issues like imperial weights and measures or Channel 4 privatisation has come up repeatedly” from the voters Johnson is seeking to court.

He added: “It is NHS, immigration, crime, wages, good jobs in my town. Those are the fundamentals that the Conservative party needs to be focusing on, not these quite small and niche issues, which only matter to a few people.”

James Johnson, a pollster at JL Partners, who worked for Theresa May, said: “Some of these things that might have raised a smile in the past will actually invite ridicule, the pounds and ounces thing being a good example of that. We’re approaching the situation with Johnson similar to one we faced with Corbyn, where the individual policies might be popular, but the brand attached to them is toxic.”