Boris Johnson has plunged the Conservative party into an acute identity crisis as a result of Partygate and U-turns over tax policy, senior Tories warned last night, as more MPs called for him to resign as prime minister.

Former Tory leader Iain Duncan-Smith told the Observer his party had been left with an “enormous identity problem” because it had raised taxes instead of cutting them under Johnson and the chancellor, Rishi Sunak, undermining a core Tory message that had helped win it successive general elections.

Doubts about Johnson’s ability to survive as prime minister, and fears that the Tories are now on a terminal electoral slide, grew as party grandee Sir Bob Neill said the combination of collapsing trust in the prime minister, coupled with a lack of clarity over what the Conservatives stood for, had created a “toxic mix” in the minds of voters. Neill called on Friday for Johnson to go.

Yesterday two more MPs – former health minister Steve Brine and Anne Marie Morris – said they had also lost confidence in Johnson, taking the total who have gone public to 24.

Several others are understood to have written letters of no confidence to 1922 chair Sir Graham Brady without going public, and more have privately indicated they will do so. If 54 or more MPs submit letters to Brady, a leadership contest will be triggered.

“I reckon we are in the 40s,” said one rebel MP. Another added: “We are just a few off.”

On Thursday and Friday there was a mood of relief bordering on jubilation in Downing Street as Johnson and his team believed they had survived the publication of a damning report on Partygate by the senior civil servant Sue Gray, and then switched focus on to measures to address the cost of living crisis on Thursday.

While a £15bn package of measures announced by Sunak was broadly welcomed, many Tories were unhappy that the government funded part of it by performing a massive U-turn which saw it adopt Labour’s plan for a windfall tax on the profits of oil and gas companies.

Duncan Smith said that instead of imposing a tax rise on business, Sunak should have cut taxes for middle and lower earners, and restored the Tory reputation as a low tax party.

“He had the headroom not to do a windfall tax. Getting growth going is the priority. We have to avoid recession and to do that we have to lower taxes and get people spending.

“Conservatives believe in lower taxes, leaving people with more of their own money because they make the best choices. Unless we get back to this approach and cut taxes in the autumn budget we are in real trouble.”

The Tories, he said, now risked earning a reputation for being a high tax, big state party, having also raised national insurance in breach of a manifesto commitment. “I don’t just think we have a bit of an identity problem, we have a massive identity problem,” he said.

Gavin Barwell, the former Tory MP who served as Theresa May’s chief of staff in No 10, said the party’s electoral prospects under Johnson were grim: “By failing to take action against a prime minister approximately 60% of the public think should resign, Conservative MPs are sleepwalking to defeat at the next election.”

The latest Opinium poll for the Observer finds that 56% of all voters, including 32% of 2019 Tory voters, believe Johnson should resign. The poll was conducted after publication of the Gray report.

Oliver Heald, the Tory MP for North East Hertfordshire, told constituents that he was talking with fellow MPs about “what happens next”. He added: “I think it will take two to three weeks to know the answer.” Key byelections in Wakefield and in Tiverton and Honiton take place in three weeks. Former cabinet minister Robert Buckland said on Saturday that “changes will have to be made” if the party suffers heavy defeats.

The Liberal Democrats say many of those calling for Johnson to resign are in Tory seats they are targeting. “Conservative MPs who spent months defending Boris Johnson over Partygate are now finally acting because they’re worried about losing their seats,” said deputy leader Daisy Cooper.

“This byelection [in Tiverton and Honiton[ is a chance to send a clear message that people won’t stand for Johnson’s lies and lawbreaking and that he has to go. If we can overturn the huge Tory majority, the trickle of letters of no confidence in the prime minister could turn into a flood.”

Johnson caused further astonishment on Friday, just two days after declaring himself to have been “humbled” by the Gray report, when it was announced that he had rewritten parts of the ministerial code so that ministers no longer have to resign for minor breaches. He also removed a section of the code which referred to ethics. Labour on Saturday announced it would hold an opposition day debate in which the party would pledge to strengthen the code.

In a further potential blow to Sunak, doubts are being raised this weekend about whether the windfall tax will raise anywhere near the £5bn the chancellor claimed it would on Thursday.

The left of centre Common Wealth thinktank said he may only raise a fraction of that sum because of complex rules which allow investments to be offset against profits.

The warning comes after the Liberal democrats said delays to the tax’s introduction meant he had missed out on £3bn from the “extraordinary profits” reported by oil and gas firms in 2021 and another £8bn so far this year.