Conservative MPs are increasingly worried about Boris Johnson’s competence and drive after he gave a rambling speech to business leaders and was accused of losing his grip over a series of key policies from social care to rail.

Senior members of his own party said they needed Johnson to get the government back on track after a disastrous two weeks amid dismay about his performance at the Confederation of British Industry (CBI) conference, where he lost his place in his speech for about 20 seconds and diverted into a lengthy tangent about Peppa Pig.

The prime minister was also facing a substantial rebellion over his social care proposals, anger at the decision to scale back rail improvements for the north and frustration over the government’s failure to keep its promises on small boats crossing the Channel. It caps a difficult fortnight for the prime minister after he admitted he “crashed the car into a ditch” in his handling of the Owen Paterson lobbying scandal.

Nervousness among Tory MPs about No 10 intensified after one Downing Street source told the BBC there was “a lot of concern inside the building about the PM … it’s just not working”, adding that the “cabinet needs to wake up and demand serious changes otherwise it’ll keep getting worse”.

A former cabinet minister also told the Guardian that there was “an accumulation of things building up, really relating to his competence and that is beginning to look very shaky” after a “pretty bad bloody fortnight”. He said it was unlikely to result in a leadership challenge while the polls were still fairly even between the Tories and Labour, but it could be “problematic for him” if that changes when an election is looming.

Another senior backbencher said Johnson’s CBI speech had been a “mess” while a third Tory MP said: “I thought today’s performance was the most embarrassing by a Conservative prime minister since last week’s PMQs. Someone needs to get a grip. He is losing the confidence of the party.”

Another Tory MP referenced the process by which MPs can submit letters of no confidence to the chair of the 1922 Committee, Graham Brady, saying: “It might not only be Father Christmas’ postbag filling up towards the end of the year – Sir Graham Brady could find he needs a bigger one too.”

The prime minister’s speech on Monday met a frosty reception among business leaders over its lack of substantial policy. In a sprawling address, Johnson lost his place and spent 20 seconds uttering “forgive me” three times as he shuffled the printed pages on his podium.

He then revealed he had spent Sunday on a trip to Hampshire’s Peppa Pig World – an amusement park dedicated to a children’s cartoon character – using it for a series of digs at the BBC and civil servants. He said the TV show, worth £6bn, “was rejected by the BBC and has now been exported to 180 countries”. He added: “I think that is pure genius, don’t you? No government in the world, no Whitehall civil servant, would conceivably have come up with Peppa.”

Johnson also imitated the sound of an accelerating car with grunts that the official Downing Street release transcribed as “arum arum aaaaaaaaag”. At another point, he compared himself to Moses over his plan to help business invest in tackling climate change. The prime minister said: “I said to my officials the new Ten Commandments were that ‘thou shalt develop industries like offshore wind, hydrogen, nuclear power and carbon capture’.”

After the speech, business leaders questioned the prime minister’s focus. Mike Cherry, national chair at the Federation of Small Businesses, said: “Last week’s HS2 announcement has left a lot of small business owners wondering if the government is already losing its way, not just in speeches but with regards to levelling-up in the round.”

After the speech, Johnson was asked if he was OK by a journalist. The prime minister defended his performance, saying: “I think that people got the vast majority of the points I wanted to make, and I thought it went over well.”

On Monday evening around a dozen Tory MPs from the 2019 intake glumly gathered in one of their parliamentary offices to share a bottle of whisky. They were said to be disappointed and disillusioned, and shared discontent that Johnson wasn’t listening to feedback.

Like business leaders, the main concern of Conservative politicians was Johnson’s failure to live up to expectations on policies such as levelling up, rail and social care. Up to 20 Tory MPs were expected to rebel on changes to the social care cap on Monday night.

Shortly after Johnson’s speech, Keir Starmer set out to business leaders what he called Labour’s pro-business agenda, including closer trade and regulatory ties with the EU. “I think that sometimes, if I may say so, our party has come across as thinking that business is to be tolerated in some way but not to be celebrated as a good in itself,” the Labour leader told another gathering of the CBI’s multi-venue conference, this one in Birmingham. “That mindset has changed under my leadership.”