Boris Johnson has been accused of showing “total contempt” for the north of England as a senior Tory MP warned him it was an “illusion” to think the party would comfortably hold seats that it won for the first time in 2019 at the next general election.

Anger erupted after the prime minister pulled out of a conference in Doncaster at the last minute, with those in “red wall” seats turning on the prime minister.

Members of the 50-strong Northern Research Group (NRG) of Tory MPs were promised that Johnson would be the headline speaker at their event for hundreds of activists – hours before he was due to speak.

It later emerged that the prime minister was in Ukraine, meeting President Volodymyr Zelenskiy. A source told the Guardian that a meeting between Johnson and Zelenskiy on Friday had been scheduled in the “government grid” for at least a week.

One Tory MP said the visit was “not an excuse” as Johnson “could have gone there any time”, and added: “Even his most loyal supporters here are pretty pissed off.” Another said it was “no loss to us”, but cautioned: “The PM ought to be making every effort to support and respect the people who hold his future in his hands.”

A senior NRG source said that the group, which represents one of the biggest caucuses within the Conservative party, represents constituencies that helped secure Johnson his 80-seat majority and “came through for him” during last week’s confidence vote.

“That goodwill is gone,” they said, adding that Johnson’s actions had “shown a total contempt for colleagues, contempt for members and contempt for the north”.

Jake Berry, chair of the NRG, earlier delivered a warning to Johnson to deliver on promised tax cuts and said the government should not take the dozens of seats it won in Labour’s former heartlands for granted. He was not confident, he said, that the Tories would hold on to Wakefield in next week’s byelection.

After the chancellor, Rishi Sunak, signalled that he would wait until spiralling inflation subsided before cutting taxes, Berry said it was “time to stop talking about being the party of low tax and become the government of low tax”.

While the support for households announced by Sunak was welcomed by Berry, he said he did not believe that “prices will have come down by this time next year”, so a “different approach” of lowering taxes was needed, “because that’s permanent”.

He said there was a “causal link” between the UK being the only country in the G7 that is “facing a cost of living crisis by putting taxes up” while experiencing “the slowest-growing economy”.

Berry said he was “really disappointed” at Johnson’s no-show, but that he understood the prime minister “does occasionally have to do other things than come to Doncaster – although for us, that would be our top priority”.

The prime minister was defended in a series of tweets by Ben Wallace on Friday evening. The defence secretary called criticism of the trip a “lot of rubbish”, adding: “As a Northern MP myself, I am not affronted by the fact he had to cancel speaking at the conference … Helping Ukraine win and trying to help at home are linked.

“Part of the inflation we see comes from gas and food prices which are partly driven upwards because of this conflict.”

Impressing the importance of Johnson engaging with the NRG, Berry said that “people who think that the Conservative party now has a right to win in the north of England are suffering from a sort of illusion … I think what was different about 2019 was that there was an acceptance of that, there was a sort of owning up by the Conservative party that we need to do better and we’ve got a plan to do it.

“People understand that Covid has intervened, but … political theory without implementation is simply hallucination – and we’ve now got to move ruthlessly to implement real change for people.”

Berry also cautioned: “Parties who fail to listen to the electorate face political annihilation.”

Although Johnson missed the chance to hear from his northern Tory MPs and activists, Berry said he would “phone him tomorrow” to push for the key ideas the NRG was promoting.

Among them was granting local areas with metro mayors tax-setting powers, a new levelling up formula to distribute money more equally across England, and a pledge by the government to increase the number of young people pursuing higher-level apprenticeships from 21% to 50% by 2030.