Boris Johnson had almost finished his most recent Downing Street press conference, when his tone suddenly changed. Sadiq Khan had left a “black hole” in Transport for London’s finances, said the energised prime minister, and the capital’s mayor should be held to account for his profligacy.

Beyond the ensuing controversy about Johnson using a Covid-focused event to launch a partisan attack, his words demonstrated something else: the prime minister was still deeply invested in a city he ran for eight years, and painfully aware that a successor as Conservative mayor looks increasingly unlikely.

A series of polls, including one on Tuesday, put Khan, the Labour mayor who took over from Johnson in 2016, on 50% or more of first-choice votes on 6 May, more than 20 points ahead of his Tory rival, Shaun Bailey.

While many Conservatives privately blame Bailey’s often hapless campaign and candidacy for the gap, it also illustrates one of the starkest changes to affect British politics in recent years – a more general Conservative collapse in London.

Tony Travers, visiting professor at the London School of Economics’ department of government, has produced a chart showing how the general election vote share in London, which broadly tracked the wider totals in Britain for decades, started to diverge widely in the late 90s, and in particular since 2010.

At the 2019 election, Labour’s London vote share was 15 percentage points higher than its national total; the Conservatives’ was 13% points lower. Since 1992, the number of London seats held by Tory MPs has nearly halved.

It is, Travers says, in part down to national factors, such as changes in party allegiance that mean younger, urban voters increasingly support Labour. Additionally, London’s demographics have changed.

Johnson’s mayoral election wins in 2008 and 2012 were largely based on a “doughnut” strategy – harvesting votes from more suburban, and often more predominantly white, outer suburbs. However, Travers says, outer London is increasingly diverse – “more like inner London used to be”. A YouGov poll last week found Khan winning easily in outer London, with a 45% to 28% lead over Bailey on first preference votes.

The Conservatives could very easily drop below their current low of eight of the 25 London assembly seats, Travers predicted, while in the next round of London council elections, in 2022, more boroughs could become, in effect, one-party Labour states.

To an extent, history is repeating itself. The Conservative-led push to create the Greater London Council in the early 60s was in part to expand the capital’s local government into the suburbs, after 31 years of continuous Labour rule on its inner-city predecessor, London county council.

However, there is a notably personal element. Johnson’s eight years running London are central to his personal mythology as a politician, and were arguably key to his appeal to Conservatives to replace Theresa May, as someone who had proved he could win in Labour strongholds. He delivered on this, winning an 80-seat Commons majority by re-colouring much of Labour’s “red wall” seats in blue.

This time, Johnson’s appeal across the party divides was very different. In city hall, Johnson combined very recognisably Conservative attitudes to issues such as policing with an at times liberal image – an Islington-dwelling, cycling mayor who backed an amnesty for undocumented migrants. In contrast, as prime minister he was the designer of an ultra-tough Brexit, presiding over a No 10 fizzing with forays into culture wars on race, statues and the BBC.

Shaun Bailey earlier this month. Many Tories privately blame his hapless campaign.

“I think he feels it’s his turf, as it were, and it’s quite possible he resents the Tories losing there,” says Jenny Jones, a Green peer who, as a London assembly member, had many clashes with the then-mayor.

Jones wonders whether the government’s recent announcement that the London election system will be moved to first past the post, penalising smaller parties such as hers, could in part be a belated revenge: “I think he did find it very uncomfortable to be so challenged. It must have annoyed him that we were there, constantly drawing attention to his flaws.”

While Johnson remains publicly loyal to Bailey, there is an inescapable sense that he feels that if he were unleashed into the mayoral fray again, he would have a chance of defeating Khan.

But those on the other side of London’s political divide view this as fantasy. “Johnson has constantly underestimated Sadiq,” says one ally of the Labour mayor. “And he does seem to take Sadiq’s success very personally.” Khan, they argue, is as adept as Johnson in waging war in the “new age of social value politics”, for example his public disputes with Donald Trump.

And while Conservatives may hope that a disastrous London election next month will be largely down to Bailey’s failings, this would be a false comfort, the ally said: “That would make sense if more than half of Londoners had even heard of Shaun Bailey. But they haven’t.”