For weeks now, Westminster has been fixated on the number 54 – the tally of Conservative MPs needed to trigger a vote of no confidence in Boris Johnson. If, as now appears increasingly likely, that threshold is breached in the coming days, another number will become all-important: the 180 MPs required to kick him out of Downing Street.

The Tories have a reputation for ruthlessness when it comes to throwing out their leaders, and the rules of a no-confidence vote are stark: if Johnson loses, he is out.

He would be expected to stay on as prime minister while a successor was chosen, but he would be disqualified from standing in the leadership race.

The House of Commons is in recess until the afternoon of Monday 6 June; but if the 54 threshold is reached this week the 1922 Committee chair, Graham Brady, could announce that a vote will be held when MPs return.

As when Theresa May faced a similar vote in December 2018, Conservative MPs would file into a House of Commons committee room to cast their votes in a secret ballot.

And with 359 sitting Conservative MPs, it would take 180 votes to defeat the prime minister.

In theory, Johnson should have a powerful built-in advantage: according to calculations by the Institute for Government, 80 Tory MPs are ministers in Johnson’s government, with another 47 serving as principal private secretaries, ministers’ aides in the Commons.

These MPs are referred to at Westminster as the “payroll vote” (though PPSs are not, in fact, paid) and can be expected to turn out reliably for the government in the division lobbies of the Commons. Another 20 are trade envoys to various parts of the world, owing these posts to the prime minister.

Yet while Johnson has used his powers of patronage ruthlessly, to shore up support and reward backbench waverers, the luxury of a secret ballot may free up some critics to vote against him, even if they have plum roles in government.

Is this the moment when the dam bursts – and his support drains away?

“It will be a secret ballot and I wouldn’t assume that the payroll will all vote for him. It’s a secret ballot for a reason,” said one Johnson critic on the back benches. Another suggested two-thirds of the payroll vote might stick with him, leaving perhaps 40 or more to peel off.

Graham Brady, chair of the 1922 Committee of backbench MPs.

At least one minister in a marginal seat is understood to have concluded that they have no chance of remaining an MP if Johnson leads the party into the next election. They have not resigned from the government, fearing that would do little damage, and are wary of submitting a letter lest their name leaks, but would “100%” vote against him if the 54-letter threshold was attained.

Some MPs suggest Johnson might prefer to face a quick vote next week, rather than in late June, after the Tories are expected to struggle in a pair of key byelections, in Wakefield and Tiverton.

These two contests are seen by backbenchers as critical tests of Johnson’s popularity, in very different types of constituency.

If they lose both – or even see hefty swings against them – it would strongly reinforce the sense that he has become an electoral liability.

The mood at Westminster on Tuesday was increasingly that the threshold might be met within days, rather than weeks, however – a sense that was underlined by reports that Johnson had begun the task of ringing round potential swing voters in his party, in an attempt to shore up his position.

When he was last imperilled, earlier this year, the former education secretary Gavin Williamson ended up with a knighthood, and pressure groups of backbenchers were able to force their pet policies on the government – fracking being just one example.

As one senior Tory put it: “One of the things his critics underestimate about Boris is, Boris knows how to stack a deck in his favour.”

Several sources said Johnson’s allies had also recently used the threat of a snap general election against his critics – saying that if they tried to move against him, he would trigger an early poll and let the public decide if he should keep his job.

If the crunch point does come next week, the prime minister’s detractors will urge their colleagues to seize what could be their last opportunity to ditch him before the next general election – because if the prime minister wins, he is safe for a year.

That would take his party perilously close to the next contest, which must be held before January 2025 – though the executive of the 1922 Committee could, in theory, change the rules and allow another vote, if they felt it was in the party’s interest.

As one MP working to remove him put it: “There will be people who wouldn’t themselves put a letter in, but when they’re given a ballot paper, you’ve basically got a straight, binary choice, and the question on the ballot paper is effectively, do you want to go into the next general election with Boris Johnson leading the Conservative party? It’s pretty much now or never.”

Many at Westminster believe the most likely outcome is that Johnson wins by a narrow majority – one veteran Tory described it as “nip and tuck”. He would then face the decision of whether to throw in the towel, rather than press on, with his authority badly undermined.

Colleagues who know him well suggest there is little or no chance of him voluntarily stepping aside, and instead would press on in the hope he can rekindle the public’s love for him.

“He would hang on even if it was a one-vote majority,” said one former cabinet colleague. “He loves the trappings.”