Sky News has cancelled the third scheduled TV debate in the Conservative leadership race after Rishi Sunak and Liz Truss pulled out of the event following a bruising set of exchanges in the first two debates.

“Two of the ​three candidates currently leading in the MPs’ ballots – Rishi Sunak and Liz Truss – have confirmed to Sky News that they do not want to take part,” the broadcaster announced in a statement.

“Conservative MPs are said to be concerned about the damage the debates are doing to the image of the Conservative party, exposing disagreements and splits within the party.”

Earlier, a source in Sunak’s campaign said the former chancellor had never committed to participating in the debate, adding: “We are very happy to do more debates if we are lucky enough to get to the next stage, including Sky News.”

Truss’s aides had said the foreign secretary was “unlikely” to take part if all the other candidates did not.

The decision came hours before the third round of voting by Tory MPs, with the backbencher Tom Tugendhat predicted to be the latest candidate eliminated, reducing the field to four.

Speaking to reporters in London on Monday, Keir Starmer said he was “astonished” at the developments.

The Labour leader said: “I can see, based on what I’ve seen in the debates so far, why they want to do so, because this is a party that is out of ideas, out of purpose – they’re tearing each other apart.”

The cancellation follows sometimes brutal exchanges in TV debates on Friday and Sunday over taxation plans, and about social issues such as transgender rights. While Sunak has faced the brunt of the attacks on the former subject, Penny Mordaunt, the international trade minister, has been repeatedly criticised for being too liberal.

After Sky News cancelled the debate a spokesperson for Mordaunt said: “It’s a shame some colleagues cannot find a way to debate one another in a civil way.” That quote was later deleted, with the spokesperson instead saying Mordaunt “hopes there will be ample opportunity for such scrutiny later in the contest”.

In the latest personal attack on Mordaunt, Anne-Marie Trevelyan, the international trade secretary, accused her of neglecting her ministerial job to focus on her prime ministerial ambitions.

“Understandably, perhaps, now it’s clear, Penny has for the last few months spent some of her time focused on preparing her leadership campaign, for which I have utmost respect – that’s how this system works,” Trevelyan told LBC.

“There have been a number of times when she hasn’t been available, which would have been useful, and other ministers have picked up the pieces.”

After Monday’s vote, two further votes among Tory MPs, on Tuesday and Wednesday, will whittle the contest down to a final two, with the new leader then being decided by a ballot of party members, the result of which is due to be announced on 5 September.

The fifth candidate is Kemi Badenoch, a former levelling up minister, who is viewed as most likely to be eliminated on Tuesday, although she has been gaining some momentum in the contest.

In Sunday evening’s ITV debate between the five candidates, Sunak criticised Mordaunt’s idea of allowing the Treasury to borrow for day-to-day spending, not just investment.

Mordaunt said “too many chancellors have had too many fiscal rules that they have then had to ditch because they weren’t able to meet them”, prompting Sunak to reply: “Literally Jeremy Corbyn didn’t think that was the right approach.”

But most of the criticism of Mordaunt has been more personal, notably over her allegedly liberal views.

The Daily Mail, a strong supporter of Truss, has targeted Mordaunt in particular. On Monday, its front-page headline said that as a minister Mordaunt met the Muslim Council of Britain, despite a government policy of not formally engaging with the group.