Dominic Raab “refused to be contacted” about some government business while he was on holiday and handed over decisions to a junior minister in the days leading up to the Taliban’s takeover of Afghanistan, the Guardian has been told.

The foreign secretary is facing mounting calls to quit, even from within his own party, after it was revealed he delegated a crucial call with his Afghan counterpart. Labour said it was a “catastrophic failure of judgment”.

A Whitehall source confirmed to the Guardian that a report in the Daily Mail was correct. The Mail said that while Raab was on holiday in Crete last week, Foreign Office officials advised him to speak by phone to his Afghan counterpart, Hanif Atmar, to request assistance on the removal of translators who had worked with the British military.

However, the officials were told Raab was not available and that a junior minister, Zac Goldsmith, a Tory peer, should make the call instead. Because Lord Goldsmith was not Atmar’s direct equivalent, there was a delay of a day in the call happening.

When asked on Thursday if he would resign, Raab simply said: “No.”

The Whitehall source told the Guardian Raab “refused to be contacted on basically anything” for more than a week, and instead directed that “everything had to go to Goldsmith”. They added that Raab’s team had told civil servants “there was an incredibly high bar to getting him to look at anything while on holiday”.

The Foreign Office said in a statement: “The foreign secretary was engaged on a range of other calls and this one was delegated to another minister.”

A separate diplomatic source also said there had been increasing frustration at a lack of support from Raab in the weeks leading up to the fall of Kabul.

They said Raab had not spoken to any of the key UK ambassadors in the region, such as in Pakistan or Uzbekistan, or regional ambassadors in London before the weekend – even to offer moral support – and commented: “You don’t need a team of staff to do that, you just need to be a decent human being to say, ‘How are you doing? It’s going to be a tough few weeks, how can I help?’” The source added: “He has completely missed the boat on everything.”

A Tory MP said Raab’s position was “untenable” and that “not coming home was his biggest mistake”, arguing he should have cut short his holiday several days earlier. They said he should accept his “Lord Carrington moment” and fall on his sword – in reference to a former foreign secretary who resigned over the failure to foresee Argentina’s invasion of the Falkland Islands.

The shadow foreign secretary, Lisa Nandy, said Raab “should be ashamed” and questioned how the prime minister, Boris Johnson, could allow him to continue serving in the cabinet “after yet another catastrophic failure of judgment”. She added: “If Dominic Raab doesn’t have the decency to resign, the prime minister must show a shred of leadership and sack him.”

The Labour leader, Keir Starmer, tweeted: “Who wouldn’t make a phone call if they were told it could save somebody’s life?”

Other opposition parties have also urged the foreign secretary to step down. Layla Moran, the Liberal Democrats’ foreign affairs spokesperson, said: “Dominic Raab must resign today. If he does not, the prime minister should finally show some leadership, and sack him.”

The SNP’s Westminster leader, Ian Blackford, said Raab’s position was “completely untenable and he must resign, or be sacked”.

In media interviews earlier, the defence secretary, Ben Wallace – who has been privately critical of Raab’s department over its handling of the Afghan crisis – defended his cabinet colleague.

Wallace told BBC Radio 4’s Today programme that by Friday the Afghan government was “melting away quicker than ice”, adding: “A phone call to an Afghan minister at that moment in time would have not made a difference.”

Challenged on whether he could be sure about this, Wallace said: “I do know for sure, because last Friday what we were absolutely worried and unsure about was whether the airport would remain open. You can speculate whether the phone call should or shouldn’t have been made, but it wouldn’t have made a blind bit of difference.”

Speaking to Sky News, Wallace said: “At that time, last Friday, those were not the problems and the barriers for us getting people out. The problem was about whether we could get the airport open, and whether people could be using the airport as the Taliban advanced. That was the block then.”

The Raab row follows intense and often furious criticism of the government from Conservative MPs and peers on Wednesday, when the Commons and Lords were recalled from the summer recess to debate Afghanistan.

Across both houses, 11 Tory former cabinet ministers were among those expressing anger and frustration at Britain’s failures in intelligence and preparation. In the Commons, more than 30 Tory MPs spoke against the government, while only a handful voiced support for its actions.