Dominic Raab is facing increasing pressure to resign, including from some of his own party’s MPs, after claims he effectively disappeared for more than a week while on holiday during the collapse of Afghanistan, delegating almost all duties to juniors.

Raab’s department did not deny he asked another minister to make a call on Friday to assist the evacuation of former British military translators while he was in Crete. Labour, the Liberal Democrats and SNP demanded Boris Johnson sack his foreign secretary if he did not decide to quit.

Some Conservative MPs said privately they agreed, with one telling the Guardian Raab’s position was now “untenable”.

Raab is fast becoming a focal point for wider political anger over the humiliating retreat from the Taliban after 20 years of military involvement and 457 UK deaths, with concerns about his department’s seeming lack of foresight exacerbated by his apparently semi-detached role.

A Whitehall source confirmed a Daily Mail report that while Raab was on holiday in Crete on Friday, a week after his arrival, Foreign Office officials advised him to speak by phone to his Afghan counterpart, Hanif Atmar, to request assistance on the removal of translators who had worked with the British military.

They were told Raab was not available and that a junior minister, Zac Goldsmith, a Tory peer, should make the call instead. Because Lord Goldsmith was not Atmar’s direct equivalent, there was a delay of a day. The paper has now reported that the call never even took place.

The source told the Guardian Raab “refused to be contacted on basically anything” for more than a week, and instead directed that “everything had to go to Goldsmith”. They added that Raab’s team had told civil servants “there was an incredibly high bar to getting him to look at anything while on holiday”.

A separate diplomatic source also said there had been increasing frustration at a lack of support from Raab in the weeks leading up to the fall of Kabul.

They said Raab had not spoken to any of the key UK ambassadors in the region, such as in Pakistan or Uzbekistan, or regional ambassadors in London before the weekend – even to offer moral support – and commented: “You don’t need a team of staff to do that, you just need to be a decent human being to say, ‘How are you doing? It’s going to be a tough few weeks, how can I help?’” The source added: “He has completely missed the boat on everything.”

One Tory MP said Raab’s position was untenable and that “not coming home was his biggest mistake” – he reportedly only arrived in the UK in the early hours of Monday morning, a position supported by some other backbenchers.

Other Tory MPs said Raab had been “lacklustre” and that he still had “big questions to answer about what he knew when”. One said they worried Raab’s actions played into Labour’s criticism of ministers’ incompetence, saying this was beginning to register with voters.

But another MP said: “There’s no pressure on him unless there is clear evidence that UK-linked Afghans are being killed due to inaction.”

Downing Street did not comment, beyond officials confirming Johnson still had confidence in Raab.

Labour – which says it has had no contact with Raab since the Afghan crisis began, unlike with the defence secretary, Ben Wallace – called on Thursday for Raab to resign or be sacked. The shadow foreign secretary, Lisa Nandy, said he “should be ashamed”. The Lib Dems and SNP also demanded that he go.

Margaret Beckett, who was foreign secretary under Tony Blair, said it was unheard of for a foreign secretary to refuse to make a call.

“I’m always reluctant to get involved in how departments are being run, but if it is the case that the Foreign Office were asking the foreign secretary to intervene with the Afghan foreign minister, then it is one of the most extraordinary things I have ever heard, that on behalf of people to whom we owe a debt of honour, he thought it could be delegated to a junior minister,” she said.

“The one thing that a foreign secretary can bring to the job is the relationships they have with other foreign ministers. Nobody can substitute, no matter how highly you think of expert officials or juniors.”

Labour’s Harriet Harman said the government should listening to women in Afghanistan rather than having Raab “simply talking to other male leaders”. She said it should appoint a minister responsible for liaising with Afghan women and that women should be included in any future negotiations with the Taliban.

Another critic was Maj Gen Charlie Herbert, who undertook three tours of Afghanistan and was senior Nato adviser in Kabul. He said it was “disappointing that the foreign secretary has not displayed the same level of determination to rescue our vulnerable Afghan interpreters and staff as that displayed by his courageous ambassador and embassy staff in Kabul”.

Wallace vigorously defended Raab on Thursday, despite the defence secretary having been privately critical of Raab’s department over its handling of the Afghan crisis.

Wallace told the BBC that by Friday the Afghan government was “melting away quicker than ice”, adding: “A phone call to an Afghan minister at that moment in time would have not made a difference.”

Challenged on whether he could be sure about this, Wallace said: “I do know for sure, because last Friday what we were absolutely worried and unsure about was whether the airport would remain open. You can speculate whether the phone call should or shouldn’t have been made, but it wouldn’t have made a blind bit of difference.”

Raab chaired a virtual meeting of foreign ministers from G7 nations on Thursday, saying in a subsequent statement that the discussion covered “the gravity of the situation and the significant loss of life and internal displacement in Afghanistan over recent days”.

He “underlined the importance of the Taliban holding to their commitments to ensure the protection of civilians and are deeply concerned by reports of violent reprisals in parts of Afghanistan”, he said, adding that the G7 “are continuing efforts to do everything possible to evacuate vulnerable persons from Kabul airport and call on all parties to continue to facilitate that”.