Priti Patel’s department is resisting growing demands to allow asylum seekers to work following a public intervention from her cabinet colleague Dominic Raab to say that he would be “open-minded’ about the proposal.

Labour leader Keir Starmer, Conservative MPs and refugee charities have all called for the Home Office to allow 70,000 current claimants to seek employment after the justice secretary said a rule change could help to solve the UK’s current labour shortage.

But in testy comments that reveal frustration at Raab’s words, a senior Whitehall source said that if applications to seek employment were allowed, it would “create a pull factor for illegal immigration like never before”. “It would drive a coach and horses though our legitimate immigration system. We would see people who want to come here to work avoiding the system by just arriving and claiming asylum, before starting work the next day,” the source said.

Most asylum seekers are not allowed to work while their case is considered and instead rely on the government for housing and essential living needs. The Home Office has been reviewing the rules around allowing asylum seekers to work for three years.

The latest data shows that over 70,000 people were waiting for a UK decision on their initial asylum application – up 73% over the past two years despite a decline in the number of applicants.

Meanwhile, resettlement of refugees has not increased at the same rate. Only 308 refugees were resettled in Q2 2021, compared to a quarterly average of over 1,400 from 2016 to 2019.

MPs including the leader of the opposition increased pressure on Patel on Thursday to be as open-minded as Raab. Starmer told the Guardian the government should consider changing rules which “defy the common sense test”.

“I met a Syrian doctor who … was unable to work, because the claim hadn’t been properly processed. He desperately wanted to use his skills to help the community that made him very, very welcome and he was prohibited from doing so. That defies the common sense test,” he said.

Reacting to Starmer’s comments, a Whitehall source said: “Allowing asylum seekers to work will see more people making dangerous journeys to enter our country illegally. It’s as simple as that.”

Andrew Bridgen, the Conservative MP for North West Leicestershire, said he would also allow asylum seekers to work because their claims are taking too long – in some cases more than a year – to process. “They should be allowed to work because the system is not working properly. I would like to see their applications to be processed a lot faster,” he said.

The Tory MP for Ruislip, Northwood and Pinner, David Simmonds, has been at the forefront of a campaign to allow asylum seekers to work and aid integration. “Because asylum seekers cannot work whilst they wait for a decision on their claim, and if successful are given just 28 days to move into new accommodation and find work or apply for universal credit, there are currently significant barriers to successful integration,” he wrote in PoliticsHome in April.

The former foreign secretary was asked by the Spectator magazine on Thursday whether he would support the measure given the current labour shortage. “I would be open-minded about it,” he replied.

“If [asylum seekers] learn the language and they can work, they integrate much better and they make a positive contribution.”

Responding, Stuart McDonald, the SNP’s home affairs spokesperson in parliament, wrote on Twitter: “A very rare but welcome occasion on which I can agree with Dominic Raab! How can Priti Patel continue to resist the overwhelming logic?”

UK policy is more restrictive than those in most comparable countries. EU law requires member states to grant asylum seekers access to work after they have been waiting for nine months for a decision on their claim. Canada and Australia allow asylum seekers to work immediately. In the USA, they are eligible to work after six months.

In 2020, Germany received the highest number of asylum applicants in the EU, with 122,015 applicants. France received 93,475 applicants. In the same period, the UK received the fifth largest number of applicants, 36,041. This represents only the 17th largest intake when measured per head of population.

Dr Peter William Walsh, researcher for the Migration Observatory at the University of Oxford, said: “The backlog in processing asylum claims has increased sharply in recent years and is currently nine times higher than it was a decade ago. For almost all of these people, it would be illegal for them to take a job – 80% of cases are not addressed within six months, and many people wait more than a year.”

Enver Solomon, CEO of the Refugee Council, said: “Thousands of skilled and talented people live on limited financial support in limbo awaiting a decision on their asylum claim for months or years on end desperate to be able to work to contribute to our communities. It’s vital they’re given this chance.”