Spending in this area rose by 20% last year to hit £3bn in England.

For many shifts, bosses have been so short-staffed they have been willing to breach the government pay caps for these agency workers, most of whom are doctors and nurses.

Separate data supplied by Labour showed some NHS trusts had paid as much as £2,500 to fill shifts.

Out of 60 responses from trusts, 10 reported the most expensive shift cost over £2,000, and for another 13 it was between £1,000-2,000.

The BBC has also seen evidence of aggressive marketing by agencies - which take a cut of the shift pay. One boasted it is the "best" time to try it, because the fees that can be commanded are increasing.

Although paying out £2,500 for one nursing shift may not be the norm - it shows how much supply shortages can cost the NHS.

Saffron Cordery, interim chief executive of NHS Providers which represent trusts in England, said: "Trust leaders consistently raise concerns about sky-high agency costs.

"They're already facing enormous financial pressures with the impact of inflation and energy costs, and the government's failure to fully fund staff pay awards.

"They are doing all they can to minimise agency spend. But safety must always come first."

The government said the NHS faced some exceptional pressures last year, but it expected spending on agency staff to fall in the future.

The BBC spoke to one cancer doctor who was offered work for £130 an hour - well above the cap for his role - and described the fees available as "astonishing" and a reflection of "desperation by management".

The agency worker: 'More than twice the pay and more control'

Dr James Barson now solely focuses on agency work

Dr James Barson qualified four years ago. But instead of taking the traditional junior doctor route, he now solely focuses on agency work in hospitals across Lincolnshire.

The reason? Pay and the flexibility. He can earn more than twice what he would get paid if he was on the NHS payroll, without the need to commit to nights and weekends.

"Being a junior doctor is very difficult, very demanding. Being a locum I can work nine to five, Monday to Friday and take days off that are entirely my decision."

He says he would not consider taking a shift for less than the market rate of £50 to £60 an hour.

But he says there are downsides to agency work, including lack of holiday and sickness pay, as well as "not always feeling a welcome part of the team".

Agency pay rates are capped at 55% above what a normal employee would receive.

But the BBC investigation found for nearly nine in 10 agency shifts for doctors and dentists, and four in 10 for nurses the caps were exceeded last year.

NHS bosses can only do this if there is a significant risk to patient safety.

The rise in 2021-22 comes after the NHS had appeared to get on top of the use of agency work following a government crackdown in 2015-16 when the cap was introduced.

Ministers set the NHS the task of moving away from agency work, encouraging the use of NHS staff banks, where employees do overtime.

After six years, during which spending had been curbed, the £500m jump has caused concern.

The BBC used a combination of Freedom of Information requests to individual NHS services, as well as analysing published data, to find out what was going on with agency spending, as well as using data obtained by Labour.

About six in 10 trusts provided information on their own levels of spending. It showed huge variation, with some services spending less than 1% of their staff budget on agency workers and others in excess of 11%.

Previously reported figures suggest spending also appears to be rising in other parts of the UK. In Scotland it has doubled in the past year, while in Wales it has risen by more than 40%.

In Northern Ireland, it is four times higher than it was three years ago.

The hospital: 'We've been left with no choice'

Colchester Hospital is just one of many services which is increasingly relying on agency workers.

Emergency department matron Andy Seale says it is a constant battle making sure there are enough staff available.

"Sometimes you can spend a couple of hours every day trying to sort rotas out."

It is a picture that is repeated across the hospital and the other sites that are part of the East Suffolk and North Essex NHS Trust.

Nick Hulme says he would prefer to use his own staff

Chief executive Nick Hulme says a combination of factors is playing a role, including the need to open extra wards to cope with rising demand, as well as staffing shortages.

Some people are even leaving their jobs, only to return later as agency workers.

He says while many agency workers do fantastic work on the wards, he would still prefer to be able to use his own staff. "We know that's best for patient care, best for patient experience. Agency staff require more supervision and tasks such as handovers take longer."

He acknowledges it is not the best use of taxpayer money, but says he has no choice. "Quality and safety of care is always going to be the priority," he adds.

Siva Anandaciva, chief analyst at the King's Fund think tank, said the NHS was caught in a "vicious cycle".

He said a combination of staffing shortages, Covid sickness absences and rising demands was driving the increase in reliance on agency workers.

"I don't think what is happening is a surprise. There will always be some reliance on temporary staff, but you want to minimise it."

And Labour's shadow health secretary Wes Streeting called it an "infuriating" use of money.

"Taxpayers are picking up the bill for the Conservatives' failure to train enough doctors and nurses."

He said Labour would train the doctors and nurses the NHS needs by abolishing the non-dom tax status, where UK residents whose permanent home is elsewhere may not have to pay UK tax on foreign income.

A spokesman for the Department of Health and Social Care said the past year had seen some exceptional workforce pressures, and a drive to get services back to normal following the pandemic.

But she said the expectation was that spending would be reduced by at least 10% this year.

"We have a clear policy to reduce agency spend through capping the hourly pay of agency and temporary staff, ensuring they are only hired through approved NHS agreements to ensure value for money, and prioritising NHS staff."

She said the government was in the process of developing a workforce plan that would help ensure the health service had the staff it needed in the future.

Currently one in 10 NHS posts is vacant.