No 10 has rejected calls for Boris Johnson to apologise after he said "too many care homes didn't really follow procedures" over coronavirus.
Labour called the PM's comments "crass" and said government advice to care homes had been "conflicting".
And one union accused the PM of blaming care workers for government "failings".
But Downing Street said Mr Johnson had been pointing out that not enough was known about the virus in the early stages of the outbreak.
The PM's official spokesman added that the care homes had "done a brilliant job under very difficult circumstances" and the government had "put in place rigorous testing and additional funding".
Care homes were hit particularly badly by the coronavirus with nearly 20,000 people confirmed to have died of coronavirus in care homes in England and Wales since the outbreak.
Labour's shadow health secretary Jonathan Ashworth said people were "insulted" by Mr Johnson's "crass remarks".
"Care providers were sent conflicting guidance throughout this outbreak, staff could not access testing until mid-April and are still not tested routinely, PPE supplies have been inadequate, thousands of families have lost their loved ones in care homes to this disease, care workers themselves have died on the front line," he told MPs.
He called on Health Secretary Matt Hancock to apologise for the PM's remarks.
Mr Hancock said the prime minister had been "explaining that because asymptomatic transmission was not known about, the correct procedures were therefore not known".
He added: "We've been constantly learning about this virus from the start and improving procedures all the way through and I pay tribute to the care homes in this country who have done so much to care for the most vulnerable throughout the crisis."
Speaking in the House of Lords another government minister, Lord Greenhalgh, admitted that guidance given to care homes during the initial stages of the pandemic was "not as clear as it could have been".
Mark Adams, who runs the charity Community Integrated Care, told BBC Radio 4's Today programme the prime minister's comments were "cowardly" and a "travesty of leadership".
And Unison, a trade union which represents workers providing public services, said: "It's despicable for Boris Johnson to blame incredible, dedicated care workers for his own government's failings.
"Care staff have kept working throughout to help the vulnerable, putting their own health at risk with little or no protective kit and without testing.
"The prime minister should be ashamed, take responsibility and commit to proper, lasting reform of social care."
On Sunday, NHS England boss Sir Simon Stevens told the BBC that plans to adequately fund the social care sector needed to be in place within a year.
Asked on Monday about his remarks, Mr Johnson said: "One of the things the crisis has shown is we need to think about how we organise our social care package better and how we make sure we look after people better who are in social care.
"We discovered too many care homes didn't really follow the procedures in the way that they could have but we're learning lessons the whole time."
During the 2019 election, the Conservatives promised an extra £1bn per year for social care in England over the next five years.
The government has given an extra £3.2bn in emergency Covid-19 funds to English councils, which can be put towards helping with social care costs.
It has also promised an additional £600m for care homes to help with controlling infections.