Chris Stark, head of the UK's Climate Change Committee, told the BBC he rates government policy on insulation as "very poor".
Insulation, together with renewable power, is the way out of the current energy crisis, he says.
Two-thirds of homes, or 19 million, need better insulation, according to government data.
That raises an obvious question: if it is such a good idea, why aren't we all doing it?
The key issue is the cost.
Britain is frequently described as having some of the oldest and least energy efficient housing in Europe. Retrofitting - adding insulation to existing homes - can be very expensive and Mr Stark says the government isn't doing enough to help fund this costly work.
The government needs to provide "a sharper incentive for most people to make these investments in improving the energy efficiency of the home that they live in," he told BBC News.
Rob Jones' four-bed Edwardian family home in Rusholme, Manchester illustrates the challenges.
When he moved, it had an Energy Performance Certificate (EPC) rating of "E", putting it amongst the least energy efficient homes nationwide.
The government estimates that 19 million of the UK's 29 million homes are on the bottom rungs of the Energy Performance rankings with a rating of "D" or below.
Lifting Rob's home into the "B" category meant improving the lagging in the roof, installing more double glazing, insulating the floors as well as putting insulated cladding on some inside and outside walls. The makeover cost £36,000.
Rob says they now use 40% less gas for heating, which is good news as energy prices soar. But, at current energy prices, it'll take at least 20 years to cover the cost.
Retrofitting homes is an even greater challenge for the providers of social housing.
I visited Jean Davidson in her one-bed council flat in Blackpool. The council has spent £2.5m making the 75 flats on her estate more energy-efficient - £33,000 a piece.
They insulated the entire block, took out her little porch and put in triple glazed windows. Her front door was changed to block drafts and some of the exterior walls of the blocks and the entrance area were insulated.
Jean's home is certainly very cosy now - "perfect", as she describes it. But the council estimates it would cost £125m to bring its 5,000 homes up to this standard.
"It's just not financially viable," says John Donnellon, the CEO of Blackpool Coastal Housing, which manages the town's council homes. "There needs to be grant aid to make it happen," he adds.
And remember, the benefits of lower energy bills go to tenants like Jean, not their landlords.
The government acknowledged the importance of improving energy performance in buildings in its delayed Heat and Buildings Strategy published in October last year.
It talked about the need for a "fabric first" approach - improving the efficiency of walls and lofts before replacing heating systems. But the report came seven months after the government cancelled its £1.5bn "green homes grant" scheme.
That scheme - described as "botched" by MPs and closed after just six months - offered people up to £10,000 towards the cost of insulation. Just 47,500 homes were improved under the scheme, far fewer than the 600,000 that was promised at launch.
Chris Stark, of the Climate Change Committee, believes the government will need a new scheme because most homeowners cannot afford to insulate their homes.
As things stand, just a few tens of thousands of homes will install insulation this year, he says. "We really need to scale that up to something more like half a million a year and to do that quickly over the next four or five years."
But with just £3.9bn earmarked by the government to supporting low-emissions homes, that may not happen as quickly as he hopes. The total is also well below the £9.2bn the Conservative Party said it would spend on energy efficiency by 2030 in its 2019 election manifesto.
Investing in insulation is "critical" if Britain is to cut energy consumption and meet its climate goals, says Jan Rosenow of the Regulatory Assistance Project, a clean energy think tank.
"About 23% of our emissions are from keeping warm heating our buildings. We need to bring those down," he explains.
For years, low energy prices have made it more cost effective to just turn up the boiler. That's why we've been caught out in the current energy crisis.
"Our homes, which we didn't insulate because there was no need for it, are now much more expensive to run," explains Mr Rosenow.