A report by the independent Office for Environmental Protection (OEP) also says the country is facing a "deeply concerning decline in biodiversity".
It found many of the government's 23 environmental targets were at significant risk of not being achieved.
The government said it would consider the report carefully.
The study assessed 32 environmental areas - from species numbers to air and water quality improvements - and found nine trends were improving, 11 were static, and eight were deteriorating. In four areas, there was not enough data to make a reliable assessment.
"The situation is poor across the board, with adverse trends across marine, freshwater and terrestrial environments," it said.
Of the 23 environmental targets it looked at, it assessed 14 as "off track" while the remaining nine could not be assessed because the evidence was not available.
It said progress towards climate change adaptation was poor, and "particularly concerning" in the agricultural sector.
A spokesman for Defra said it would publish a new environmental improvement plan later this month that would help it to meet its targets to protect the natural world, tackle climate change and halt the decline in species populations by 2030.
Richard Benwell, CEO of campaign group Wildlife and Countryside Link, said that the report showed that "rapid, concerted action and investment" was needed if environmental targets were to be met.
"To halt the decline of nature, the days of fluffy wish lists, and back-of-the-settee funding for nature policy must end," he added.
The OEP - set up under the 2021 Environment Act to hold the government and other public bodies to account - said there had been "a frequent failure" by the government to meet its own legally-binding targets.
It said the Covid pandemic, the war in Ukraine and the cost of living crisis had worsened a lack of coherence in environmental strategy within Defra and across government.
"Overall, we do not think the current pace and scale of action will deliver the changes necessary to improve the environment in England significantly," the report said.
Dame Glenys Stacey, the chair of the OEP, said there now needed to be co-ordinated action across all levels of government and improved data collection and monitoring to ensure targets were being met.
"Progress on delivery of the 25 Year Environment Plan has fallen far short of what is needed to meet government's ambition to leave the environment in a better state for future generations," she said.
Despite some improvements in air quality and people's engagement with nature, she said: "Many extremely worrying environmental trends remain unchecked, including a chronic decline in species abundance."
A Defra spokesperson pointed to the work it had done since it published its 25 year environmental plan in 2018, including the funding of more than 120,000 hectares of nature recovery projects, increased tree planting rates and restoration of peatland.
A new environmental improvement plan, to be launched at the end of the month, will set out "the comprehensive action this government will take to reverse the decline in nature, achieve our net zero goals and deliver cleaner air and water," he added.