Some of Britain’s biggest seaports are considering legal action against the government to recover the costs of building border control posts they fear will never be used, after confirmation that post-Brexit import checks will be delayed for a fourth time.

Physical checks on fresh food and plants from the EU were due to begin in July but have been pushed back to the end of 2023, the Brexit opportunities minister, Jacob Rees-Mogg, confirmed in a written statement published on Thursday. Instead, he announced plans to digitise all checks and paperwork at the border, with a new strategy published in the autumn.

The decision not to implement controls means Britain will effectively continue to rely on the EU to monitor food and plant safety. Food producers said they were being placed at a disadvantage compared with European competitors who would have less red tape to deal with.

The British Ports Association (BPA), a lobby group for the industry, said it was concerned the expensive border posts, subsidised with nearly £200m from the taxpayer, may never be used. The group said its members would ask for permission to bulldoze the new buildings if the government confirmed this was the case.

Richard Ballantyne, the BPA’s chief executive, said ports had rushed to get infrastructure ready on time: “This announcement is a major policy change, meaning the facilities will effectively become white elephants, wasting millions of pounds of public and private funding”.

Ports had already begun hiring staff in preparation for the additional post-Brexit checks. Meanwhile, the government spent public money building inland border control facilities at sites where there was not enough space for infrastructure next to the quay.

While the EU introduced checks on goods arriving from the UK immediately after Brexit, ministers are now targeting the end of 2023 for a new border control regime, three years after the end of the Brexit transition period. Checks on meat were due to start on 1 July and on dairy on 1 September, with all remaining goods including fish and composite foods to be subject to checks from 1 November. A date for controls on live animals has yet to be agreed.

During a tour of Eurotunnel’s Folkestone facilities on Thursday, Rees-Mogg conceded money had been spent on facilities that now may not be needed.

“I do accept that some money was spent in preparation for 1st July which won’t now be needed, but the ports will benefit, as they are saying at Eurotunnel, from the easing of flow,” he told the Guardian.

Rees-Mogg said the move could save British businesses “up to £1bn in annual costs”, although all post-Brexit paperwork and checks that have already been introduced will remain in place. He said it would be wrong to impose new checks now, during a cost of living crisis, as this could drive up food prices further.

The operator of Eurotunnel, through which a quarter of all trade between the UK and EU passes, welcomed the announcement.

“We would have had to check more certificates, more declarations, and would not have been able to board trucks which didn’t have the right paperwork to go with the goods,” said John Keefe, director of public affairs at Getlink.

However, the National Farmers’ Union called the move “unacceptable” and said it was another blow for British food producers, as they grapple with soaring costs.

“This is a question of fairness,” said NFU’s president, Minette Batters, calling import controls crucial “to the nation’s biosecurity, animal health and food safety”.

“Our producers have to meet stringent controls to export their own products abroad, all while being left at a continued competitive disadvantage to our EU competitors, who are still enjoying an extended grace period which gives them access to the prized UK market relatively cost and burden free,” she said. “Without them we really do leave ourselves at risk.”