The Cabinet Office has been accused of “delay and deception” over its blocking of the release of files dating back more than three decades that reveal the inside story of the intelligence agent Peter Wright and the Spycatcher affair.

Wright revealed an inside account of how MI5 “bugged and burgled” its way across London in his 1987 autobiography Spycatcher. He died aged 78 in 1995.

Details of Wright’s disclosures and the mostly futile attempts by Margaret Thatcher’s government to quash publicity of Wright’s revelations are detailed in 32 files containing government documents from 1986 and 1987.

Tim Tate, an award-winning documentary-maker and author who is seeking access to the files, accuses officials of withholding documents that may cause political embarrassment. “They have behaved appallingly, as if the law does not apply to them,” he said. “It is a waste of public money over documents which should be in the public domain.”

Most government documents are released after 30 years, but officials have cited various exemptions under the Freedom of Information Act to block publication of the Spycatcher files. They initially said the request was “vexatious” because of the amount of work involved in collating the files and redacting any sensitive material.

It is the latest controversy involving the Cabinet Office and freedom of information laws. It has been accused of wasting public funds in a legal battle over the personal diaries of Lord and Lady Mountbatten in which costs are expected to exceed £600,000.

Wright was a senior MI5 officer from 1955 to 1976. After retiring to Australia, he wrote his memoirs, which alleged illegal activities by the security services.

Wright claimed that he was a member of a small group of MI5 officers who plotted to try to force the resignation of the Labour prime minister Harold Wilson because of their suspicions he was a communist spy.

Peter Wright’s 1987 memoir Spycatcher lifted the lid on the British intelligence services.

He also alleged agents burgled and bugged the embassies of hostile countries and allies. One of his most sensational claims was that Sir Roger Hollis, the former head of MI5, was a Soviet agent. A review in 1974 by Lord Trend, a former cabinet secretary, found there was no evidence to show that Hollis had been a Soviet agent.

The government launched legal proceedings to stop the book being published in Australia, but lost the action in 1987. During the hearing, Sir Robert Armstrong, Thatcher’s cabinet secretary, admitted he was prepared to be “economical with the truth” to protect national security under cross-examination from barrister Malcolm Turnbull, who later became Australia’s prime minister.

Ministers also gagged newspapers in England from reporting on Wright’s claims against MI5. The government was found in November 1991 to have violated the right of freedom of speech because of gagging orders against the Observer and the Guardian.

Tate first made a request for the files to the Cabinet Office in April 2019. Under the Public Records Act 1958, records selected for permanent preservation are required to be transferred no later than 30 years after their creation either to the National Archives in Kew, south-west London, or another suitable place of deposit. The government is reducing this timeframe from 30 to 20 years.

Tate says the Cabinet Office acted unlawfully by failing to transfer the files to the National Archives within the required time period. Officials say they were permitted under the rules to delay transfer.

On 12 December 2019, Tate submitted a new request to the Cabinet Office, covering only the first two files in the Spycatcher series. The Cabinet Office said in April 2020 that material was exempt because it was intended for future publication.

Tate complained to the Information Commissioner’s Office (ICO) on 22 June 2020. The Cabinet Office then said it was also withholding the material because it all related to the intelligence services. The ICO has upheld the Cabinet Office’s decision.

“It is like playing a game of Whac-a-Mole,” said Tate last week. “One excuse pops up and you take the time to go through the process and say: ‘No, that’s not lawful’, and then they change tack. It is public money that is paying for this obfuscation and delay, but it never faces any sanctions for failing to meet deadlines imposed by law.”

Maurice Frankel, director of the Campaign for Freedom of Information, said the case raised questions about the Cabinet Office overseeing a large number of freedom of information requests. It is under investigation by the Commons’ public administration and constitutional affairs select committee over its clearing house unit, which supports the handling of freedom of information cases across government.

He said: “There are errors scattered throughout their decisions. There is a real question of whether they are competent enough to advise government on their own freedom of information requests, let alone to advise government on round-robin and sensitive requests.”

The Cabinet Office said: “The information commissioner concluded the Cabinet Office was legally entitled to withhold the information requested under the Freedom of Information Act. We handle all FOI requests in line with the legislation.”

The Observer’s battle to publish

On 22 June 1986, an article published in the Observer revealed details of a court battle on the other side of the world which was sending shockwaves through the British establishment.

The story revealed how Margaret Thatcher’s government was fighting in the Australian courts to block publication of memoirs written by former MI5 officer Peter Wright. He had retired to Tasmania where he was breeding Arabian horses.

The Observer article set out some of the allegations in the Spycatcher memoirs, including Wright’s claim that Britain had bugged diplomats from France, Germany and Greece. It was also claimed that the suite at Claridge’s in London where Nikita Khrushchev, who led the Soviet Union during the height of the cold war, stayed, had been bugged during his visit to London in the 1950s. The next day, the Guardian detailed further claims in the case.

Sir Michael Havers, the attorney general, instituted proceedings for breach of confidence against both newspapers, winning an injunction restraining them from publishing any of the Spycatcher material. The actions triggered a two-year legal battle over the gagging orders, which were extended to other British newspapers. The government lost its case in the Australian courts in 1987 and the memoirs were also published the same year in the US, as Wright’s revelations went around the world

Spycatcher became an international bestseller. On 13 October 1988, the appellate committee of the House of Lords said the injunction against the two newspapers and other media outlets should be lifted because Spycatcher was now in the public domain and its revelations were no longer confidential. Both newspapers subsequently took the government to the European Court of Human Rights, claiming violation of freedom of speech over the gagging orders.

The judges ruled in November 1991 that the government had violated article 10 of the European Convention on Human Rights – freedom of expression – by keeping the injunction in force once the revelations were in the public domain. They ruled the government should pay £100,000 to the two newspapers.