The EU’s police agency, Europol, will be forced to delete much of a vast store of personal data that it has been found to have amassed unlawfully by the bloc’s data protection watchdog. The unprecedented finding from the European Data Protection Supervisor (EDPS) targets what privacy experts are calling a “big data ark” containing billions of points of information. Sensitive data in the ark has been drawn from crime reports, hacked from encrypted phone services and sampled from asylum seekers never involved in any crime.

According to internal documents seen by the Guardian, Europol’s cache contains at least 4 petabytes – equivalent to 3m CD-Roms or a fifth of the entire contents of the US Library of Congress. Data protection advocates say the volume of information held on Europol’s systems amounts to mass surveillance and is a step on its road to becoming a European counterpart to the US National Security Agency (NSA), the organisation whose clandestine online spying was revealed by whistleblower Edward Snowden.

Among the quadrillions of bytes held are sensitive data on at least a quarter of a million current or former terror and serious crime suspects and a multitude of other people with whom they came into contact. It has been accumulated from national police authorities over the last six years, in a series of data dumps from an unknown number of criminal investigations.

The watchdog ordered Europol to erase data held for more than six months and gave it a year to sort out what could be lawfully kept.

The confrontation pits the EU data protection watchdog against a powerful security agency being primed to become the centre of machine learning and AI in policing.

The ruling also exposes deep political divisions among Europe’s decision-makerson the trade-offs between security and privacy. The eventual outcome of their face-off has implications for the future of privacy in Europe and beyond.

The European commissioner for home affairs, Ylva Johansson, has argued that Europol supports national police authorities with the ‘herculean task’ of analysing lawfully transmitted data.

The EU home affairs commissioner, Ylva Johansson appeared to defend Europol. “Law enforcement authorities need the tools, resources and the time to analyse data that is lawfully transmitted to them,” she said. “In Europe, Europol is the platform that supports national police authorities with this herculean task.”

The commission says the legal concerns raised by the EDPS raise “a serious challenge” for Europol’s ability to fulfil its duties. Last year, it proposed sweeping changes to the regulation underpinning Europol’s powers. If made law, the proposals could in effect retrospectively legalise the data cache and preserve its contents as a testing ground for new AI and machine learning tools.

Europol denies any wrongdoing, and said the watchdog may be interpreting the current rules in an impractical way: “[The] Europol regulation was not intended by the legislator as a requirement which is impossible to be met by the data controller [ie Europol] in practice.”

Europol had worked with the EDPS “to find a balance between keeping the EU secure and its citizens safe while adhering to the highest standards of data protection”, the agency said.

Founded as a coordinating body for national police forces in the EU and headquartered in The Hague, Europol has been pushed by some member states as a solution to terrorism concerns in the wake of the 2015 Bataclan attacks and encouraged to harvest data on multiple fronts.

Europol buildings in The Hague.

In theory, Europol is subject to tight regulation over what kinds of personal data it can store and for how long. Incoming records are meant to be strictly categorised and only processed or retained when they have potential relevance to high-value work such as counter-terrorism. But the full contents of what it holds are unknown, in part because of the haphazard way that EDPS found Europol to be treating data.

Only a handful of Europeans have become aware that their own data is being stored and none is known to have been able to force disclosure. Frank van der Linde, who was placed on a terror watchlist in his native Netherlands and later removed, is one of the rare visible threads in an otherwise unseen mesh.

The political activist, whose only serious run-ins with police amount to breaking a window to gain entrance to a building and create a squat for homeless people, was removed from the Dutch watchlist by authorities in 2019. But a year prior to this removal he had moved to Berlin, which unknown to Van der Linde at the time prompted Dutch police to share his data with German counterparts and Europol. The activist discovered his entanglement with Europol only when he saw a partially declassified file at Amsterdam city hall.

To get his personal data removed from any international databases he turned to Europol. He was surprised when in June 2020 it responded saying it had nothing he was “entitled to have access to”. The activist took his complaint to the EDPS. “I don’t know if they deleted the data after Dutch authorities updated them [that] they don’t consider me an extremist … Europol is a black box.”

“The ease of getting on such a list is horrific,” Van der Linde said. “It’s shocking how easily police share information over borders, and it’s terrifying how difficult it is to manage to delete yourself from these lists.”

Concerns over Europol’s treatment of sensitive data prompted the watchdog to raise its own questions in 2019. Its initial findings in September of that year showed that data sets shared with Europol were stored without the proper checks to verify whether people scooped up in them ought to be monitored or their data retained. Access to the ark is restricted to authorised personnel and a lot of its content has been examined, cleansed and used legally.

When Europol failed to convincingly answer the watchdog’s concerns, the EDPS publicly admonished the police agency in September 2020 making clear what was at stake: “Data subjects run the risk of wrongfully being linked to a criminal activity across the EU, with all of the potential damage for their personal and family life, freedom of movement and occupation that this entails.”

The tussle that followed is captured in a series of internal documents obtained under freedom of information laws. They show Europol stalling for time and the watchdog telling them that they have failed to resolve “the legal breach”. The police agency appears to be holding out for new EU legislation to provide retrospective cover for what it has been doing without a legal basis for six years.

The European Commission’s nervousness over a public clash was enough to pull Monique Pariat, the EU’s director general for home affairs, into a meeting between the two agencies in December 2021. Sources said the watchdog had been encouraged to “tone down” its public criticism of Europol.

But the head of EDPS, Wojciech Wiewiórowski, told the Guardian that the meeting was “the last moment for Europol to add some information that wasn’t added in their last replies to our letter”.

As the meeting did nothing to answer Wiewiórowski’s concerns on lawful retention of data “there was no other way to solve the problem, for us” he said, “than to issue a decision to erase the data which is over six months”.

Niovi Vavoula, a legal expert at Queen Mary University of London, said: “The new legislation is actually an effort to game the system. Europol and the commission have been attempting an ex-post rectification of illegally retaining data for years. But putting new rules in place does not legally resolve previously illegal conduct. This is not how the rule of law works.”

Experts’ concerns are not confined to Europol’s flouting of rules on data retention. They also see a law enforcement agency that aspires to conduct mass surveillance operations.

Members of the civil liberties, justice and home affairs committee of the European parliament during a hearing in June 2021 compared the agency to the NSA. Wiewiórowski surprised attenders by endorsing the comparison in relation to Europol’s practice of retaining data. He pointed out that Europol was using similar arguments to those used by the NSA to defend bulk data collection operations and mass surveillance as revealed by Snowden.

“What the NSA said to Europeans after the Prism scandal started was that they are not processing the data, they are just collecting it and they will process it only in case it is necessary for the investigation they are doing,” Wiewiórowski told MEPs. “This is something that doesn’t comply with the European approach to processing personal data.”

Eric Topfer, a surveillance expert at the German Institute for Human Rights, has studied the proposed new Europol regulation and said it foresees the agency pulling in data directly from banks, airlines, private companies and emails. “If Europol will only have to ask for certain kinds of information to have them served on a silver platter, then we are moving closer to having an NSA-like agency.”

The struggle with EDPS over data storage is the latest evidence of Europol favouring technosolutions to security concerns over privacy rights. Europol’s boss, previously Belgium’s top cop, co-wrote an op-ed in July 2021 which argued that the needs of law enforcement agencies to extract evidence from smartphones should trump privacy considerations. The article argues for a legal right to the keys to all encryption services.

No mention was made of Pegasus spyware revelations that showed that many governments, including some in Europe, were actively attempting to intercept the communications of human rights defenders, journalists and lawyers for whom encryption offers their only protection.

Europol’s boss, Catherine de Bolle, has argued that the needs of law enforcement agencies to extract evidence from smart phones should trump privacy considerations.

In 2020, Europol trumpeted its involvement together with French and Dutch police in hacking the encrypted phone service EncroChat, unleashing a torrent of personal data into the ark. When the secret operation was revealed by Europol and its judicial counterpart, Eurojust, it was hailed as one of the biggest successes in battling organised crime in Europe’s history. In the UK alone, about 2,600 people were taken into custody by August 2021 and Nikki Holland, the director of investigations at the UK National Crime Agency, compared the hack to “having an inside person in every top organised crime group in the country”.

Europol copied the data extracted from 120m EncroChat messages and tens of millions of call recordings, pictures and notes, then parcelled it out to national police forces. The flood of evidence of drug trafficking and other offences drowned out qualms about the implications of the operation. The hacking operation that turned EncroChat phones into mobile spies acting against their users has important similarities with surveillance malware such as Pegasus.

Lawyers from Germany, France, Sweden, Ireland, the UK, Norway and the Netherlands, all representing clients caught up in the aftermath, met in Utrecht in November 2021. They found that cases were being built across Europe based on evidence of which authorities were unwilling to reveal the provenance. “Investigators and prosecutors were hiding or deforming the facts,” said the German attorney Christian Lödden. “We all agree that these are not the best people in the world, but what are we ready to sacrifice in order to convict one more person?”

Police officers during a raid in a business park in Weißensee, Germany, in October 2021 as part of an investigation into drug trafficking and arms dealing. The raid was triggered by decrypted data from the short message service Encrochat.

EncroChat clientele included non-criminals, people such as lawyers, journalists and business people. The Dutch attorney Haroon Raza was one of them and said he bought an EncroChat handset at a phone shop in Rotterdam. He demanded that his data be erased. “As far as I could understand, a copy still lies in Europol’s databases where it could remain forever.”

French lawyer Robin Binsard is convinced that the whole operation amounts to mass surveillance. He said: “Dismantling a whole communication system is like the police searching all the apartments in a block to find the proof of a crime: it violates privacy and it’s simply illegal.”

Since 2016, Europol has also been running a mass screening programme in refugee camps in Italy and Greece, sweeping up data from tens of thousands of asylum seekers in search of alleged foreign fighters and terrorists. According to a partially declassified EDPS inspection report obtained under freedom of information laws, “routine checks” by Europol of migrants crossing EU borders “are not allowed” as there is “no legal basis” for such a programme. The screening may have resulted in migrants’ personal data being stored on a criminal database regardless of any links being found to crime or terrorism. Europol has declined to reveal any operational details.

Internal documents make clear that by spring 2020 Europol was developing its own machine learning and AI programme, even as the EU data watchdog was snapping at its heels. Finding itself with a growing cache of data, the agency turned to algorithms to make sense of it all. A month after the data supervisor publicly admonished Europol, the agency came back with a question: if it wanted to train algorithms on the data it had already been admonished for retaining, could it start the data protection impact assessment process for this without EDPS oversight?

The request makes it clear that the algorithms, which included facial recognition tools, would not be designed nor used to retrieve sensitive data such as health status, ethnic background, sexual or political orientation, even though, as Europol admitted, such data would inevitably be processed by the tools: “We recognise that the produced results will contain sensitive data and its processing will be in line with Europol Regulation.”

When the watchdog did not provide the green light, Europol decided in effect to sideline the EDPS and go ahead regardless, confirming as much in a January 2021 letter.

(L-R) European commissioner for home affairs, Ylva Johansson, executive director of Europol, Catherine de Bolle, the French minister of interior, Gérald Darmanin, German MP Stephan Mayer, and the Belgian minister of the interior, Annelies Verlinden, on the sidelines of their meeting to discuss ways of preventing migrants crossing the Channel, in Calais, France on 28 November.

The watchdog responded by saying it would open a formal monitoring procedure. By the end of February 2021, Europol pulled the brake on its machine learning programme. Europol told the Guardian that, to date, it “has not made use of own machine learning models for operational analysis and has also not carried out ‘training’ of machine learning.”

But there are clear signs that the brake will be released soon. Europol has already started a recruitment round for experts to help with the development of AI and data mining.

The emerging shape of Europol is alarming some MEPs such as Belgium’s Saskia Bricmont. “In the name of the fight against criminality and terrorism we have an evolution of an agency, which performs very important missions, but they are not executed in the right manner. This will lead to problems,” she said.

Chloé Berthélémy, an expert with the European Digital Rights network of NGOs, said that while Europol lags behind the US in terms of technological capacity, it is on the same path as the NSA.

“Europol’s capacity to hoover up huge amounts of data and accumulate it, in what could be called a big data ark, after which it is almost impossible to know what they are used for, makes it a black hole.”