A police chief who criticised his colleagues for failures on racial justice has been ousted as a candidate to be the next commissioner of the Metropolitan police.

Jon Boutcher, the former head of the Bedfordshire force and a former Met senior counter-terrorism detective, applied three weeks ago to succeed Cressida Dick as Britain’s top officer. But the Guardian has learned that he has been told by the Home Office that his application will no longer be considered.

Boutcher left policing in 2019 but retained leadership of an investigation into a string of killings in Northern Ireland.

While a chief constable, Boutcher was in the vanguard of those who realised policing still had huge problems with race, at a time when many chiefs were downplaying them.

Only after the murder of George Floyd in the United States in May 2020 did chiefs accept that large-scale problems continued here and merited radical reforms. In a 2018 Guardian interview, Boutcher said of making policing less white-dominated: “It’s not rocket science, increasing the numbers of BME [black, minority and ethnic] officers. It’s about having a genuine commitment, empowering people to make it happen and throwing your weight behind it.”

He led for police chiefs in England and Wales on race and, also in 2018, said: “My challenge to policing is that the pace of change is too slow. I think it’s about commitment at a senior leadership level. I don’t accept that everything has been done … There have been the words, but not the actions.

“The police establishment need challenging on race … If we don’t remember the lessons of history, then there is a danger you repeat the mistakes of history. Racial disparity in policing undermines legitimacy and threatens policing by consent.”

In 2019 he also attacked government cuts for damaging the fabric of society: “I’m not a liberal-hearted, lefty softie. But let us police those who want to damage our society and choose to damage our society, but provide the support for those who are caught by circumstance and give them pathways away from crime. That needs more money. Let us concentrate on those who choose to be criminals.”

Boutcher was a career detective and led investigations for Scotland Yard’s counter-terrorism command, including the hunt for men who tried to bomb London on 21 July 2005 and the investigations into the attempted car bombing of London’s Haymarket in 2007 and the Glasgow airport attack.

The next commissioner will be chosen by the home secretary, who has to show due regard for the views of London’s mayor. The job advert for the Met commissioner said the force needed radical reform to boost flagging public confidence and rectify “serious failings”.

Among those who applied is Mark Rowley, a former head of counter-terrorism who left the Met in 2018. A former chief constable of the Surrey force, he is seen as the favourite to be the next commissioner.

Also waiting to see if he has got through to the next stage is Shaun Sawyer, the head of Devon and Cornwall police, who steps down in August as chief constable.

The only Met leader to apply is assistant commissioner Nick Ephgrave. Mike Bush, the former commissioner of police in New Zealand, who led the force during the Christchurch terrorist massacre, has also applied.

The news of the thwarting of Boutcher’s attempt to reach the top of policing comes days after the Guardian revealed that Neil Basu had pulled out of the race to be the next director general of the National Crime Agency.

Downing Street intervened to halt the process to select the next leader of the NCA after Basu and Graeme Biggar were selected by an expert panel to be the final two candidates. The former Met commissioner Bernard Hogan-Howe is believed to be No 10’s preferred candidate.

Basu had annoyed the government with comments he made about the need for greater action on racial justice. He dropped his candidacy after a senior Home Office official told him no appointment would be made and that the post would be re-advertised.

Boutcher is currently leading Operation Kenova, which is examining the role of “Stakeknife”, a double agent who was the head of IRA’s internal security unit while working for British intelligence, and was allegedly implicated in murders and torture.