Boris Johnson has announced the end of Britain’s military mission in Afghanistan following a hasty and secretive exit of the last remaining troops 20 years after the post-9/11 invasion that started the “war on terror”.

The prime minister confirmed to MPs that the intervention, which claimed the lives of 457 British soldiers, would end even as the insurgent Taliban were rapidly gaining territory in rural areas and UK, US and other forces withdrew.

Speaking in the Commons, Johnson said “all British troops assigned to Nato’s mission in Afghanistan are now returning home”. While he would not disclose the exact timetable of the departure for security reasons, the prime minister added: “I can tell the house that most of our personnel have already left.”

In a separate defence briefing, the head of the armed forces, Sir Nick Carter, acknowledged that recent news from Afghanistan had been “pretty grim” but said the Afghan military had been regrouping to defend urban areas.

He said that while it was fair to say the Taliban now held nearly 50% of the rural districts in Afghanistan and that the Afghan army would also no longer have access to western air power from within the country, he hoped there would eventually be peace talks between the Taliban and Afghan government.

Britain’s remaining 750-strong contingent, part of a wider Nato stabilisation mission, has been leaving the country over the past few weeks to an operational deadline of 4 July after the US president, Joe Biden, said he wanted to pull out most of the remaining 2,500 American combat troops.

Flag-lowering ceremonies have been largely conducted in secret as British forces have pulled out. The last of them took place on 24 June, when the union flag was handed to the British ambassador. Defence sources said the secrecy was at the request of the US, citing operational security.

Johnson and Carter both argued that the intervention in Afghanistan had helped foster fundamental changes in the country, citing improvements in access to electricity, water and education, with 8.2 million more children in school.

Classrooms included 3.6 million girls, added Johnson, highlighting that before 2001 in Taliban-ruled Afghanistan, “virtually no girls attended school”.

Labour said Britain was leaving without having secured the gains of the past 20 years. Angela Rayner, the party’s deputy leader, said that while nobody wanted to see British troops permanently stationed in Afghanistan, “if we simply wash our hands or walk away it is hard to see a future without bloodier conflict and wider Taliban control”.

In reply, Johnson said he did not believe the Taliban were “guaranteed the kind of victory that you sometimes read about”. Later in the debate he added that the British intervention in Afghanistan “was never intended at any stage to be an open-ended commitment”.

There was also unease on the Conservative benches. The former party leader Iain Duncan Smith told MPs the withdrawal was “a little bit like the last days of Vietnam, an unprecedented and hurried exit with no commitment”.

Waheed Omer, adviser to the Afghan president, Ashraf Ghani, and the director general of the public and strategic affairs office, said: “We respect the decision, we understand that our allies make their decisions based on their national interests. We cannot live in the past. What we are now concerned with is the new chapter, and how these relations are going to be realistically aligned to the new realities.”

Britain will leave behind a small number of troops to support a US-led protection force for diplomats in Kabul, although the Ministry of Defence would not say how many were remaining on the ground.

The RAF could become involved in providing air support from airbases outside Afghanistan, following the abandonment last week of the main Bagram base. Losing easy access to air power is a big loss for the Afghan army as it battles to fend off the Taliban advance.

The Foreign Office also intends to maintain an embassy in Kabul, although it will not, at least initially, be guarded by British troops. The UK government will provide £100m in aid and £58m for Afghan defence forces.

Carter said no provincial capital had fallen in Afghanistan and that it was “entirely possible that the Afghan government defeats the Taliban for long enough for the Taliban to realise that they have to talk”.

But he admitted it was one of three possible scenarios for the country, the others being a return to warlordism and a Taliban victory.

Johnson also resisted calls to hold a public inquiry into the war along the lines of the Chilcot report on Iraq, which Tobias Ellwood, the chair of the defence select committee, called for in the Commons. It would take too long, Johnson said, adding that the Chilcot inquiry had taken seven years and “cost millions”.

British soldiers who served could hold up their heads “very high”, Carter suggested. He paid tribute to the soldiers who had died over the past 20 years. The British army now had a new “combat ethos” learned from fighting in Helmand. “They were never defeated on the battlefield,” he said.