Ben Wallace rejected reports that some flights from Kabul contained only a few people, saying the UK was "absolutely ploughing through the numbers".
He told BBC Breakfast "every hour counts" and confirmed "the Taliban are letting our people through".
But Taliban checkpoints ring the perimeter of the airport and chaotic scenes have been unfolding outside.
About 4,500 US troops are in temporary control of Karzai International Airport, with about 900 British soldiers also on patrol at the site as part of efforts to secure the evacuation flights.
The Taliban are blocking Afghans without travel documents from entering. Twelve people have been killed in and around Kabul airport since Sunday, according to a Taliban official quoted by the Reuters news agency.
But even those with valid papers have struggled to get to the airport, with reports that some have been beaten by Taliban guards.
An Afghan interpreter who worked for the British army said he received permission to come to the UK last week, but was now in hiding and would "face death" if the Taliban found him.
Gharghasht Hidai, a British Afghan, said there was shooting at the airport as he and his family prepared to leave.
"The situation was chaotic", he said, with one of his children having to sit on the floor of the packed RAF plane. "We are the lucky ones, we made it."
Peymana Assad, a London councillor who was visiting relatives in Kabul, said she joined hundreds of people walking along the traffic-jammed road to the airport to escape - only to find she reached the UK meeting point too late.
Ms Assad, who had previously left Afghanistan as a three-year-old refugee, said a local family allowed her to wait at their home until she could contact the Foreign Office and arrange to join a later RAF evacuation flight.
A British medical student, another former refugee who returned to Kabul in July to get married, spoke to the PA news agency on Thursday as he tried for a fourth time to get to the airport.
On a previous attempt, he said he waited for 10 hours but could not get past the crowds to reach the airport gate. This time, he said there were gunshots being fired and people trying to flee.
The defence secretary said the UK would continue to stay in Afghanistan as long as the US ran the airport, but said the government was also already investing in "third country hubs" for processing people "if they get out to other countries in the region".
It's now a race against the clock to get as many people out of Kabul as possible.
The BBC has been told there are several thousand UK nationals and dual passport holders who still need to get out of the country.
Eight RAF flights went in and out of Kabul yesterday with a similar number expected today. They have the capacity to fly more than 1,000 people a day.
But the BBC understands that not all flights have been full. The airport is operating to a strict schedule. That means aircraft can't just wait around until all their seats are filled.
Most are being flown to a Middle East hub. Commercial charters are being used to make the onward journey.
The greatest challenge at present appears to be processing the people who qualify to be flown out. One person who's trying to get people out told the BBC the "system is working, but it's slow".
Those wishing to fly out are being told to wait for a phone call before setting out for the airport - not least to avoid the risks of travelling through a city under Taliban control.
At some stage someone will have to make the decision to leave.
That will once again be the decision of the US, which is providing the bulk of the security at the airport as well as overseeing Air Traffic Control. They too need to be flown out.
Mr Wallace said seven to 10 RAF planes were taking off every day, with at least 138 due on the next flight out.
He said the passengers on the flights out of Afghanistan this week had included British government personnel, British citizens, media and human rights staff and Afghans who had worked for the UK.
Downing Street said military flights took about 1,200 people out of Kabul between Saturday and 08:00 BST on Wednesday, about 900 of whom were Afghan nationals and the remainder British citizens.
Since late June, 2,000 Afghans who worked for the UK have been resettled with their families under the Afghan Relocations and Assistance Policy (ARAP), according to the Home Office, with a target of 5,000 by the end of 2021.
The UK has also committed to take in up to 20,000 Afghan refugees over the next few years under a separate resettlement scheme - including 5,000 this year.
Two evacuation flights came into the UK on Wednesday, the Ministry of Defence said - a military plane into RAF Brize Norton in Oxfordshire and a passenger plane into an unnamed civilian airport.
On board were Afghans being relocated under the ARAP scheme for those who worked with the British military, British citizens and some other foreign nationals. The MoD did not provide exact numbers but said each flight could hold about 250 passengers.
Mr Wallace said a plane left Kabul on Thursday morning carrying "115 people and their families - those are the Afghans on there as opposed to other personnel".
He added: "None of our planes are leaving empty... our planes never leave empty. If we have spaces on them, we offer them up to other nations."
Downing Street said the UK helped to fly 76 Australians out of Afghanistan on an RAF plane on Wednesday.
He said: "We have a full programme today of many more people coming out - trying to reach our... capability towards the end of the month. And that is so far on track.... we are doing it as fast as we can."
Mr Wallace said many of the flights were at full capacity.
"You and I wouldn't be allowed to fly some of the way those planes are flying in safety, so we are taking considerable risk", he said, adding: "Alongside those people will be troops or others coming in and out."
The defence secretary said additional UK troops would be deployed to Afghanistan to help manage public order on the ground at Kabul airport.
Meanwhile, Labour has accused Foreign Secretary Dominic Raab of putting interpreters' lives at risk after he declined to phone the Afghan foreign minister to get help for the evacuation. The BBC has been told the call was made by a junior minister instead.
But Mr Wallace said "the only thing that mattered" was whether Kabul airport would continue to allow people to get out, telling BBC Breakfast: "No amount of phone calls to an Afghan government at that time would have made any difference."
This plane from Kabul, carrying Afghans and British nationals, landed at a Midlands Airport on Wednesday