Officials blamed slow border processing and more coaches than expected for 12-hour queues for ferries from Dover.
On Sunday, Suella Braverman said it would be unfair to view the delays as an "effect of Brexit".
But the PM's spokesman said the government was "in discussion" to speed up new passport checks in France.
Rishi Sunak's official spokesperson said "a combination of factors" were to blame for delays - including poor weather and the high volume of traffic.
Asked whether Brexit was one of the factors, the spokesman noted French officials now manually inspected and stamped every passport as passengers left the UK, which required time.
The spokesman said: "We recognise there are new processes in place - that's why authorities were given a long time to prepare for the new checks, including during the transition period, of course.
"And we are in discussion with our French counterparts about how we can further improve the flow of traffic."
Delays to access ferries to France from Dover were first reported on Friday night when the port declared a critical incident.
Extra ferries that were laid on overnight on Saturday were not enough to prevent the queues at Dover increasing through much of Sunday.
Officials have explained that long border processing times were partly to blame for delays - and ferry companies said bad weather had disrupted some journeys.
The port said ferry companies received 15% more coach bookings for the Easter period than what had been expected - which take longer to process than cars.
Dover also faced enormous disruption ahead of the spring getaway last year, with thousands of lorries queuing to leave the country.
However, Christine Dixon, director of Cranberry Coachways, said the situation was much worse than previous years.
"We have had delays [before] but nothing at all like this," she told BBC Radio 4's PM programme.
She added that coaches were booked 12 months in advance by holiday companies and the port "should have known what to expect".
Teacher Will Gresswell waited at Dover for 40 hours with 67 teenagers on a coach heading on a school football trip to Costa Brava.
"We had heard a bit on the news that there might be queues, and we had plenty of water and some crisps and bits and pieces on board," he said.
"But there were a number of other coaches in the queues that didn't have anything."
He added that a nearby coffeeshop had a "constant queue" of around an hour and a half.
"People were trying to get information but there was no real information coming forward," he said.
Home Secretary Suella Braverman told the Sophy Ridge On Sunday programme on Sky it would not be fair to view the delays as "an adverse effect of Brexit".
Sir Keir told LBC on Monday: "Of course Brexit has had an impact - there are more checks to be done.
"That doesn't mean that I am advocating a reversal of Brexit, I am not. I have always said there is no case now for going back in."
He added: "Once we left, it was obvious that what had to happen at the border would change.
"Whichever way you voted, that was obvious. Whichever way you voted, you are entitled to have a government that recognises that and plans ahead."
Alistair Carmichael, the Liberal Democrats' spokesman for home affairs, accused the government of being "in complete denial about the impact of their botched deal with Europe".
"Businesses and travellers are tied up in reams of red tape but ministers are refusing to lift a finger," he said.
"It shows the Conservative Party is out of touch, out of excuses and should be out of power."