Bad weather and a shortage of ferries are being blamed.
Capacity at the Port of Dover was already reduced due to the suspension of P&O services.
Lorry driver Stuart Orme, who was stuck for several hours, told the BBC people were being "treated like animals" with little or no help.
A spokesperson for the Port of Dover said: "Operators have been working hard all day to process the traffic and get people on their way as swiftly as possible.
"Tourist traffic continues to move through the port and local traffic congestion is easing but still busy."
The Kent Resilience Forum - a partner organisation preparing for emergencies in Kent and Medway - said problems had been made worse by "impatient" lorry drivers attempting to avoid the Dover TAP and Operation Brock traffic management plans and causing a bottleneck in the town.
A section of the M20 is being used to park lorries, leading to long tailbacks and under Operation Brock, the motorway remains closed to anything other than freight between junctions eight and nine.
The Department for Transport said efforts are being made to minimise disruption.
Operation Brock's concrete barrier, found between Ashford and Maidstone on the M20, was originally introduced to keep traffic moving amid fears that a no-deal Brexit plan would lead to delays at the border.
Lorry driver Stuart Orme told BBC South East on Saturday he had been stuck between junctions eight and nine of the M20 for more than six hours and had moved less than 0.6 miles (1km) in that time.
"We're being treated like animals. No welfare, no toilet facilities and no information. We're just left here," Mr Orme said.
"We're trying to get to the Ashford truck stop because we won't get over [to their destination] today. They won't let us go down the middle of the queue to get to Ashford to use facilities."
Another freight firm Logistics UK told the BBC it had built in plans for a 15-hour wait at Dover to cross the channel, and again at Calais on the way back.
Toby Howe from the Kent Resilience Forum said it had been prepared for disruption caused by P&O Ferries suspending services, but the stormy weather was "the straw that broke the camel's back" and blamed "impatient" HGV drivers for causing a bottleneck in Dover.
"There is a traffic management plan in place which controls freight getting to Dover. Ordinarily, that plan would have worked, but because we then had lorries getting very impatient and avoiding that, that's what caused the problem.
"It's all heading straight to the port and there isn't any other outlet for it. So that's the problem. Once it happens it's very hard to undo."
NHS nurse Matthias Mueller left his home in London with his family at 05:00 BST to try to be among the first to swap his P&O ticket for a DFDS one at the port.
"We are going to Germany to see my parents for the first time in two years. We thought this would be an early start to the Easter holidays, it's come to a standstill," Mr Mueller said.
Nick Gale, a teacher from Kent travelling to Amsterdam with family, criticised the "awful" communication around Dover after missing the ferry they were booked on.
"We've got no food and an eight-year-old in the back moaning," Mr Gale said.
"Around Dover it's awful, there is no communication for what non-freight customers [are] to do. We're local to the area so knew a couple of ways to beat the huge queues but it's literally not moving."
Dover MP Natalie Elphicke said: "The traffic disruption caused by P&O's actions is very serious", adding that the adverse weather and Easter break had made the situation "severe" and likely to continue for days.
"The problems have caused an unacceptable impact on our community, particularly at Whitfield, Aycliffe, Capel-Le-Ferne and in Dover town itself."
Some schools have already broken up for Easter and an increase in traffic was expected as many families made the most of the easing of travel restrictions.
P&O services have been reduced after the company sacked 800 workers last month. It has yet to be given permission to resume sailings to France using cheaper agency staff.
The shortage of cross-Channel services was made worse after a DFDS ferry hit a berth in Dunkirk in strong winds, leading to that being taken out of service for repairs.
It will not be available until Monday at the earliest.
Travel journalist Simon Calder described it as a "perfect storm" for Easter travellers.
With P&O's three vessels at berth in Dover, he said DFDS services had been coping well with the extra passengers until Saturday.
A "surge in demand" had been coupled with difficult weather conditions in the Channel overnight, he said.
"It's very difficult to see how that will be relieved," he added, with bookings also high at the Eurotunnel terminal at nearby Folkestone.
Mr Calder said disruption was affecting airports as well with Easyjet cancelling flights to Mykonos, Copenhagen and Corfu at Manchester Airport.