About a dozen people were told they could not fly from Manchester on Saturday as e-tickets covering flights and the Hajj were not issued in time, the Council of British Hajjis said.
They had to return home as they wait to hear if flights can be rearranged.
The BBC has contacted the Saudi Embassy for a response.
The country recently launched an online portal where Muslims from Europe, North America and Australia must register if they want to take part in this year's Hajj pilgrimage, which is due to start in July.
Selected applicants received booking confirmations for the packages in the past few weeks, although the system - known as Motawif - has prompted several complaints, including from some who had previously booked with separate operators and were concerned they could lose money.
Many pilgrims save for years, however the costs - ranging from £6,000 to more than £10,000 per person in the UK - often draw criticism about over-priced packages.
Rashid Mogradia, chair of the Council of British Hajjis (CBHUK), said Saudia staff at Manchester had been "very helpful" but it was "heart-breaking" to see some people unable to board the plane.
He said about 250 travellers, including England cricketer Adil Rashid, were able to fly out.
* It is an annual pilgrimage to Islam's holiest sites in and around Mecca, where Prophet Muhammad was born
* Pilgrims imitate the rites of his ancestral predecessor Abraham - believed by Muslims to have built the cubic shrine known as the Kaaba - and his family
* Hajj is one of the five key pillars of the Islamic faith and is compulsory for any adult Muslim who is physically and financially eligible
* Many Muslims also perform a shorter Umrah pilgrimage, which - unlike the Hajj - is optional and can happen throughout the year
* Prices for the tour packages usually cover flights, transport, accommodation, catering and visa services
Some applicants have told the BBC that they had received mixed messages about whether they could travel this year.
Ali Haider said he and his wife had returned to their Sheffield home after being unable to fly from Manchester on Saturday, despite paying £18,500 for their total package.
He said they had received booking and visa confirmations in the past week, adding: "We assumed everything was sorted, but we were never emailed the e-ticket."
Mr Haider spoke to a Motawif representative on the phone from the airport, but said "it was astonishing how inept they were".
The organisation eventually apologised to Mr Haider on Twitter and said it planned to "rearrange some flights due to high demand for plane seats on some dates".
"It feels to us like they are kind of unanswerable to anybody," Mr Haider said.
"It would be unthinkable that a UK company would do anything like this. I think the whole process has been run badly."
On Monday, Motawif tweeted that it was aware of problems in a number of cases related to booking confirmations and was attempting to contact those affected within 72 hours.
The BBC has attempted to contact the organisation.
British and other Western pilgrims could previously book via Saudi-licensed agents in their home countries.
Prof Sean McLoughlin, from the University of Leeds, has been researching the Hajj industry since 2011, and said the new system was "part of Saudi Arabia's strategy to centralise and simplify packages sales".
The kingdom has said it will host a million pilgrims this year as it accommodates foreign visitors for the first time since the coronavirus pandemic.
However it is not clear how many Hajj visas are being allocated to the UK, which was designated about 25,000 in 2019. Prof McLoughlin said figures have ranged from 3,000 to 12,000.
"While the previous system was sometimes exploited by fraudsters, the best agents sought to make the complexities of Hajj easier for their pilgrims," he said.
"Motawif's failed miserably as it's come far too late - it's not been piloted or trialled, and it hasn't proved as flexible or competitively priced as first claimed.
"The other problem is that Motawif is not communicating clearly and is contradicting itself - they don't seem to have the capacity to deliver and their technical system has let them down badly."
He said the process was "at crisis point" as people were "turning up at airports and being told they could not travel".
A number of people have contacted the BBC with concerns that their bookings may not be fulfilled.
Yasmin Qureshi MP, who chairs the All Party Parliamentary Group on Hajj and Umrah, had previously called on the Saudi government to delay the new system until next year.
She warned it was "not going to work" and asked Hajj minister Tawfiq bin Fawzan Al-Rabiah if refunds would be given.
The UK's Foreign and Commonwealth Office has said it is "aware that not all problems have been resolved and continue to seek information from the Saudi authorities".