A breakthrough in the Brexit talks has failed to materialise after a weekend of intensive negotiations, with European Union capitals concluding that it may now be impossible for the UK to leave the EU by 31 October with a deal.

In a briefing to EU ambassadors on Sunday evening, the EU’s chief negotiator, Michel Barnier, raised the prospect of the talks having to continue after the forthcoming leaders’ summit on Thursday, such was the lack of progress.

Barnier told diplomats for the member states that the latest British customs proposals for the Irish border remained an “untested” risk that the bloc could not countenance. He said that it would require a fresh “political impulse” from Boris Johnson for a deal to be realisable this week.

On the UK’s proposed Stormont veto on Northern Ireland staying in the EU’s single market for goods, the Irish representative told the room that such an arrangement was “not a notion that is included in the Good Friday agreement”.

A Brexit extension – whether “technical” if the two sides get close to a deal in late October or longer to accommodate a general election – was raised in the EU27’s discussions for the first time in months.

Johnson had hinted at the problems during an update of his cabinet on Sunday lunchtime. He offered few details, but a No 10 spokesperson said the prime minister had told his colleagues that “a pathway to a deal could be seen, but that there is still a significant amount of work to get there and we must remain prepared to leave on 31 October”.

The lack of progress over the weekend has left scant hope, however, of a decisive moment in the Brexit saga this week.

Barnier said the EU should give Britain “one last chance” to find a realistic replacement for the Irish backstop as he proposed last-ditch talks until the eve of the EU summit on Wednesday.

The German chancellor, Angela Merkel, reminded the UK that it would become a “potential competitor” with the EU in global markets “along with China and the US” after Brexit.

“Europe needs to show what it’s made of” as Brexit looms, she said before a dinner with the French president, Emmanuel Macron, in Paris.

“A lot of work remains to be done,” an official statement from the commission said following Barnier’s briefing for the EU capitals.

Johnson executed a major policy U-turn at the end of last week by accepting that there could not be a customs border on the island of Ireland, raising some optimism on both sides of the channel.

Under his alternative proposal as briefed by Barnier, a pared-down version of Theresa May’s customs partnership, Northern Ireland would leave the EU customs union. But the UK would agree to enforce the bloc’s customs rules and tariffs on goods moving from Britain to Northern Ireland. There would be a rebate system to compensate affected businesses.

Barnier told the ambassadors that it would involve an untried system of tracking goods and he spoke of the risks that would be facing the single market. “There were lots of ifs and maybes – too many,” said one EU diplomat of the discussions.

A second senior EU diplomat warned it would be impossible to hammer out such a “totally new concept” to replace the Irish backstop within days.

The source said the UK would not be able leave the EU with a deal as scheduled on 31 October unless it embraces a previous plan, the Northern Ireland-only backstop – the EU’s original proposal which would leave the region in an EU customs union and following most of the single market’s rules.

If the UK wanted a brand-new concept on customs for Northern Ireland a deal is “impossible” by the end of October, said the diplomat.

Earlier on Sunday, the European commission president, Jean-Claude Juncker, said he would back such a prolongation of the UK’s membership.

“It’s up to the Brits to decide if they will ask for an extension,” Juncker told the Austrian newspaper Kurier on Sunday. “But if Boris Johnson were to ask for extra time – which probably he won’t – I would consider it unhistoric to refuse such a request.”

Despite scepticism in Brussels, the prime minister is still hoping to make sufficient progress at the European council on Thursday and Friday to be able to hold a Commons vote on his Brexit plans in a rare Saturday sitting next weekend.

The Benn act, passed by parliament in a bid to block a no-deal Brexit, obliges the prime minister to request a delay if he has not agreed a deal with the EU27 and secured the backing of the House of Commons for it by 11pm on Saturday.

Backers of the People’s Vote campaign, which will be holding a mass rally outside parliament on Saturday as MPs meet inside, hope to amend any motion tabled by the government to attach a requirement for a referendum – though it is unclear whether a majority exists to do so.

Meanwhile the first independent economic assessment of Johnson’s proposed deal, published on Sunday by the thinktank UK in a Changing Europe, suggests it would be significantly more damaging than May’s plans.

The analysis suggests GDP would be between 2.3% and 7% smaller in a decade’s time than if Britain remained in the EU – compared with a range of 1.9% to 5.5% under the May deal.

Prof Jonathan Portes, a senior fellow at UK in a Changing Europe, said: “Our modelling shows that the much more distant economic relationship with the EU envisaged by Boris Johnson will mean that the economic impacts of Brexit on trade and hence growth will be considerably more severe. The hit to the UK public finances could be up to about £49bn a year. However, these could in part be mitigated by a more liberal immigration policy.”