Bereaved families have accused Boris Johnson of showing “flagrant disregard” for the public as ministers struggled to explain the justification for a wine and cheese event in Downing Street at the height of lockdown.

A Cabinet Office inquiry into other alleged government parties in breach of Covid rules could be expanded after the Guardian published an image showing the prime minister alongside his wife and up to 17 staff in the Downing Street garden in May 2020.

Amid growing fury over the photograph, Johnson said on Monday: “Those were meetings of people at work, talking about work.” Labour dismissed Downing Street’s explanation and said it amounted to evidence of law-breaking.

The Covid-19 Bereaved Families for Justice group said the lack of any signs of work, with red wine and a cheeseboard visible rather than laptops or pens, showed Johnson “presided over a culture of believing that the rules applied only to other people”.

In a sometimes confusing series of explanations over the course of Monday, Boris Johnson, his deputy, Dominic Raab, and Johnson’s spokesperson all sought to argue that the 19 people, split into four groups across the Downing Street terrace and lawn, were not gathered socially.

Raab told the BBC that Johnson and his staff were “were having a drink after the formal business had been done” and that this was common practice among other workplaces that had remained open during the lockdown.

Speaking later, Johnson’s official spokesperson stressed that work was in fact continuing during the drinks event, with the photo showing “colleagues who were required to be in work, meeting following a press conference to discuss work”.

Asked why several groups, including Johnson’s, had bottles and glasses of wine, the spokesperson said: “These were individuals working outside of normal working hours. It was not against the regulations for those individuals to have a drink outside working hours, but still discussing work.”

In a TV clip recorded later in the afternoon, Johnson reiterated this argument, saying: “This is where I live, it is where I work. Those were meetings of people at work, talking about work.”

There was also some apparent uncertainty about why the photo showed people standing together, when regulations said they should keep 2 metres apart for work meetings in person. Raab appeared to argue this did not matter as they were outside, while Johnson’s spokesperson said you could not tell from the picture how close together they were.

At a government press conference directly before the May 2020 gathering, the public were told that, socially, “you can meet one other person from outside your household in an outdoor, public place but please keep 2 metres apart”.

The shadow chancellor, Rachel Reeves, told the BBC the photo seemed to show “evidence of law-breaking”. Keir Starmer, the Labour leader, said: “Just look at the photo and ask yourself: is that a work meeting going on, or is that a social event? I think the answer is pretty obvious.”

A leading lawyer and expert on Covid regulations, the barrister Adam Wagner, said he now doubted whether the gathering, particularly as described by Raab, did comply with the rules of the time.

In the first lockdown, Wagner said, people could gather in a non-public space if there was a reasonable excuse such as work. But this did not seem to be the case here, he told Sky News. “Having heard Dominic Raab say that this was a drink after the formal business had ended, after a press conference, doesn’t sound convincing as a work gathering. It sounds a lot like a social gathering.”

After a series of reports about other alleged parties in the winter of 2020, when all indoor social mixing was barred, Johnson commissioned an inquiry into any potential wrongdoing. This was headed by Simon Case, the cabinet secretary, but he stepped down on Friday following allegations that a social event had taken place among his own staff.

Case’s alleged gathering was a day before another alleged party at No 10 on 18 December 2020, which was the original focus of the inquiry. Sue Gray, another senior official, will now lead the inquiry, which has been widened to cover other alleged events.

Asked whether it could be expanded again to examine the photo of the Downing Street garden in May last year, the Cabinet Office pointed to the terms of reference of the inquiry, which say it is able to investigate “credible allegations relating to other gatherings”.