The Guardian, which published the image, said the gathering took place in May 2020, after a press conference held during the first lockdown.
At the time, there were strict restrictions on mixing.
No 10 said work meetings often took place outside during summer months.
Asked about the image on Monday, Mr Johnson told reporters: "Those were people at work, talking about work."
Pressed on whether it was acceptable for people to be drinking whilst working, he replied: "I have said what I have to say about that."
Earlier, deputy Prime Minister Dominic Raab suggested the event took place after work, telling BBC Breakfast: "Sometimes after a busy working day people have a drink - that was not against the regulations."
Labour leader Sir Keir Starmer said it was "a stretch" to call the gathering a work meeting, pointing to the "contrast" between the picture of the PM and people who were unable to attend funerals at the time because of the lockdown.
An investigation is currently taking place into a number of alleged parties in government buildings - including Downing Street - in November and December last year.
The 15 May photograph shows a gathering of staff in the Downing Street garden following a news conference held by then-Health Secretary Matt Hancock, according to the Guardian.
A No 10 spokesman said: "Work meetings often take place in the Downing Street garden in the summer months.
"On this occasion there were staff meetings following a No 10 press conference."
They added: "Downing Street is the prime minister's home as well as his workplace. The prime minister's wife lives in No 10 and therefore also legitimately uses the garden."
England was in its first lockdown. By law, people were only allowed to be away from their home if they had a "reasonable excuse".
One of the listed excuses was for work, "where it is not reasonably possible for that person to work... from the place where they are living".
Deputy Prime Minister Dominic Raab told BBC Breakfast that the people pictured were having a drink, "after the work meetings of the day".
But, if they weren't working, then the work excuse wouldn't count, except for the prime minister and his partner, who could argue that the garden was part of their home.
The law at the time did not mention socialising with colleagues.
There was also government advice that you should stay at least two metres away from anyone who wasn't part of your household, but that was not a legal requirement.
Mr Raab argued the risk of coronavirus transmission among those present had been lowered "because they were outdoors", saying: "It's clear it wasn't against regulations... this is a workplace, and what we can see is consistent with the rules that applied to workplaces."
He also told BBC Radio 4's Today he believed "someone with an animus" had been leaking details of goings-on in Downing Street last year to the media in order to "damage the government".
But Labour's Sir Keir said: "Everybody will have looked at that photograph and to suggest it is a work meeting a bit of a stretch by anyone's analysis."
Sir Keir added: "Just look at the photo and ask yourself is it a work or social event? It is pretty obvious.
"During that period, there were funerals where very few people could go and mourn the people who died.
"That's the contrast. That picture… up against pictures of those who have lost someone and not even been able to go to the funeral."
The SNP's deputy Westminster leader, Kirsten Oswald, has called for a "judge-led inquiry" to determine what happened.
There is debate about what, if any, rules were being broken when this picture was taken in the Downing Street garden in May last year.
But, as far as many people seeing this photo will be concerned, the details don't matter.
To those who were working hard at the time in difficult conditions, or meticulously following the rules, it won't look good.
Awkward questions for ministers are met with the defence that Downing Street is a place of work and colleagues were having a drink at the end of a long day.
It may not be as damning for some as allegations of a party inside No 10 last Christmas or staff caught on camera laughing about it afterwards.
But it's been seized upon by Boris Johnson's political opponents as further evidence of their claim that it's one rule for those in charge and another for everybody else.
The Guardian pointed out that, in the news conference on the same day as the photo, Mr Hancock had warned the public to stick to social distancing rules, despite the warm weather.
A spokesman for Mr Hancock said he had left the building at 18:30 GMT, after speaking to Mr Johnson in the garden, adding: "There is no suggestion that Mr Hancock did anything wrong."
The investigation into whether alleged gatherings in government buildings in November and December 2020 broke Covid rules is being led by senior civil servant Sue Gray.
She replaced Cabinet Secretary Simon Case, who stepped down from the review after it was revealed he had known about a quiz held in his office.
13 November 2020: Sources have told the BBC there were impromptu drinks to mark the exit of the former director of communications at No 10, Lee Cain - but it was staff having drinks at their desks and was over by 20:30 GMT.
13 November 2020: Also the date Dominic Cummings left as chief adviser to the PM. Sources told the BBC that several Downing Street staff members attended a gathering with Carrie Johnson in the flat where the prime minister and his wife live above No 11 after exits. A source said music was blaring and could be heard elsewhere in the building. But others said to be at the party denied it took place, as did a spokeswoman for Mrs Johnson.
25 November 2020: The Times reported Treasury officials held a drinks party in their office to celebrate Chancellor Rishi Sunak's spending review during the lockdown. The newspaper said two dozen civil servants were understood to have attended the event. But a spokesman for the department insisted it was an "impromptu" event, with a "small number" of staff who celebrated around their desks.
27 November 2020: Another No 10 aide, Cleo Watson, left Downing Street marked by an event. Again, sources said it was not formally organised, but people were having drinks and Mr Johnson made a speech.
10 December 2020: The Department for Education has confirmed it had a gathering in the office to thank staff for their work during the pandemic. Drinks and snacks were brought by those who attended and no outside guests or support staff were invited.
14 December 2020: The Conservatives have confirmed a report in the Times that there was an "unauthorised social gathering" in the basement of their party's building, held by the team of the London mayoral candidate at the time, Shaun Bailey. It was described as "raucous", with No 10 aides
15 December 2020: Multiple sources told the BBC there was a Christmas quiz for No 10 staff in the Cabinet Office, with invites sent out in advance via email, telling people to form teams of six. Downing Street have said it was a "virtual gathering" but while some joined in that way, sources said there were groups in the room sat in their teams. This is the event the Sunday Mirror released a picture of the PM attending virtually.
17 December 2020: The BBC understands a quiz was held for members of Cabinet Secretary Case's private office, and invitations were sent out beforehand titled "Christmas Party!" About 15 people were invited to the gathering at 70 Whitehall, near Downing Street, although it is understood not everyone turned up.
18 December 2020: A Downing Street Christmas party took place on this date, as first reported by the Mirror. A source told the BBC there was food, drinks and games at the gathering that went on past midnight.
DWP gatherings: The Department for Work and Pensions confirmed a "core team" of staff drank alcohol and ate takeaways "on a number of occasions" whilst working late in the office while Covid restrictions were in place. It followed a report in the Sunday Mirror alleging political staff and officials working for minister Therese Coffey frequently drank after work until the early hours of the morning.