The PM was speaking during a trip to Scotland to emphasise the strength of the UK working together during the pandemic.
The SNP said he was panicking as opinion polls show declining support for the union.
First Minister Nicola Sturgeon also questioned if his trip is essential.
The PM started his day-long visit by going to the Lighthouse Laboratory - which processes Covid tests - at the Queen Elizabeth University Hospital campus in Glasgow.
He later visited troops who are setting up a vaccination centre in the Castlemilk area of the city, and toured the Valneva vaccine factory in Livingston.
The factory is expected to deliver 60 million doses to the UK by the end of the year if its vaccine is approved.
Mr Johnson used the visit to argue that the priority should be "fighting this pandemic and coming back more strongly together" rather than arguing about the constitution.
And he praised the "amazing performance" of Scottish people in the "national effort" to fight the pandemic.
The prime minister said: "I think endless talk about a referendum without any clear description of what the constitutional situation would be after that referendum is completely irrelevant now to the concerns of most people".
Mr Johnson also criticised the SNP's record in government, and added: "We don't actually know what the referendum would set out to achieve.
"We don't know what the point of it would be - what happens to the army, what happens to the Crown, what happens to the pound, what happens to the Foreign Office. Nobody will tell us what it's all meant to be about."
He told reporters that "the very same people" who wanted independence "also said only a few years ago, in 2014, that this was a once-in-a-generation event".
"I'm inclined to stick with what they said last time," Mr Johnson said.
Under the current Covid regulations, people are only able to travel between Scotland and England for essential reasons, with similar regulations also in place to stop travel across council boundaries within Scotland.
Asked at her daily coronavirus briefing on Wednesday how she felt about the prime minister's visit while the strict travel restrictions were in place, Ms Sturgeon replied she was "not ecstatic" about it.
She argued that leaders should abide by the same rules they impose on the general public, adding that she had herself rejected a suggested visit to a vaccine centre in Aberdeen for this reason.
Downing Street has insisted it is important for the prime minister to be "visible and accessible" across the whole of the UK during the pandemic.
In response to Ms Sturgeon's criticism, the prime minister's official spokesman said: "These are Covid-related visits. You've seen the prime minister do a number of them over the past few weeks.
"It is obviously important that he is continuing to meet and see those who are on the front line in terms of those who are providing tests, in terms of those who are working so hard to deliver the vaccination plan."
Mr Johnson's visit to Scotland is widely seen as being part of a "charm offensive" in response to polls indicating a rise in support for independence.
However, polls have also suggested that the independence question is currently a lower priority for many people than other issues such as the pandemic, health and education.
Cabinet Office Minister Michael Gove said it was "only right" the prime minister visited people on the front line of the vaccine roll-out to make sure it is operating effectively.
He told BBC Breakfast Mr Johnson has visited other crucial locations in the UK's pandemic response, such as the Wrexham plant making the Oxford-AstraZeneca vaccine, adding: "No one thinks that's illegitimate."
Labour leader Sir Keir Starmer also said he backed the visit. "I'm with the prime minister on this one," he told LBC Radio.
"He is the prime minister of the UK. It's important that he travels to see what is going on, on the ground."
It comes as the Scottish government sets out its budget, described as the "most important in the history of devolution" in the wake of huge spending increases to support people and businesses during the pandemic.
Boris Johnson had a clear purpose on his visit to Scotland - to talk up what he calls the power of cooperation across the UK.
Dressed in white lab coat and protective gear, he was happy to tell me how the UK government is supporting the fight against coronavirus in Scotland.
That includes spending lots of money supporting jobs and businesses, building test centres, and procuring vaccine supplies from companies like the one he was visiting in Livingston.
But here's the paradox.
No matter what the prime minister does, or that the UK and Scottish governments are following broadly similar Covid strategies - the public in Scotland perceives that Nicola Sturgeon and her team are handling the pandemic response better.
This visit was controversial because it happened during lockdown but it went ahead because the UK government recognises how much work it has to do to make the case for the union in Scotland, with Scottish elections due in May when the question of indyref2 will be to the fore.
On Sunday, the SNP revealed an 11-point "roadmap to a referendum" on Scottish independence, which sets out how the party intends to take forward its plan for another vote on the issue.
It says a "legal referendum" will be held after the pandemic if there is a pro-independence majority at Holyrood following May's election.
And it says it will "vigorously oppose" any legal challenge from the UK government.
Mr Johnson has repeatedly stated his opposition to a referendum, and has suggested that another one should not be held for 40 years.
Opposition parties in Scotland have also accused Ms Sturgeon and the SNP of putting the push for independence ahead of the Covid pandemic.
But SNP deputy leader Keith Brown said the prime minister's trip was evidence that he is in a "panic" about the prospect of another referendum.
The most terrifying words in the English language are: I'm from the government and I'm here to help.