Mr Swann was speaking after the resignation of First Minister Paul Givan.
This was in protest at the Northern Ireland Protocol.
Mr Givan's resignation means the Northern Ireland Executive cannot meet.
The executive - Northern Ireland's government - was due to discuss removing remaining Covid restrictions on Thursday.
However, Mr Swann said he was not "prepared to allow a failure of politics to get in the way of lifting now unnecessary restrictions".
"I am now seeking that legal guidance on how I myself can replace the bulk of the remaining restrictions, if not all of them, with clear guidance and advice," he told a press conference on Monday.
"When we had an executive, ministers always said we wouldn't retain restrictions for a day longer than necessary, so that same logic should still apply.
"While we must move forward carefully, we can't stay locked in emergency restrictions forever."
He added: "We need to start getting normal back and that means moving cautiously out of the current legal phase of the Covid response - I hope this will be a permanent decision.
"The virus remains unpredictable, but we will take one day at a time."
So-called Covid passports remain in place for nightclubs as well as for indoor unseated and partially-seated events with 500 or more people in attendance.
A maximum number of 30 people are permitted to gather.
The minimum self-isolation period for people testing positive for Covid-19 is five full days, subject to negative lateral flow tests on days five and six of a person's isolation.
Other rules still in force include:
* the legal duty on retail to take reasonable measures to reduce the risk of transmission
* the legal requirement to wear face coverings and the associated duty on businesses to take reasonable measures to ensure compliance
* the legal requirement for risk assessments in prescribed settings
* the legal requirement for recording visitor information in prescribed settings
* the remaining legal requirements in relation to Covid-status certification
* the guidance on the regular use of lateral flow testing, and in particular before meeting up with others.
Last month, Northern Ireland ministers had announced a relaxation of Covid-19 restrictions that were introduced in December due to the Omicron variant.
Mr Swann said on Monday that while the pandemic was "certainly not over", it was his hope "that we are through the worst of this current wave".
"Whilst transmission levels are likely to remain high for some time, as a result of the vaccination programme and the high levels of natural immunity we can expect hospital numbers to continue to gradually decline.
"We have real reason to be optimistic."
He said the "story this week" should have been about public adherence to the rules and the "sacrifice of health workers" that had brought about the potential for restrictions to be removed.
However, he added, "the absence of a first or deputy first minister has removed the opportunity for those discussions at an executive meeting".
Mr Swann said removing legal regulations around coronavirus "does not mean a free for all or throwing caution to the wind".
"We can and must keep making safer choices in our daily lives for the sake of others, as well as ourselves."
Two further Covid-19-related deaths were reported in Northern Ireland by the Department of Health on Monday.
Another 3,105 cases of coronavirus were reported in Northern Ireland on Monday, up from 2,694 on Sunday.
The total number of deaths linked to Covid-19 in Northern Ireland since the start of the pandemic is 3,130.