In his first foreign policy speech, Mr Sunak said the closer economic ties of the previous decade had been "naïve".
The prime minister said the UK had to replace wishful thinking with "robust pragmatism" towards competitors.
But he warned against "Cold War rhetoric", adding that China's global significance could not be ignored.
Mr Sunak has faced pressure from Tory backbenchers to toughen the UK's stance on China since he took over as prime minister last month.
His speech, at the Lord Mayor's Banquet in London, comes after protests in China over the weekend against the country's strict Covid lockdown laws.
Police have made several arrests, and a BBC journalist was detained while covering a protest in Shanghai on Sunday. He was beaten and kicked by the police during his arrest and held for several hours before being released.
Mr Sunak told the audience of business leaders and foreign policy experts that, in the face of the protests, China had "chosen to crack down further, including by assaulting a BBC journalist".
"We recognise China poses a systemic challenge to our values and interests, a challenge that grows more acute as it moves towards even greater authoritarianism," he said.
He added that the "golden era" of UK-China relations was "over", along with the "naïve idea" that more trade with the West would lead to Chinese political reform.
The phrase "golden era" is associated with closer economic ties under former Prime Minister David Cameron - but relations between London and Beijing have since deteriorated.
However, Mr Sunak stressed that "we cannot simply ignore China's significance in world affairs - to global economic stability or issues like climate change".
He added that the UK would work with allies including the US, Canada, Australia, and Japan to "manage this sharpening competition, including with diplomacy and engagement".
"It means standing up to our competitors, not with grand rhetoric but with robust pragmatism," he added.
Mr Sunak and Chinese President Xi Jinping were set to meet for the first time at the G20 summit in Indonesia earlier this month, but the encounter was cancelled following a missile blast in Poland.
Mr Sunak's predecessor Liz Truss was reportedly planning to re-categorise China as a "threat" to the UK as part of a review of its foreign policy.
In his speech, Mr Sunak echoed the phrase used in the review - that China is a "systemic challenge". He said there would be more details of the review in the new year.
But the "robust pragmatism" line in the speech was criticised by former Conservative Party leader Sir Iain Duncan Smith - one of a number of backbenchers pushing for a tougher line.
During an urgent question in the Commons on the safety of British journalists in China, he asked the Foreign Office minister to explain "how 'robustly pragmatic' will worry the Chinese any one bit?"
Labour's shadow foreign secretary David Lammy called the speech "thin as gruel", accusing the government of "flip-flopping its rhetoric on China".
Nigel Inkster, senior China advisor at foreign affairs think tank the International Institute for Strategic Studies, told BBC Radio 4's Today programme he did not think the China-UK golden age "was ever real and substantial".
He said: "It attempted to focus on economic relations with China while putting geopolitics to one side, and experience shows you simply can't do that."
Renewed debate about the UK's relationship with China comes as members of the Commons Foreign Affairs Select Committee have travelled to Taiwan - which Beijing claims as it territory - to meet its president.
The committee is holding an inquiry into UK foreign policy in the Indo-Pacific region and how it can maximise its influence there.
Watch: BBC journalist Ed Lawrence arrested covering China Covid protests