Downing Street said reductions in staff were needed but the government would not be putting a specific number on it.
The 91,000 target was introduced when Boris Johnson was prime minister.
The shift in policy comes as Mr Sunak's government seeks to shore up the UK's finances to fill a gap worth an estimated £50bn.
The Treasury has warned of spending cuts and tax rises ahead of a financial statement by Chancellor Jeremy Hunt on 17 November.
The statement was delayed by two weeks after Mr Sunak replaced Liz Truss as prime minister, following a period of economic turmoil, exacerbated by her tax-cutting mini-budget in September.
The target of reducing the Civil Service by around a fifth predates both Ms Truss and Mr Sunak.
Jacob Rees-Mogg, who has now quit government and returned to the backbenches, was working towards the target when he was a Cabinet Office minister under Mr Johnson.
At the time, Secretary Simon Case sent a letter to civil servants, telling them the aim was to return to 2016 staffing levels within three years.
There were 384,000 civil servants employed in 2016 - the lowest number since World War Two - but as the UK prepared to leave the EU, numbers steadily rose until they reached 475,000 at the end of last year.
But on Tuesday, Mr Sunak's official spokesman said reductions in civil servant numbers would be up to individual departments, and no targets would be set.
In a tweet, the Trades Union Congress wrote: "Big win for civil servant trade unions."
Writing to civil servants, Mr Sunak said: "Together, we must make sure every taxpayer pound goes as far as it possibly can.
"I do not believe that top-down targets for Civil Service headcount reductions are the right way to do that.
"Instead, the chancellor and I will be asking every government department to look for the most effective ways to secure value and maximise efficiency within budgets so that we can use taxpayers' money sustainably in the long term."
The Cabinet Office said government departments "have been asked to look for the most effective ways to maximise efficiency within their budgets".
A spokesperson said: "It is the role of a responsible government to identify how to deliver the best outcomes for the public as efficiently as possible."
The spokesperson also confirmed the Fast Stream programme to recruit civil servants would resume next month after a pause.
Jacob Rees-Mogg: "During an issue with the cost of living... you need to have control of budgets"