The call comes as part of fresh advice from the Competition and Markets Authority (CMA) amid concerns that consumers are being ripped off.
UK travellers must take PCR tests when they return from abroad, choosing from a list of 400 or more authorised firms.
But the CMA said basic standards should be upgraded and enforced.
It also said consumers should be able to "get the information they need to compare providers properly and find the best deals for them".
Last week, the CMA said it had begun an investigation into one firm, Expert Medicals, following complaints that Covid test results had not been provided on time or at all.
It has also warned a further 19 firms about advertising misleading prices, following a review that found customers often faced poor service, hidden fees and refund issues.
In its latest advice, the CMA said there were features of the PCR testing market which meant competition alone would not deliver the right outcomes for consumers.
"There is a risk of a 'race to the bottom', in which providers compete on grounds other than high clinical quality and travellers end up losing out," it added.
As a result, it said, it was making a number of recommendations to the government, including:
* Creating a one-stop shop list of quality, approved test providers by significantly improving the basic standards to qualify for the government list
* Introducing a comprehensive monitoring and enforcement programme to ensure providers on the list meet these basic standards and swiftly removing and sanctioning those that do not
* Developing the NHS Test and Trace travel test as a benchmark for quality and price to drive higher standards and more competition
* Monitoring prices and costs on a continuing basis
Tests cost about £75 per person on average in the UK, but there are reports of some being offered for as little as £20, and others for more than £500.
There have also been reports that some government-authorised providers do not have full accreditation.
The government has already issued warnings to 82 firms that they could be removed from its list of authorised providers if they advertise misleading prices.
Rory Boland, Which? travel editor, said the CMA's review "reinforces Which?'s repeated warnings to the government that the current private testing system is not fit for purpose".
"The government must now urgently set out how it will implement the regulator's recommendations and ensure safe, reliable and affordable tests are available for all travellers," he added.
Avi Lasarow, EMEA chief executive of Project Screen by Prenetics, which is on the government list of test providers and provides testing at the major UK airports, welcomed the CMA's findings, saying: "Hopefully this will drive the cowboy testers out of town."
He added: "Holiday PCR testing will be a way of life for travellers for some time to come as we need to protect the UK from new variants entering the country.
"However, the government could help travellers even more by abolishing VAT on holiday PCR tests, so they can travel safer and cheaper."