Edinburgh University's Prof Linda Bauld said there was concern about people travelling from "high to low prevalence areas" to see their loved ones.
Wales' Health Minister Vaughan Gething said changes would cause "huge issues about trust", but could happen if case rates stayed high.
The UK has recorded another 519 deaths within 28 days of a positive test.
The latest government figures also include 21,502 new positive cases in the UK, taking the total number in the past seven days to 124,988.
On Saturday, the number of positive Covid tests in Wales surpassed 100,000, one day after the country reported its highest case rates since the pandemic began.
Up to three households can stay together and form a "Christmas bubble" from 23 to 27 December, as agreed by all four UK nations.
Prof Bauld told BBC Breakfast that "from a public health perspective, I have to be perfectly honest, I think this is a mistake and I think people, even though we're permitted to do this, I think people have to think very carefully whether they can see loved ones outside or do it in a very, very modest way".
She added there was "nothing to stop" governments reversing the rules, "but the problem is they've made that commitment to people across the UK, and that may affect trust in government if they roll back on that".
Mr Gething said "of course we could" change the rules around Christmas, but "much of what we have done during the course of the pandemic is because people have trusted the government - when we said things we kept our word".
He said was concerned a change in rules now would have people "completely ignoring the rules", and was worried that may be the case even with the current five-day relaxation agreement in place.
"That's why we are anticipating an increase after Christmas, we expect there'll be an increase after New Year's Eve as well," he added.
Prof Chris Whitty, England's chief medical officer, earlier this week warned people to be "very, very sensible" and not go "too far" over Christmas, which he called a "very risky period".
According the Scientific Advisory Group for Emergencies (Sage) - which feeds into UK government decision-making - just because people can meet up, it does not mean they should.
Mr Gething said he would not rule out another lockdown if cases continued to rise, echoing a warning made by First Minister Mark Drakeford on Friday.
He added: "But the agreement around the Christmas period isn't just a political settlement. It really is because we understand that sort of shape to that period of time, we might see many people make up their own rules with the real potential of even greater harm.
"So it isn't as simple as saying 'this is just a political choice, and you could make it safer' because actually a lot of people invest lots of time, energy and effort in Christmas in travelling around the UK to see family when they don't do that in the rest of the year.
"Now we have to take account of the reality of the position as well and what would like to happen, how we want people to behave."
When I was a boy I was told that anybody could become President. I'm beginning to believe it.