Earlier this month, Lisa Smith, from Newmarket, said Hancock going on to I'm A Celebrity was "a disgrace", that he did a "poor job" representing constituents and did not deserve to be their MP.
Now the contest is over, she believes he has not redeemed himself.
"For me it's not just about the Covid situation, it's about the way he has treated his constituents, both before and after the pandemic," she says.
"Him disappearing and going off to film reality TV shows is not dedicating his time to his constituents who needed someone here dealing with issues that he is being paid to deal with.
"Sadly I think he will come back and carry on like it's all perfectly normal, however I don't think he should come back to being an MP for West Suffolk, whether that be as a Tory [Mr Hancock has lost the Conservative whip for going on the show] or as an independent.
"He should walk away and let someone else take over."
When Mr Hancock went into the jungle, Stuart Dillon, from Haverhill, told the BBC he had been a poor MP, which he held with a majority of 23,194 in the 2019 general election.
Reflecting on the MP's third-place finish, the 47-year-old says: "I didn't watch the show but everyone voting for him to do the tasks did him a huge favour.
"Nothing has changed for me because he still left his constituents without parliamentary representation, all for fame-chasing.
"Whether he survives as an MP most likely will depend on whether the whip is restored.
"If it is, the Tories will rally around him and this is a safe seat for that party.
"Personally I think he should resign as an MP and I would be saying that regardless of what party he was in - an MP is paid very well and that's the job he should be doing."
David Venables, 73, sitting in a cafe in Mildenhall, says he enjoyed watching his MP in the jungle.
"I think Matt has done very, very well, but he should have been here laying a wreath on Poppy Day," he says.
"I think he's proved his point, he's only an normal human being and he's done very well in there and I say good luck to him.
"He took everything that they threw at him.
"It would have been nice if he had let people know where he was going, then perhaps other people would have taken it the same [as me].
"A lot of people have been very disappointed he wasn't here."
Asked if he would be pleased to see him back at work as an MP, he says: "Why not? He did a lot of a good when he was here, so I haven't got a problem with that at all."
Ian Houlder, local Conservative Association member and West Suffolk district councillor, has called for Mr Hancock's deselection.
He said he was "absolutely speechless" when he learned his MP was going into the jungle, and described him as a "damn awful role model".
"He's letting down his party, his colleagues in Parliament who are doing a difficult job," he says.
"We're not in happy times in the country, whether you're Labour, Green, or whatever, everyone's in the same boat - we're fiscally up the creek with very few paddles to pull on.
"The whole thing [I'm a Celebrity] is false - it's staged, it's an act.
"It's that attitude that annoyed me - not that he was going to do a game show - just don't go out there to do something you're not paid to do."
He says Mr Hancock "broke the trust of the public" and should have resigned as MP, not just as a secretary of state, when he broke his own social distancing rules during the Covid lockdown.
"I think he should say 'I've had a good innings, I'm not going to stand at the next election'.
"But let's be truthful - he's got little honour, little principle, little integrity and little probity - and he's shown he doesn't care about the constituency.
"Do the right thing and step down, or tell your constituents you're stepping down at the next election.
"If you think you're going to be a media star, great, but you shouldn't be my representative."
* The father-of-three was first elected MP for West Suffolk in 2010
* He was appointed health secretary in 2018, before overseeing the nation's response to the Covid pandemic with lockdowns, social distancing and measures to protect the NHS and social care from March 2020
* However, he was caught breaking his own social distancing rules and resigned in June 2021 when an affair with an aide was exposed
* He has had the party whip suspended - a disciplinary move in parliament - which means he has been kicked out of the Conservative Party, but remains an MP
* It means Mr Hancock will still receive his £84,144 salary while in Australia
In June last year, Mr Hancock resigned as health secretary after he was caught breaking coronavirus social distancing rules by having an affair in his ministerial office with Gina Coladangelo, who was his aide at the time. She was there to greet him as he left the jungle camp after losing Sunday's vote.
After he lost the Tory whip campaign groups 38 Degrees and Covid-19 Bereaved Families for Justice demanded his removal from the jungle.
He has previously stressed the first thing he will do after leaving Australia will be to return to West Suffolk and hold a surgery with his constituents.
Speaking to BBC Look East, Business Secretary Grant Shapps criticised his fellow MP's decision to join the show, as had Prime Minister Rishi Sunak at the time.
"I think it's right for people to expect their MP to be in Parliament and he will be back, I assume, next week," he says.
"If you're elected you need to be in touch with your constituents and that's why it's inappropriate to disappear off for several weeks without that contact.
"It's good that he's coming back, but he shouldn't have gone in the first place."