Amid testy exchanges at Prime Minister's Questions, Labour MP Dame Angela Eagle told Mr Johnson he was loathed and should resign.
But the prime minister said his political career had "barely begun".
Four in 10 Tory MPs declared a lack of confidence in Mr Johnson's leadership in Monday's vote.
The worse-than-expected result followed several months of difficulties for the prime minister, who has received ongoing criticism over parties that took place in Downing Street during lockdown.
Following the vote, the prime minister promised to "draw a line" under his recent difficulties and announce a widespread programme of reforms to help deal with crime, the NHS and the rapidly rising cost of living.
His ministers have stood by him, but in the first departure from government since the vote, Baroness Helena Morrissey left her job at the Foreign Office on Wednesday.
In an interview with LBC radio, Baroness Morrissey said some Tory MPs considered the prime minister "a liability" and she had not seen "any contrition" from him after the confidence vote.
The BBC understands Foreign Secretary Liz Truss found Baroness Morrissey's comments "outrageous" and the peer "jumped before she was pushed".
Meanwhile, at a noisy Prime Minister's Questions, Mr Johnson reiterated his commitment to remain in power.
Asking the first question, Dame Angela said: "This week's events have demonstrated just how loathed this prime minister is, and that's only in his own party.
"As his administration is too distracted by its internal divisions to deal with the challenges we face, can the prime minister explain if 148 of his own backbenchers don't trust him why on earth should the country?"
Mr Johnson replied: "I can assure her in a long political career so far - barely begun - I've of course picked up political opponents all over and that is because this government has done some very big and very remarkable things which they didn't necessarily approve of.
"And what I want her to know is that absolutely nothing and no-one, least of all her, is going to stop us with getting on delivering for the British people."
Labour leader Sir Keir Starmer focused his questions on the government's record on the NHS, while the SNP's Westminster leader, Ian Blackford, called Mr Johnson, who was fined in April for attending an illegal Downing Street gathering during lockdown, a "dead man walking".
He accused the prime minister of "failing to recognise that these parties were against the law, it's that signal that the rules don't apply to him the way that they do for everyone else".
"That's not the behaviour of someone who can remain in office, its someone that quite simply needs to be removed from office," Mr Blackford added.
The prime minister's victory in the confidence vote means that, under Conservative Party rules, he cannot face another such test of his leadership for another year.
But Mr Johnson is coming under pressure from some of his own MPs to cut taxes in an effort to boost the economy and help ease the burden on households caused by the rapidly rising cost of living.
Business Secretary Kwasi Kwarteng, Health Secretary Sajid Javid and trade minister Penny Mordaunt all said they would like this to happen.
But the prime minister's spokesperson played down the immediate prospect of this happening, saying it was important to be "responsible" at a time when finances were in "very difficult position following the global pandemic".
They added that plans for further tax cuts would be set out "as soon as it is fiscally responsible to do so".
There were no unfriendly questions from Tory MPs about his future to the PM in the chamber, but in a column in the Times newspaper, Conservative former cabinet minister David Davis accused Mr Johnson of "letting things slide" and warned him to "get his act together" quickly or leave office.