John Baron, David Simmonds, Stephen Hammond and Julian Sturdy have joined a list of Conservative MPs urging the prime minister to resign.
Mr Hammond said he "cannot and will not defend the indefensible".
Mr Johnson has said he is "humbled" by Ms Gray's findings, but did not lie to MPs and is not going to quit.
In a news conference on Wednesday, he said he wanted to "keep moving forward" and focus on the "people's priorities".
Labour's shadow levelling-up secretary, Lisa Nandy, accused the prime minister of "passing off responsibility" for Partygate to junior staff, telling the BBC: "He's not sorry that he did it, he's sorry that he was caught."
But Downing Street's chief of staff, Stephen Barclay, said the prime minister had made a "significant change" to No 10 by shaking up his team and apologising for the lockdown-busting events.
Mr Johnson's cabinet colleagues have rallied to his defence, with Chancellor Rishi Sunak telling the BBC he trusted the prime minister "100%".
"Not only has he apologised and taken responsibility, he's acted and learnt the lessons," he added.
Conservative MPs can force a leadership contest if enough of them write letters of no confidence - the BBC is aware of about 18 who have done so, well short of the 54 needed.
But some may keep their letters private. Only the chairman of the 1922 Committee of backbench Conservative MPs, Sir Graham Brady, knows the precise number.
Mr Baron and Mr Simmonds went public with calls for the PM to resign on Thursday morning.
They released statements hours before Chancellor Rishi Sunak unveiled a new package of measures to help households with rising living costs.
Veteran Brexiteer Mr Baron said "the most serious charge" against the PM was the allegation that he knowingly misled Parliament about parties in Downing Street.
"Given the scale of rule-breaking in No 10, I can not accept that the prime minister was unaware," Mr Baron said.
"Therefore, his repeated assurances in Parliament that there was no rule-breaking is simply not credible."
The prime minister faces an inquiry by the Commons Privileges Committee about whether he lied to MPs. Under government guidelines, ministers who knowingly mislead Parliament are expected to resign.
In his statement, Tory MP David Simmonds said he had reflected on what the prime minister had said about Ms Gray's report, and taken on board the views of his constituents.
He said it was "clear that while the government and our policies enjoy the confidence of the public, the Prime Minister does not".
The latest MP to announce his move was Mr Hammond, a former minister, who indicated he had sent a letter of no confidence in Boris Johnson to the 1922 Committee.
The Wimbledon MP said since December he had been "critical of the prime minister's behaviour and the culture that existed in Number 10".
Sue Gray's conclusions were "damning " for the PM and the Civil Service, he added.
Mr Hammond said the Conservative party "cannot move on without regaining public trust and I am not sure that's possible in the current situation".
"All I can do as a backbencher is speak out and submit a letter."
Earlier another Conservative MP, John Stevenson, told the BBC that he and his colleagues would make "final judgements" on Mr Johnson's future in the next two to three weeks.
Mr Stevenson suggested that Tory MPs were asking themselves: "Can he change? Can he regain the public's trust? Is he the right person to lead?".
On Wednesday evening, backbencher Julian Sturdy said Ms Gray's report showed Mr Johnson "has presided over a widespread culture of disregard for the coronavirus regulations".
The long-awaited report by senior civil servant Sue Gray detailed examples of excessive drinking, mistreatment of cleaners and security staff and repeated Covid rule-breaking during the pandemic.
Addressing MPs on Wednesday, Mr Johnson said he took "full responsibility for everything that took place on my watch".
He told MPs that when he had previously said "the rules and guidance had been followed at all times", it had been "what I believed to be true".
He said he had also apologised personally to Downing Street cleaners and custodians for the "unacceptable" behaviour of some of his officials.
Ms Gray's report followed the conclusion of a separate Metropolitan Police investigation into lockdown parties in Downing Street and on other government offices.
The force handed out 126 fixed penalty notices for rule breaches, with the prime minister receiving a single fine for attending a birthday party in the Cabinet Room in June 2020.