Mr Wright said the controversy over lockdown parties in Downing Street had done "lasting damage" to the party.
The scale of Covid rule-breaking within government was laid bare in a report by senior official Sue Gray last week.
Eleven Tory MPs have publicly urged the PM to quit since it was published, bringing the total number to 27.
Most of the 27 have submitted letters of no confidence in Mr Johnson's leadership, including Elliot Colburn, who was elected in 2019.
On Monday Mr Colburn, who won his South London seat of Carshalton and Wallington from the Lib Dems by 629 votes, confirmed he had taken that step, followed by his Conservative colleague Andrew Bridgen.
In a statement emailed to his constituents, Mr Bridgen said "it may well be the numbers are close to triggering a vote of no confidence".
Others may have submitted letters without without making it known publicly.
Only Sir Graham Brady, chairman of the 1922 Committee of backbench Tory MPs, knows the exact number, and he is not expected to say anything until he has received 54 letters. That is the threshold that needs to be reached to trigger a confidence vote.
Mr Wright, who was attorney general under former Conservative prime ministers David Cameron and Theresa May, is among the most senior Conservatives to break ranks with Mr Johnson.
The barrister was culture secretary for a year until Mr Johnson took office in July 2019, when he was replaced in a cabinet reshuffle.
In a lengthy statement on his website, the Conservative MP said that the controversy around gatherings in Downing Street and Whitehall while Covid restrictions were in place "have done real and lasting damage to the reputation not just of this Government but to the institutions and authority of government more generally".
Mr Wright went on to say: "Putting that right matters hugely to the essence of government authority and to the effectiveness of government policy, and I cannot see that the moving on of civil servants or apologies, however heartfelt, will succeed in doing so."
Mr Wright said that "for the good of this and future governments, the prime minister should resign".
The MP, who campaigned for Remain ahead of the EU referendum in 2016, is a member of the Committee on Standards in Public Life, which advises the prime minister on ethics within government.
The committee made recommendations for reforms to the ministerial code, some of which were implemented in a contentious shake-up of the government guidelines unveiled last week.
The reforms came as Mr Johnson faces an inquiry by another committee into whether he misled MPs about lockdown-busting parties in Downing Street.
Cabinet ministers have rallied to Mr Johnson's defence since the publication of Mr Gray's report, while other Tory MPs have not backed calls for his resignation.
Conservative MP Anthony Browne told the BBC that while the revelations about parties had been "very painful", the prime minister had got the "big measures right" on the pandemic and the war in Ukraine.
Jeremy Wright's lengthy statement clearly isn't a knee-jerk reaction to the Sue Gray report.
Stretching to more than 2,000 words, the statement seems at pains to make clear he has considered his position before calling on the prime minister to quit.
He says in his view there is not yet enough evidence to conclude the prime minister deliberately misled parliament about parties - which is perhaps one of the most serious charges he faces in the eyes of many Tory MPs.
However, Mr Wright says the whole saga has done such damage to the government's reputation that it could hinder its ability to respond to the country's challenges.
That's ultimately what might prove most dangerous for the prime minister.
No 10 might hope people can "move on" from the ins and outs of Partygate itself, but if enough Conservative MPs conclude it has done lasting damage to the credibility of government, then the trickle calling on Mr Johnson to go might not dry up any time soon.
Meanwhile, Labour has called for the publication of text messages sent by Mr Johnson's wife about a potentially rule-breaking event in the Downing Street flat.
The Sunday Times has reported there is evidence Carrie Johnson organised a gathering for the PM's 56th birthday in their flat, on 19 June 2020.
There was no mention of this gathering in Ms Gray's report, which only referenced Mrs Johnson once in connection with a birthday party for the PM in the Cabinet Room earlier that day.
The long-awaited report by Ms Gray detailed examples of excessive drinking, mistreatment of cleaners and security staff, and repeated Covid rule-breaking during the pandemic.
Ms Gray's report followed the conclusion of a separate Metropolitan Police investigation into 12 lockdown parties in Downing Street and on other government offices.
The force handed out 126 fixed penalty notices for rule breaches, with the prime minister receiving a single fine for attending a birthday party in the Cabinet Room in June 2020.
In Parliament last week, Mr Johnson said he took "full responsibility for everything that took place on my watch".
He told MPs that when he had previously said "the rules and guidance had been followed at all times", it had been "what I believed to be true".
He said he had also apologised personally to Downing Street cleaners and custodians for the "unacceptable" behaviour of some of his officials.