The lobby group has now dismissed a number of people after a series of allegations including two rapes.
But Baroness Morrissey, a well-known City fund manager who currently chairs investment firm AJ Bell, said the actions were "too little, too late".
CBI president Brian McBride has acknowledged it needs to rebuild trust.
There has been a mass exodus of members, with household names quitting the group including John Lewis, BMW, Virgin Media O2, insurers Aviva, Zurich and Phoenix Group, banking firm Natwest, credit card company Mastercard; B&Q owner Kingfisher and media firm ITV.
Other firms have said they are "pausing" their engagement with the lobby group while many more called by the BBC are yet to comment on their standing with the organisation.
Asked on the BBC's Today programme whether she thought the CBI as an organisation is finished, she said: "I do I'm afraid."
"They've left it too little too late and I think that losing trust is so quick, easy to do and then regaining it is so difficult."
Baroness Morrissey, who campaigns for female inclusion within company management, also criticised firms who have yet to publicly leave the CBI.
"I don't quite understand why people don't come out and say: 'We're going to join those who have decided to leave and we'll work out what we're going to do in replacement of the CBI', because I feel they will be on the wrong side of history," she added.
On Monday, the CBI detailed the actions it was taking to rebuild trust in the group after a report from an independent law firm found it had hired "toxic" staff and allowed a "very small minority" of staff to believe they could get away with harassment or violence against women.
The CBI has been engulfed in crisis since the Guardian published claims of workplace misconduct, harassment and serious sexual assault including two allegations of rape.
The City of London Police is investigating both allegations of rape.
Speaking to the BBC on Monday, Mr McBride said that the CBI would now be adopting a "zero tolerance culture".
But Andy Wood, chief executive of brewing company Adnams, which has cancelled its membership of the CBI, said he had not heard anything so far that "reassures me that I should become a member of the CBI again".
He said he was not sure if the group was "salvageable".
"Zero tolerance of bullying and sexual harassment - that has to be a given in a modern organisation," Mr Wood said.
"It just shows really how archaic the CBI was behind the scenes," he said.
But Michael Plaut, the former chairman of CBI Wales, said: "I just think nobody gains if we simply cancel the CBI. Letting the CBI die is really easy.
"I think we just need, as members, to get our hands dirty and rebuild this organisation, build it so it's kind of a model organisation for the 21st Century."
The CBI - the Confederation of British Industry - is an organisation which speaks to the government on behalf of around 190,000 businesses. These firms employ millions of people.
It is one of the UK's most prominent lobby groups and, according to its former president Paul Drechsler, was instrumental in protecting millions of jobs during the Covid pandemic by helping the rapid roll-out of the furlough scheme.
The CBI campaigned against Brexit. Once the UK voted to leave the European Union, it lobbied the government to secure a trade and co-operation agreement.
Another function of the CBI is to promote and share best practice among its members. Founded in 1965, today it employs around 300 people. Its director generals have been predominately male with the exception of Dame Carolyn Fairbairn, who led the group between 2015 and 2020. It has appointed its former chief economist, Rain Newton-Smith, as the new director general to take over from Tony Danker who was fired in April over separate complaints of workplace misconduct.
Rolls-Royce, the engineering giant, Diageo, owner of alcoholic drink brands such as Guinness and accountancy group PwC confirmed that they are continuing to pause engagement with the organisation.
As well as having offices across the UK, the CBI has overseas sites in Washington DC, Brussels, India and China.
Baroness Lane-Fox, president of the British Chambers of Commerce, said: "Who couldn't have massive empathy for the poor people that are working in the CBI right now? It has been an absolute rollercoaster."
The government had already decided to pause any activity with the lobby group, but on Monday, Chancellor Jeremy Hunt said there was "no point" engaging with the CBI when its own members had deserted them.