The UK government had promised to ban conversion therapy but last week decided to exclude conversion therapy for transgender people in the ban.
Groups are now withdrawing from the government's Safe To Be Me conference.
The government says it is going to look into how it can ban trans conversion therapy.
The BBC understands the conference may even be scrapped all together, with a decision on the future of the conference to be made this week.
According to NHS England, conversion therapy tries to change someone's sexual orientation or gender identity.
The government's three-day Safe To Be Me conference was due to be held this summer to coincide with the 50th anniversary of the first ever London Pride marches.
It was set to be the biggest event of its kind, and was seen by many in government as a way for the Government Equalities Office - part of the government which oversees LGBT equality - to "double down" on its commitment to LGBT rights.
But the LGBT+ Consortium, an umbrella body for LGBT+ community organisations, has issued a statement branding the government's U-turn on conversion therapy "abhorrent".
Eighty-two member organisations of the consortium have signed an open letter, which is written by LGBT+ charity and campaign group Stonewall, pulling out of the conference.
It says they will only participate in the conference if Prime Minister Boris Johnson reverts to his promise to include transgender people in any legislation against conversion therapy.
"This is a decision we take with a heavy heart. As the UK's first global LGBT+ conference, Safe To Be Me should be a moment for redoubling efforts globally to improve LGBTQ+ people's rights and experiences," it said.
"This is why we have worked hard with government and civil society organisations over the last few months to try to make the conference work."
The Terrence Higgins Trust and 22 other HIV charities and organisations have also since joined the boycott, meaning a total of 105 organisations have now pulled out.
A Terrence Higgins Trust spokesperson said: "Trans rights are human rights - progress without or at the expense of trans people is not progress. We stand together and will not be divided.
"As HIV charities and community groups we stand in solidarity with Stonewall and the whole LGBT+ sector and will not be supporting or attending the upcoming global LGBT+ 'Safe To Be Me' conference unless the government reverts to their promise of a trans-inclusive ban."
Boris Johnson has previously called the practice of conversion therapy "repulsive and abhorrent" and had promised plans to outlaw it on a number of separate occasions.
However the plans to do so have since changed; meaning the legislation will mean conversion therapy to attempt to change people's sexuality will be outlawed, but those practices carried out to try to change people's gender identity will not.
The BBC understands a series of crunch talks between No 10, the Government Equalities Office and charities hoping to get transgender people included in the ban have failed.
A government spokesperson said they will carry out "separate work" on the issue of transgender conversion therapy but that they were keen for any legislative method brought forward not to have "unintended consequences".
They added: "This is a legally complex area and we have a responsibility to ensure unintended consequences are not written into legislation, particularly in the case of under 18s."
A number of gender critical groups had fought for the ban not to include conversion therapy relating to issues of gender identity.
Responding to the legislation on Friday, Nikki da Costa, a former director of legislative affairs at No 10, said elements of the law would have had "profound consequences for children struggling with gender dysphoria".
She told BBC Radio 4's Today programme: "Doctors, therapists and parents would be deterred from exploring with a child any feelings of what else may be going on for fear of being told they're trying to change a child's identity", adding that it was "deeply concerning".
Government insiders say it was seen as "too complicated" to avoid any unintended consequences of the legislation - which may affect teachers, parents and therapists helping children who are struggling with their gender identity.
The Welsh government is now looking at legal advice to see what action it can take to ban trans conversion therapy.