Their report found bullying allegations in all services - and inspectors say this could be "the tip of the iceberg".
Cases include male firefighters telling a colleague they would rape her and a senior officer using a racial slur.
The government called the findings "deeply concerning".
His Majesty's Inspectorate of Constabulary and Fire & Rescue Services (HMICFRS) independently assesses and reports on the effectiveness and efficiency of police forces and fire and rescue services.
The watchdog's Roy Wilsher said he was "shocked and appalled" by the findings and urged the sector to "get a grip" on how it handles misconduct matters.
Last year, the inspectorate placed fire services in London and Gloucestershire in special measures, amid complaints of racism, misogyny and bullying.
The inspectorate looked at the values and culture of all 44 fire and rescue services in England, and drew on evidence collected during inspections since 2018.
Its report said there were allegations of racist, homophobic and misogynistic behaviour in 11 of these - but said it would not name them because allegations were made in confidence and some incidents were ongoing.
Those who spoke to inspectors likened the fire service to a "boys' club" and said they felt unable to report bad behaviour for fear of reprisal.
One feared their "card would be marked" if they complained about offensive behaviour; another described it as "career suicide".
In one reported incident, male firefighters used the women's on-site toilets, but female colleagues did not feel confident challenging the behaviour.
Inspectors were also told of staff being humiliated for making mistakes during training sessions and being shouted down by senior staff.
One senior officer allegedly threatened to "make life hell" for a firefighter who complained about a racist comment.
The report also highlighted there was no obligation for staff to undergo background checks and called for changes to the recruitment procedure. It makes 35 recommendations, including:
* appropriate background checks on all firefighters and staff
* the introduction of new misconduct standards including a national list of barred personnel
* better systems to help staff raise concerns
Roy Wilsher, from the inspectorate, said: "Firefighters can be called upon to do an incredibly difficult job. They should be able to trust each other implicitly, just as the public need to be able to trust them.
"Unfortunately our findings show this is not always the case. Instead, we found trust and respect is too often replaced with derogatory, bullying behaviour, often excused as banter."
The report found the fire service to be the least ethnically diverse workforce in the public sector, adding that the sector has made the least progress in achieving diversity.
"The majority of fire and rescue staff act with integrity and we are in no doubt of their dedication to the public," said Mr Wilsher.
But he said he had assumed such actions belonged to "the dim and distant past" and told firefighters it was "time for this behaviour to stop".
Matt Wrack, general secretary of the Fire Brigades Union (FBU), welcomed the report's acknowledgement of "the scale of the problem" and promised the union would "take a leading role" in transforming its culture.
"It is clear, both from our experience and from the contents of this report, that the failure to address discrimination and harassment in the service goes right to the top," said Mr Wrack.
The government department responsible for fire and rescue services described the findings as "deeply concerning" and promised to address the issues, alongside the Inspectorate and fire chiefs.
"We want to see fire and rescue services where everyone is welcome, treated with respect and able to thrive," said a Home Office official.
Last week, a report by Baroness Louise Casey called the Metropolitan Police, the UK's biggest police force, institutionally racist, misogynistic and homophobic - and said a "boys' club" culture was rife .
Its commissioner, Sir Mark Rowley, admitted they had let Londoners down and said the findings were brutal.