Rape, sexual assault or being touched without consent accounted for more than one in five cases.
Most incidents - 58% - involved patients abusing staff.
The data was collected by the British Medical Journal (BMJ) and the Guardian, and shared with BBC File on 4.
Freedom of Information requests were received from 212 NHS trusts in England.
The data that came back from trusts showed at least 20% of incidents involved rape, sexual assault or inappropriate physical contact - including kissing. Other cases included sexual harassment, stalking and abusive or degrading remarks. One in five cases involved patients abusing other patients - although not all trusts provided a detailed breakdown.
One former patient told File on 4 how she had been sexually assaulted by a male patient while she had been showering in a hospital.
"The door flew open and there was a man in the doorway," said Mary (not her real name). "I screamed as I had no clothes on. He ran towards me and shouted 'I'll have some of that'."
When the man grabbed her arm, she explained, she "managed to elbow him and push him away".
"I was trying to run and cover myself with my hands and parts of my body because there's everybody staring."
Mary, now in her early 80s, had been admitted to the hospital after suffering a heart attack in 2020.
Her attacker had been sitting in a chair outside the shower room.
The incident was reported to the police but the force involved said there wasn't enough evidence to proceed with a prosecution.
An NHS spokesperson has told the BBC that all NHS organisations must have robust measures in place to ensure immediate action is taken in any sexual cases reported to them.
But the BMJ and Guardian investigation found that fewer than one in 10 NHS trusts has a dedicated policy to deal with sexual assault and harassment - and that managers are also no longer obliged to report abuse of staff to a central database.
Mary says she wasn't given any support by the hospital after the attack - and the NHS trust involved has admitted to File on 4 that more should have been done.
The Liberal Democrats' health spokeswoman Daisy Cooper MP is calling for a new sexual complaints system to protect patients, visitors and staff. She says Mary's story is an example of the "absolutely unacceptable" number of sexual attacks happening in NHS hospitals across the UK.
"I'm hearing time and time again, this has happened to so many other people.
"There is not a simple, clearly signposted way for people to make a complaint of a sexual nature and for it to be treated with respect and be treated efficiently."
Consultant Philippa Jackson told File on 4 she had been sexually assaulted by a senior NHS colleague when working as a junior doctor.
"He rubbed himself against my thigh, he then made a comment about his erection," she said.
"Then he kissed me on the bottom of my neck and again I felt very uncomfortable."
When Ms Jackson made a formal complaint about the individual, she says she was quizzed by senior managers at the hospital.
"I was asked a series of questions about whether or not I had led him on, essentially, whether or not I was being flirtatious," she says.
No action was taken against the man, and Ms Jackson says she was told the incident hadn't been witnessed and could not be proved - so there was no case to answer.
Although more than 4,000 NHS staff were accused of rape, sexual assault, harassment, stalking, or abusive remarks towards other staff or patients in 2017-22, the BMJ and Guardian investigation found that only 576 have faced disciplinary action.
In a statement, the Department of Health and Social Care said: "The health and social care secretary is working closely with the NHS and recently convened an urgent meeting with NHS leaders to discuss how to root out this vile behaviour and ensure services are always safe for staff and patients."