Laura Castle was trying to adopt Leiland Corkill when he died in January 2021 after suffering brain injuries at her home in Barrow, Cumbria.
Mrs Castle, 38, admits manslaughter but denies murder. Her husband Scott Castle denies allowing the child's death.
Preston Crown Court was told one neighbour heard a loud bang on the day Leiland was fatally injured.
The Castles, who both deny an additional charge of child cruelty, moved into their semi-detached home in August 2018, jurors were told.
Leiland, who had been taken into local authority care at birth, joined them in August 2020 ahead of the proposed adoption.
Their immediate neighbours, the Lloyd family, said they heard Mrs Castle frequently shouting, slapping noises and a child screaming through the walls of their home.
Another neighbour, Peyton Murphy, said she heard Mrs Castle shouting "countless" times, "mainly just a lot of swearing", adding: "You would hear the most horrible things."
Jane Lloyd said she would hear "frequent" shouting and swearing over the sound of her own television, as well as "the sound of like a slap followed by crying".
The regularity of the noises increased in the run-up to Leiland's death, the jury was told.
"The shouting was nearly everyday and what sounded like smacking was two to three times a week," she said.
Prosecutor Michael Brady QC asked if Mrs Lloyd ever reported her concerns to the authorities, to which she replied: "No, that's one thing that's not sitting right at all."
When asked why not, she said the Castles would have known it was her and she wanted to avoid an "awkward situation with the neighbours".
Mrs Lloyd's daughter, Ellie Lloyd, said she would hear Mrs Castle "shouting" three or four times a week for between five and 10 minutes a time.
She said on 4 January, two days before Leiland was injured, she heard the defendant shouting and screaming and "toys being thrown".
On 6 January she said she heard a "loud bang as if something was being dropped on the floor".
Mrs Castle called an ambulance at about 08:15 GMT and Leiland was taken to hospital, first to Furness General Hospital then Alder Hey Children's Hospital in Liverpool where he died.
She said the boy had fallen from the sofa but medical staff were sceptical and raised the alarm, prosecutors said.
The trial continues.