He was found guilty in January of a serious sexual assault against a boy and the attempted rape of a young girl.
The abuse happened in Rotherham when he was a teenager, Sheffield Crown Court heard.
Passing sentence, Mr Justice Lavender said his actions had had "profound and lifelong effects" on the victims.
The court heard Lord Ahmed, who was tried under his birth name Nazir Ahmed, attempted to rape the girl twice in the early 1970s, when he was aged 16 or 17 but she was much younger.
The attack on the boy, who was aged under 11 at the time, also happened during the same period.
The 64-year-old had denied the charges, calling them a "malicious fiction", but a phone recording of a conversation between the two victims in 2016 showed they were not "made-up".
The judge said the offences were "so serious that only a custodial sentence can be justified".
He said: "Your actions have had profound and lifelong effects on the girl and the boy, who have lived with what you did to them for between 46 and 53 years.
"The statements which they have made express more eloquently than I ever could how your actions have affected and continue to affect their lives in so many different and damaging ways."
The victim of the attempted rapes, who cannot be named due to the nature of the offences, said she had lived with "an overwhelming feeling of shame".
She told the court: "It was a burden I was made to carry, and it silenced me for many years.
"It is now time for me to pass that burden to him - the paedophile who I know feels no personal shame."
The male victim, who also cannot be named, said in a personal statement read out in court: "I buried the abuse and carried it with me on my own for years and years.
"I feel shame because of what these men did to me."
In court number seven at Sheffield Crown Court today, both the judge and defendant could technically be addressed as "my Lord", "your Lordship" or any other variation of the title.
Lord Ahmed sat in the dock wearing a smart suit and tie, listening to barristers and a judge discuss his sentence.
One victim came to the witness box to describe the impact the peer of the realm had on her life.
"He is a paedophile who has no personal shame. But in the end, all tyrants fall," she said.
A statement from the second victim said: "This is not about revenge, this is about justice."
And part of that justice is wanting to see Lord Ahmed stripped of his title and known forthwith by his real name, Nazir Ahmed.
They don't want him to use his title to impress anyone again.
The judge told him his actions had a profound and lifelong effect on his victims. He stood to listen to the judge but showed no emotion.
Shortly afterwards, Ahmed picked up his rucksack, waved at his family and disappeared through a door to start his five-and-a-half year sentence.
Ahmed, who was convicted following a retrial, resigned from the House of Lords in November 2020 after a conduct committee report concluded he had sexually and emotionally exploited a vulnerable woman who sought his help.
The inquiry into his behaviour followed a BBC Newsnight investigation.
There have been calls for him to be stripped of his title. This would require an Act of Parliament but none currently exists.
Ahmed was charged along with his two older brothers, Mohammed Farouq, 71, and Mohammed Tariq, 65, but both were deemed unfit to stand trial.
Both had faced charges of indecent assault against the same boy abused by Ahmed.
Though the men did not face a criminal trial, jurors concluded that they did commit the alleged acts after hearing evidence in the case.
Both men were given absolute discharges after the judge said the only other two options - a hospital order or a supervision order - would not be appropriate in this case.
Sarah Champion, Labour MP for Rotherham, paid tribute to the "courage, bravery and determination of the victims".
She said: "The verdict in this case sends a clear message to offenders: no matter your power or your status, justice will be served, however long it is eluded.
"The fact Mr Ahmed is able to retain his peerage makes a mockery of our honours system. I urge the government to bring forward legislation to enable his title to be stripped from him."