Questioned about the government's record, in his final answer at PMQs, Mr Johnson said: "They jabber, we jab."
Labour's Jess Phillips said the remark showed he did not take tackling sexual violence against women seriously.
No 10 insisted the PM was making a broader attack on Labour's tendency to talk about issues but not take action.
At Prime Minister's Questions, Mr Johnson denied claims by Sir Keir Starmer that budget cuts were to blame for the fact that rape prosecutions in England and Wales have more than halved since 2016.
He said it was down to "evidential difficulties" and a lack of "joined-up" working in the criminal justice system, rather than a lack of cash.
This would appear to contradict his own justice secretary Robert Buckland, who told the BBC last week cuts were "self-evidently" a factor, as he announced plans to fix the system.
Sir Keir - who is the former head of the Crown Prosecution Service - said: "On the prime minister's watch, rape prosecution and convictions are at a record low, court backlogs are at a record high, victims are waiting longer for justice and criminals are getting away with it.
"This wasn't inevitable, it's the cost of a decade of Conservative cuts and even now the government isn't showing the urgency and ambition that's needed."
He said that 98.6% of reported rapes did not end in charges or a sentence, which he called an "appalling" record.
Mr Johnson apologised to rape survivors and victims "for the trauma they have been through" and the "frustrations" they feel at the "inadequacies" of the criminal justice system.
He said the government was putting money into clearing court backlogs and a plan to improve rape conviction rates.
And he accused Labour of not supporting tougher sentences for serious sexual and violent offenders.
He ended the exchanges with an attack on Labour and praise for the government's coronavirus vaccination programme, saying: "We are getting on with the job, they jabber, we jab."
This prompted an angry response from Labour's shadow minister for domestic violence and safeguarding, Jess Phillips.
"For the prime minister to describe questions about rape convictions as 'jabber' is disgraceful," said Ms Phillips.
She called on the PM to apologise for his words, claiming he "simply doesn't care about tackling sexual violence".
She later appeared in the Commons to make a point of order calling for the PM to apologise and "correct the record" over his claims about Labour's voting record.
Labour's shadow justice secretary David Lammy said the PM's "idiotic jibe" was a "total disgrace", and showed how little he cared about the issue.
Earlier, Victims' Commissioner for England and Wales, Dame Vera Baird, told the Home Affairs Committee the government's long-awaited rape review was "underwhelming" but also a "watershed moment".
She said there was a "very large amount of kicking the can down the road" but the apology made by ministers for falling prosecution rates was "powerful" and campaigners were seeing that as a "signal of intent".
She said as long as the "strength" of that apology survived then she was hopeful "we can really bring a change...and drive the government further".
But the treatment of victims was "appalling", she added, with police forces focusing on whether the victim was credible and not the defendant.
Last week, Home Secretary Priti Patel said ministers were "deeply ashamed" of falling conviction rates and joined Mr Buckland in apologising for the situation and promising to "do much better".
Asked in a BBC interview whether government cuts to the legal system were a factor, Mr Buckland said: "Like all parts of public service big choices were made in the last decade, because of the position that we all faced economically and that's, I think, self-evidently the case."
He said the government was now "seeking to make the necessary investment" but also insisted "it's about more than just money, it's about culture."
He argued that "for too long" rape allegations have focused on the credibility of the victim rather than the evidence.