Answering children's questions ahead of the COP26 climate change summit, the prime minister said reusing plastics "doesn't begin to address the problem".
Instead, he said, "we've all got to cut down our use of plastic".
The Recycling Association said the PM had "completely lost the plastic plot".
The association's Simon Ellin told BBC Radio 4's World at One programme Mr Johnson's comments were "very disappointing" and seemed to conflict with government policy.
But some anti-plastic campaigners praised the prime minister's stance and urged him to follow it up with measures to dramatically reduce plastic at source.
Sian Sutherland, co-founder of A Plastic Planet, said: "Less than 10% [of plastic] is actually recycled in the UK. Despite being touted by industry as a solution to the problem, all it has done is justify overproduction and created an industrial addiction to this indestructible, toxic material."
During the special event organised by Downing Street, Mr Johnson told an audience of eight to 12-year-olds that rather than relying on recycling, people should reduce their consumption of plastic products.
Tanya Steele, chief executive of the World Wide Fund for Nature, told the event: "We have to reduce, we have to reuse - I do think we need to do a little bit of recycling, PM, and have some system to do so."
But the PM said it was a "mistake" to think society can recycle its way out of the problem, and added: "It doesn't work."
Asked later about Mr Johnson's comments, his official spokesman said the PM continued to encourage recycling - though he said relying on it alone would be a "red herring".
The prime minister's words might have been careless, but they have an element of truth.
Polls consistently show many people feel they're doing their bit to protect the climate if they gather up plastic bottles and take the car up to the recycling point on a Saturday.
But in truth, recycling is a bit of a soft option. It's far less important than many other actions to curb emissions, such as stopping flying, or buying an electric car, or giving up meat. But these actions are harder to do.
And as the PM said, there's a limit to how many times plastic can be recycled.
But having said that, it's far better to recycle a plastic bottle than to throw it away.
Because recycling plastic isn't just helping a little to rein in climate change, it's also keeping waste out of the oceans, conserving energy and saving some of the earth's precious resources.
There are plans to increase recycling in England, which the government has said "typically results in lower carbon emissions in comparison to manufacturing products from virgin materials".
"Priority goes to preventing the creation of waste in the first place, followed by preparing waste for reuse; to recycling, and then recovery," the waste management plan for England said.
The most recent figures for England showed a recycling rate of 45.5% for household waste. Waste policy is largely set by the devolved administrations in the UK.
Each UK council collects its plastic recycling differently. A BBC analysis in 2018 showed there were 39 different sets of rules for what can be put in plastic recycling collections.
Meanwhile, with just days to go before COP26 begins in Glasgow next week, the PM said it was "touch and go" whether the summit would deliver progress.
Millions of lives could be affected by any changes agreed at the conference - and we have devised a quiz for you to discover which policies would have the most impact.
By Esme Stallard, BBC Reality Check
* Recycling plastics can reduce the amount of CO2 produced per tonne of plastic by between 1.1 and three tonnes, compared with producing the same tonne of plastic from "virgin" fossil fuels
* But not all plastics can be recycled, including some resins. Even for some plastics that can be recycled, the process can use a lot of energy
* Crucially, plastic cannot be recycled indefinitely - unlike glass or aluminium
* Every time plastic is recycled its quality decreases, so on average a plastic product can only be recycled two to three times before it is no longer used. Therefore, reusing products and reducing consumption is important
The prime minister criticised drinks giant Coca-Cola for being among 12 big corporations "that are producing the overwhelming bulk of the world's plastics".
The company sells more than 100 billion throwaway plastic bottles each year, causing problems in countries too small to effectively recycle them, BBC Panorama has found.
Coca-Cola said it was making "steady progress" in efforts to cut waste and all its smaller bottles in Great Britain were made from 100% recycled plastic.
During the event, Mr Johnson also signalled the government does not want to support new coal mines, as ministers face pressure to prevent a site opening in Cumbria.
And he jokingly suggested feeding human beings to animals to "bring nature back".
Ms Steele told the audience: "97% of the mass of mammals on this planet is humans and our animals, our domestic animals. Just 3% is left for the wild."
In response, Mr Johnson said it was "so sad", and quipped: "We could feed some of the human beings to the animals."
Ms Steele replied: "We could have a vote later and ask if there's any candidates."
How many tonnes of CO2 per year each action would save for an average person, according to research by the University of Leeds:
* 2.0 - Live without a car
* 2.0 - Use an electric car instead
* 1.9 - Cut out long-haul flights
* 1.6 - Switch to renewable energy
* 0.9 - Go vegan
* 0.6 - Use public transport often
* 0.1 - Recycle