Sajid Javid told the BBC: "We shouldn't be doing things for the sake of it."
It was thought the plan, which came under criticism from venues and some MPs, would be introduced at the end of this month.
Just a week ago, the vaccines minister had defended the scheme as the "best way" to keep the night industry open.
No 10 stressed the plan - which had been set to be introduced at the end of this month - would be kept "in reserve" should it be needed over autumn or winter.
Under the scheme, people would have been required to show proof - whether of double vaccination, a negative Covid test or finishing self-isolating after a positive PCR test - in order to gain entry to clubs and other crowded events.
The Night Time Industries Association had said the plans could have crippled the industry and led to nightclubs facing discrimination cases.
The industry body welcomed Sunday's announcement, saying it hoped businesses could now plan with some certainty and start to rebuild the sector.
The Music Venue Trust, which aims to protect grassroots venues, also said it was glad vaccine passports would not be going ahead, describing them as "problematic".
There had been opposition from Tory MPs on the Covid Recovery Group as well as the Liberal Democrats, whose leader Ed Davey called vaccine passports "divisive, unworkable and expensive".
Speaking on The Andrew Marr Show, Mr Javid said: "We just shouldn't be doing things for the sake of it or because others are doing, and we should look at every possible intervention properly."
He said he had "never liked the idea of saying to people you must show your papers" to "do what is just an everyday activity".
"We've looked at it properly and, whilst we should keep it in reserve as a potential option, I'm pleased to say that we will not be going ahead with plans for vaccine passports," he added.
Mr Javid denied the government was "running scared" on the policy after criticism from its own backbenchers. He said the passports were not needed because of other things in the "wall of defence" including high vaccine uptake, testing, surveillance and new treatments
The move to scrap vaccine passports appears to be a sharp U-turn by the government.
On the same TV programme last week, Vaccines Minister Nadhim Zahawi said the end of September was the right time to start the vaccine passport scheme for sites with large crowds because all over-18s would have been offered two jabs by then and it was the "best way" to keep the night industry open.
In the interview, Mr Javid also said:
* he wanted to "get rid" of PCR tests for travel and has asked for advice on the issue
* he was "not anticipating" any more lockdowns, although it would be "irresponsible to take everything off the table"
* if the UK's chief medical officers advised 12 to 15-year-olds should be vaccinated, "we can start within a week" and schools were already preparing for it. The UK's advisory body - the Joint Committee on Vaccination and Immunisation (JCVI) - has recommended against doing so except for children with particular health problems - but the final say is with the CMOs.
Scotland is taking a different approach to England - they will bring in a vaccine passport for over-18s for entry to nightclubs and many large events from October.
In Wales, ministers will decide next week whether to introduce the scheme. There are no current plans for a similar scheme in Northern Ireland.
On Sunday, the latest government figures showed there were 29,173 new cases of coronavirus in the UK and 56 further deaths, of people who had tested positive within the previous 28 days.
By Ione Wells, BBC political correspondent
The UK government had faced pressure from a number of its own Tory MPs, as well as from nightclubs and the events sector, to ditch plans for vaccine passports in England.
First, there was a hint they were pushing ahead. Last week, Vaccines Minister Nadhim Zahawi said they would be required in nightclubs and other indoor venues in England by the end of the month.
Then came the row-back. On Friday, Culture Secretary Oliver Dowden said they would "almost certainly" be necessary for nightclubs this autumn but said he'd prefer a more limited use of them.
By Sunday at 08:30 BST, the health secretary said on Sky News that the government hoped to avoid having them, and within the next hour told the BBC they will not be going ahead with plans.
Clearly there has been debate within government itself about their use but a decision has, for now, been made - even if the option will be kept in "reserve".
Labour's deputy leader Angela Rayner said the government's approach to Covid passports had been "shambolic from the start" and lacked any clarity from ministers about the purpose of the passports and how they would work.
Liberal Democrat home affairs spokesperson Alistair Carmichael accused the Conservatives of needlessly sowing confusion among businesses for months and called for them to scrap the "unnecessary and draconian Coronavirus Act altogether".
Some large venues such as football stadiums, live music venues and music festivals have already been asking people to prove their vaccination status to gain entry.
An Office for National Statistics survey, covering 25 August to 5 September, found about 1 in 10 adults across Great Britain reported that they had been asked to show proof of vaccination or a recent negative test to be allowed into a venue or event.
On Saturday, Manchester United introduced Covid spot-checks on match days at Old Trafford, with the club saying it expected proof of full vaccination to become mandatory in the Premier League from 1 October.
The Premier League said at the start of the 2021-22 season that fans would face random spot-checks of their Covid-19 status at grounds over the first few match days. Brighton, Chelsea and Tottenham have introduced mandatory checks for fans at their stadiums.
A series of key government announcements and decisions are expected in the coming days.
Boris Johnson is expected to outline plans for booster jabs soon. Mr Javid said that if the JCVI advised having a broad booster programme, he was "confident" it could start this month "as planned all along".
And on Tuesday, the prime minister will set out his Covid Winter Plan for England, likely to include contingency measures that would be implemented if the NHS was at risk of becoming overwhelmed.