Grant Shapps said there were "lots of questions" to answer about "how safe it will be in June to travel".
People in England could be allowed to go on holiday abroad from 17 May at the earliest, according to current plans.
A Greek minister said his country hoped to welcome visitors from mid-May.
Cyprus and Portugal are also among the countries keen to welcome UK visitors.
Under the current rules, foreign travel from the UK is banned apart from for exceptional reasons.
While 17 May has been marked as the date that rule could change - the decision is subject to a review closer to the time.
According to the latest figures, another 190 people have died in the UK within 28 days of a positive coronavirus test, and a further 5,926 positive cases have been reported.
Asked on BBC Radio 4's Today programme whether it was too early to book a summer holiday, Mr Shapps said: "Yes."
He added: "We have said it will remain illegal to travel internationally until at least 17 May - that's an at-the-earliest date."
Mr Shapps said people should look towards 12 April, when he would publish a global travel taskforce report, and said the government was weighing up not only the "extraordinary" UK vaccination rollout but also other countries' vaccine programmes.
"So there are lots of questions, the answer to which we simply don't know yet, about how safe it will be in June to travel around," he said.
Wales, Scotland and Northern Ireland have not set a date for the start of international travel.
Once travel is allowed again, the success of the UK vaccination programme is likely to make British holidaymakers attractive to countries looking to rebuild their tourism sectors, which have been badly hit by the pandemic.
Greece tourism minister Haris Theoharis said his country hoped to welcome foreign tourists from 14 May, during an appearance at a travel trade show.
He said: "Regarding 2021, in Greece we are more than optimistic."
Visitors will be required to have been vaccinated, have had a recent negative Covid-19 test or have coronavirus antibodies, he said.
A pilot reopening of borders is likely to take place next month, he added.
Last week, Cyprus and Portugal said they hoped to welcome vaccinated British tourists from May, and Spain has already said it will not restrict the right of entry to travellers who have been inoculated.
Many countries are working on their plans to allow tourists and it is not yet clear if there will be some form of international vaccine passport or certificate.
Cabinet Office minister Michael Gove is leading a UK government review of the issue.
Greece and Cyprus have agreed to admit Israeli tourists who can prove their Covid status via Israel's "green" digital certificate.
Israel is leading the world in terms of vaccinations, with 101.1 doses given per 100 people.
Domestic cruises in the UK could resume from 17 May, according to the maritime minister Robert Courts, while UK holiday firms reported a surge in bookings following the announcement of the government roadmap back in February.
Most accommodation and holiday providers have offered flexible booking policies for future trips, to boost confidence, and the Competition and Markets Authority (CMA) has suggested people should usually get their money back if coronavirus restrictions ruin holiday plans.
If a business cancels a booking or cannot provide a service due to lockdown laws then consumers should receive a full refund under most circumstances, the CMA said.
The watchdog added that customers should not face "disproportionately high" charges if they cancel a trip due to government guidance.
If you were surprised Epstein committed suicide, imagine how surprised he was!