Fifteen others were taken to hospital but the French authorities believe at least one more could be lost at sea.
Care4Calais urged the government to create legal routes for refugees to reach the UK.
UK PM Boris Johnson pledged to "crack down" on gangs facilitating crossings.
An investigation into the cause of the sinking has been opened by the public prosecutor in Dunkirk.
Searches which paused overnight are expected to begin again on Wednesday morning, but the BBC's Simon Jones said - now 24 hours on from the sinking - it was likely to be "a recovery operation rather than a rescue operation" if there are still people out in the Channel.
Charity Care4Calais said the "loss of life should be a wake-up call for those in power in France and the UK".
It said creating a new system which would allow asylum seekers to apply for refuge in the UK from outside its borders would "put an end to terrifying, dangerous sea crossings and stop tragedy striking again".
Save The Children called for a "joint plan" from London and Paris to ensure the safety of vulnerable families, adding: "The English Channel must not become a graveyard for children."
Mr Johnson said his thoughts were with the victims' loved ones.
"We have offered the French authorities every support as they investigate this terrible incident, and will do all we can to crack down on the ruthless criminal gangs who prey on vulnerable people by facilitating these dangerous journeys," he said.
Environment Secretary George Eustice said the deaths were a "terrible tragedy" and that there was not a reason for people to take the "highly dangerous crossing" from France, adding that it was down to "criminal gangs callously taking their money".
He said the government's approach was to "remove the incentive for people trying to take this crossing".
Dan O'Mahoney, the clandestine Channel threat commander, will go over to France to discuss what can be done with his counterparts there.
Conditions in the English Channel had been rough throughout the day, with a gale warning issued overnight by the Met Office.
The boat was spotted about 2km off the French coast by a passing sailboat at about 09:30 local time on Tuesday, which alerted French authorities.
Four French vessels, one Belgian helicopter and a French fishing boat took part in the rescue and a search operation. But by the time rescue teams reached the struggling vessel, it had already sunk.
One man drowned, while a woman, along with two children - aged five and eight - suffered cardiac arrests and never recovered.
Fifteen other people, who are believed to be from Iran, were taken to hospitals in Dunkirk and Calais.
Labour MP Yvette Cooper, chair of the home affairs select committee, said the deaths were "truly awful", and that criminals were profiting from "other people's desperation".
Natalie Elphicke, the Conservative MP for Dover, said it was "terrible that tragedy has struck in the Channel again", adding: "People traffickers have no regard for life, no matter how old or young."
The UK government has vowed to make the crossings "unviable" and called for boats to be stopped at sea and returned to France.
The UN Refugee Agency said in August it was "troubled" by the plans to intercept and return boats, adding that the numbers making the crossing "remain low and manageable".
More than 7,400 migrants have reached the UK in small boats this year, up from 1,825 in 2019.
At least two other people are thought to have died while attempting the crossing this year, with one body found on a beach near Calais on 18 October.
Abdulfatah Hamdallah, from Sudan, died while trying to row to the UK in August.
“This last is important. Even in corporate environments, it is very difficult to remove an underling for incompetence if that underling has seniority and a long history of good performance reviews. As in government bureaucracies, the easiest way to deal with such people is often to “kick them upstairs”: promote them to a higher post, where they become somebody else’s problem.”