Karim Khan, 50, currently heads a United Nations investigation into war crimes committed by the Islamic State group in Iraq.
BBC diplomatic correspondent James Landale said success in a closely-fought election would be seen as a diplomatic win for the UK after Brexit.
The ICC is the only permanent body investigating crimes against humanity.
Mr Khan won the votes of 72 out of 123 countries in the second round of voting, and will begin his nine-year term at the court in the Hague in June.
He defeated candidates from Ireland, Italy and Spain.
Mr Khan will be only the third chief prosecutor in the court's 18-year history, succeeding Gambian judge Fatou Bensouda.
Foreign Secretary Dominic Raab welcomed his election, saying: "Karim's extensive experience in international law will be pivotal in ensuring we hold those responsible for the most heinous crimes to account and gain justice for their victims."
Since 2018, Mr Khan has led a UN investigation aimed at identifying and bringing to justice the perpetrators of IS atrocities in Iraq.
In his 27-year law career, he has previously worked for the prosecution in the International Criminal Tribunals for crimes committed in the wars in the former Yugoslavia and during the Rwandan genocide.
Among his first tasks as ICC chief prosecutor will be to decide how to handle controversial investigations into war crimes in Afghanistan, and potentially in the occupied Palestinian territories.
Last year the US - which along with China and Russia does not recognise the court - imposed sanctions on the current chief prosecutor Ms Bensouda when she tried to investigate US actions in Afghanistan.
And Israel - which is also not a member of the ICC - accused the court of infringing "the right of democratic countries to defend themselves" when it ruled it had jurisdiction over war crimes and atrocities committed in the Palestinian territories.
Don't be afraid to give up the good to go for the great.