Mark Carney told the BBC the fall in the pound and shrinking economy after the UK left the EU had added to "inflationary pressure".
No 10 says soaring prices are being driven by Covid and the Ukraine war.
On Thursday, the Bank warned the UK was facing its longest recession since records began.
In an attempt to cool rising prices it raised interest rates from 2.25% to 3% - the biggest jump since 1989.
A recession is defined as when a country's economy shrinks for two three-month periods - or quarters - in a row.
Inflation - the rate at which prices rise - is at the highest level for 40 years.
The invasion of Ukraine has driven up the price of food and energy, as supplies are disrupted by the war and the West tries to phase out Russian oil and gas.
However, Mr Carney, who was governor of the Bank of England between 2013 and 2020, said Brexit was also helping to fuel inflation and had "slowed the pace at which the economy can grow".
He told BBC Radio 4's Today programme the pound had fallen "sharply" against other currencies after the Brexit referendum in 2016 and "hasn't recovered".
"If I can actually cast your mind back to a few years ago, this is what we said was going to happen, which is that the exchange rate would go down, it would stay down, that would add to inflationary pressure," he said.
"The economy's capacity would go down for a period of time because of Brexit, that would add to inflationary pressure, and we would have a situation - which is the situation we have today - where the Bank of England has to raise interest rates despite the fact that the economy is going into recession."
He added that the UK had experienced "a big hit to our productivity" and "we have to take some tough decisions in order to get it back up".
But Downing Street blamed the impact of the Covid pandemic and the war in Ukraine for the problems in the UK economy.
The prime minister's official spokesman said: "What we are seeing is challenges caused by the pandemic and by war in Europe which have been driving factors in terms of inflation, and we're seeing high inflation in a number of countries around the world."
Asked if he was denying Brexit had caused financial issues, he said: "Our focus is on ensuring we have stability and fiscal credibility. That's what the chancellor and the prime minister are focused on rather than on a decision taken a number of years ago where people made a clear decision."
A fall in the value of the pound makes goods and services which are imported from overseas more expensive, while making exports more competitive.
In September the pound fell to a record low against the dollar, after then-Chancellor Kwasi Kwarteng announced sweeping tax cuts without saying how they would be paid for.
It has since recovered to the level it was at before the so-called mini-budget, after new Chancellor Jeremy Hunt reversed almost all the planned tax cuts.