Mr Welby criticised delays in taking down a plaque to slavery financier Tobias Rustat at a Cambridge college chapel.
"Why is it so much agony to remove a memorial to slavery?" he said.
Earlier at the General Synod, the head of a Church racial justice body said the lack of change was "a scandal".
A Church court is considering whether a plaque to 17th Century courtier Mr Rustat, who invested in the Royal African Company, can be removed from the wall of a chapel at Jesus College, Cambridge.
The college, which proposed the removal, says Mr Rustat's financial involvement in the company helped it to enslave 150,000 African men, women and children over 50 years - more than any other institution.
Mr Welby pointed out that Jesus College now has a black woman as its master, Sonita Alleyne, who has to look at a memorial to a man who financed slavery "every time she sits in her stall".
The archbishop questioned why it was "so difficult" to remove the marble plaque, when the college only wanted to move it to another location where it can be interpreted and explained.
"We need to change our practices," he said.
Mr Welby was responding after Labour peer Lord Boateng - chair of the Archbishops' Racial Justice Commission, which aims to identify and root out systemic racism in the church - drew applause as he said the Church's lack of action on race issues was "chilling".
He said statues and monuments to people involved in the slave trade caused him hurt, while the failure to improve diversity meant "the government front bench looks better" than the Church's leadership.
Only one out of 42 senior bishops is from an ethnic minority, compared to six out of 23 members of the cabinet.
"Imagine what it is to go into a place of worship to look up and to see a monument to someone who was a party to the enslavement of your ancestors," said Lord Boateng.
"We are members in churches which have themselves benefitted from the horrors of the slave trade. That is the reality."
He referred to the church's missionary organisation, the Society of the Propagation of the Gospel, which owned a plantation in the 18th Century - and which branded the word "Society" on the skins of the people they enslaved.