Clarke said he was "deeply saddened" for the offence he had caused by using the term "coloured footballers".
The comments came as he was talking about the racist abuse of players by trolls on social media to the Department for Digital, Culture, Media and Sport (DCMS) select committee via video link.
Clarke said his words were a "disservice to our game".
He prompted further criticism when referring to gay players making a "life choice" and a coach telling him young female players did not like having the ball hit hard at them.
He also said there were "a lot more South Asians than there are Afro-Caribbeans" in the FA's IT department because "they have different career interests".
"We can confirm that Greg Clarke has stepped down from his role as our chairman," said an FA statement.
"Peter McCormick will step into the role as interim FA chairman with immediate effect and the FA Board will begin the process of identifying and appointing a new chair in due course."
Following his resignation, Clarke said: "My unacceptable words in front of Parliament were a disservice to our game and to those who watch, play, referee and administer it. This has crystallised my resolve to move on.
"I am deeply saddened that I have offended those diverse communities in football that I and others worked so hard to include."
A statement from Show Racism the Red Card
said Clarke's comments "only serve to demonstrate the power of language and the damage of stereotyping groups of people".
During the parliamentary hearing, Clarke apologised after being prompted to say sorry by MP Kevin Brennan.
Brennan said Clarke's language in reference to black players was the kind that did not encourage inclusion, while fellow committee member Alex Davies-Jones called it "abhorrent".
Clarke had earlier spoken of the need to attract people into the sport from a diverse range of communities.
The equality charity Kick It Out said his remark about black players should be "consigned to the dustbin of history" and criticised his comments concerning people from South Asia, gay players and female footballers.
Clarke had been called to give evidence to the DCMS committee about the Premier League's potential bailout of English Football League clubs and the structural reforms proposed as part of 'Project Big Picture'.
"As a person who loves football and has given decades of service to our game, it is right that I put the interests of football first," added Clarke in the statement confirming his departure.
"2020 has been a challenging year and I have been actively considering standing down for some time to make way for a new chair now our CEO transition is complete and excellent executive leadership under Mark Bullingham is established."
Speaking before the resignation was announced, Sanjay Bhandari, executive chair at Kick It Out, said Clarke's comments to the DCMS were outdated.
"I was particularly concerned by the use of lazy racist stereotypes about South Asians and their supposed career preferences. It reflects similar lazy stereotypes I have heard have been spouted at club academy level," he said.
"Being gay is not a 'life choice' as he claimed too. The casual sexism of saying girls do not like balls hit at them hard is staggering from anyone, let alone the leader of our national game. It is completely unacceptable."
Nigel Huddleston, sports minister: "Greg Clarke's comments have caused deep offence and were completely unacceptable. I acknowledge his decades of service to football and his apology, but he was right to stand down as chairman of the FA. We must ensure that opportunities are open to everyone in the sports sector - from athletes to board members - and all forms of discrimination must be tackled head on."
David Bernstein, former FA chairman, told BBC Sport: "I am just surprised that the chair of any organisation who's got a feel of what's going on in the year 2020 could use those types of words, that sort of language. It's just inappropriate."
Darren Bent, former England striker: "Slip of the tongue was it? Awful, just awful."
Anton Ferdinand, former West Ham, Sunderland and QPR defender: "Clearly education is needed at all levels."
Julian Knight, DCMS select committee chairman: "It's right that Greg Clarke apologised before the committee. However, this isn't the first time that the FA has come to grief over these issues. It makes us question their commitment to diversity."
Alex Davies-Jones, committee member: "The language used by Greg Clarke in our meeting this morning was absolutely abhorrent. It speaks volumes about the urgent progress that needs to made in terms of leadership on equalities issues in sport. I can't believe we're still here in 2020."
BBC sports editor Dan Roan
Three years ago - in front of the same parliamentary committee, Greg Clarke was criticised for referring to institutional racism as "fluff". He apologised after being chastised by MPs and reminded that language matters.
It appears the message did not get through.
Two weeks after the FA launched a new diversity code with the aim of finally tackling racial inequality in the game, the governing body's commitment to diversity has once again been called into question.
Amid under-representation of BAME managers and board members, many critics will see Clarke's comments as evidence of the attitudes and language that has prevented the organisation from overseeing the progress hoped for in recent years, and it is no surprise that he has decided to step down.
Despite having barely been seen since the start of the year, Clarke was already under pressure over his role initiating secret talks over the Project Big Picture plans for a radical overhaul of the English game. Indeed earlier in the committee hearing, he was asked if his authority was "shot", something he strongly denied.
But then came his comments on diversity. Amid an unprecedented financial crisis for the sport and damaging divisions with fans, leagues and government, the FA chairman has now had to go over yet another controversy.
This is another grim day for the game, at the worst possible moment.
It doesn’t matter how many times you fail. It doesn’t matter how many times you almost get it right. No one is going to know or care about your failures, and neither should you. All you have to do is learn from them and those around you because all that matters in business is that you get it right once. Then everyone can tell you how lucky you are.