On US news programme The Daily Show earlier this week, Noah had said there had been a "backlash" over Mr Sunak.
His comments were criticised in the UK, with many including ex-chancellor Sajid Javid calling Noah "simply wrong".
But Noah has now defended his segment, saying he was reacting to racists, and: "That's why I said. 'Some people'."
Mr Sunak is the UK's first British Asian prime minister and officially took over as Conservative leader and PM on Tuesday, after a leadership process.
In the original comments on the US programme, Noah - who is South African and grew up during apartheid - said: "You hear a lot of the people saying 'Oh, they're taking over, now the Indians are going to take over Great Britain and what's next?'
"And I always find myself going 'So what? What are you afraid of? I think it's because the quiet part that a lot of people don't realise what they're saying is, 'We don't want these people who were previously oppressed to get into power because then they may do to us what we did to them.'"
During his skit, Noah played a clip from radio station LBC during the latest Conservative leadership race a week ago, when a caller falsely claimed Mr Sunak was "not even British".
Mr Javid tweeted in response that the comments from the comedian were "so wrong" and that Britain "is the most successful multiracial democracy on earth and proud of this historic achievement".
Former Tory leadership contender Rory Stewart said Mr Noah's remarks were "completely bizarre" and an example of "lazy stereotyping".
Downing Street said on Thursday that Mr Sunak did not believe Britain was a racist country.
Presenter Piers Morgan also tweeted that US media was "falsely portraying Britain as a racist country".
Noah finally responded on Friday evening, saying: "C'mon Piers, you're smarter than that.
"I wasn't saying "The entire UK is racist", I was responding to the racists who don't want Rishi as PM because of his race. That's why I said. "Some people"."
Noah has long spoken about racial equality, publishing a book in 2017 titled Born a Crime, a reference to the fact he was born in South Africa to a white Swiss father and a black Xhosa mother, at a time when such a relationship was punishable by imprisonment.
He began his career in South Africa, releasing a string of stand-up specials and hosting a late-night talk show before relocating to the US in 2011.
He has hosted The Daily Show - a late-night talk and satirical news programme - since 2015 but last month announced he would be standing down.
Mr Sunak was born in Hampshire, south-east England, to Indian parents - a pharmacist mother and a GP father - and is married to Akshata Murty, with whom he has two young daughters Krishna and Anoushka.