Mr Khan joined a large crowd that gathered near Hyde Park ahead of a march in solidarity with the war-torn country.
The Labour mayor said the actions of the UK government compared with other European countries were "embarrassing".
Mr Khan said it needed to be made "much more easier for those who are fleeing Ukraine to come here".
Earlier this month the government opened it Homes for Ukraine scheme, which allows UK nationals to individually sponsor a Ukrainian national's visa so that those without family connections can still come to live with a host family in the UK.
Protesters made their way towards Trafalgar Square whilst holding Ukrainian flags and banners that said "stop Putin's war", "no fly zone" and "solidarity with Ukraine".
The crowd chanted "we stand with Ukraine" as traffic behind was brought to a standstill.
About 10 million people have fled their homes in Ukraine because of the Russian invasion, the United Nations High Commissioner for Refugees said.
As well as the 3.6 million who have left for neighbouring countries, another estimated 6.5 million people are thought to be displaced inside the war-torn country itself.
By Chi Chi Izundu, BBC News
A group of children clutching balloons in the colours of the Ukrainian flag marched in the heat of the sun in central London leading today's protest.
Joined by thousands of others, mini flags were being handed out for people to show their solidarity.
A sea of blue and yellow surrounded Nelson's column as special guests addressed the crowd, trying to teach Ukrainian phrases.
Most of the people that we stopped to speak to were either Ukrainian born or had Ukrainian links, like Olaksana. She told us that her family had just fled Kharkiv and were in Hungary and Germany.
She said: "It is really hard to be in another country when your country is suffering.
"The least I can do is come to this protest and the least I can do is stand with Ukraine and unite as a nation and show the world they're not going to kill us as a nation."
Mr Khan praised the actions of Londoners who had "shown that we're opening our arms to Ukrainian refugees, whether it's by donating money to those charities doing a brilliant job in Ukraine or by opening our homes to those fleeing Ukraine".
"I'm full of admiration for anybody who has the space to provide a room for those who have fled Ukraine," he said.
"It's really important families are not left to their own devices, the government has to step up and support them."
The BBC has contacted the Home Office for comment.