Ministers are facing calls to investigate the chair of England’s university regulator about his participation in a conference for the Republican party, which platformed a notorious racist and antisemite.

Labour and the Liberal Democrats called on Conservative party leaders and ministers to take action after James Wharton, chair of the Office for Students (OfS), addressed the Conservative Political Action Conference (Cpac) Hungary in Budapest on Friday.

On the same day that Lord Wharton, who ran Boris Johnson’s leadership campaign, gave his video message, Zsolt Bayer, a television talkshow host in Hungary who has been widely denounced for his racism, took to the stage. Bayer has called Jews “stinking excrement”, referred to Roma as “animals”, and used racial epithets to describe Black people.

Senior academics of colour said Wharton’s participation in the event raised questions as to whether the OfS would effectively address racism in UK universities.

In his speech, which has been posted on Twitter, Wharton endorsed the recent victory of the Hungarian prime minister, Viktor Orbán, adding that he was present in Budapest during the recent elections to see that “we can win”. He also said Cpac was a “great chance to pick up new ideas”, to “reconnect with friends across the world” and to “fight for the values that we all hold dear”.

Other speakers at the conference included Jack Posobiec, a far-right US blogger who has used antisemitic symbols, the former US president Donald Trump, and the former Ukip leader Nigel Farage. Orbán was the star speaker on the first day.

Anneliese Dodds, the shadow women and equalities secretary and chair of the Labour party, called on the Conservatives to clarify whether Wharton was given permission to attend Cpac, and whether the party shared the views he expressed.

In a letter to the Tory co-chair Oliver Dowden, she wrote: “Will you urgently explain whether the Conservatives think it is acceptable for the chair of the Office for Students to publicly endorse Mr Orbán at a time when Lord Wharton is responsible for defending academic freedom in this country? And will you condemn Lord Wharton for sharing a platform with racists, antisemites and rightwing extremists?”

Munira Wilson, the Liberal Democrat spokesperson for education, said: “It’s incredibly disturbing that the head of the independent regulator for higher education moves in such circles. There’s no reasonable excuse for keeping such company.

“This is a really bad look for the OfS and completely undermines their stated zero-tolerance policy on racism and antisemitism. The education secretary must investigate Wharton’s attendance at this event without delay. Students from ethnic minority groups and Jewish communities deserve answers.”

The appointment last year of Wharton, a former Conservative MP with no experience in higher education, to chair the OfS attracted widespread criticism from politicians and students, who said he lacked independence.

Prof Kalwant Bhopal, director of the Centre for Research in Race and Education at the University of Birmingham, said: “It is hugely disappointing that the chair for the OfS has chosen to speak at this event. This suggests we cannot be confident that the OfS will address issues of racism and take them seriously. This is a step backwards and reinforces the rhetoric from rightwing politicians that racism is no longer an issue.”

Kehinde Andrews, a professor of Black studies at Birmingham City University, said: “It is a worrying time for students from diverse, underprivileged and minority backgrounds. It is pretty clear the chair of the OfS will not be fighting their corner.”

In 2018, Orbán shut down the independent Central Europe University as part of his campaign against George Soros. More recently he has embraced the “great replacement” conspiracy theory, which involves promoting the belief that the white population is being deliberately reduced by leftist policies and diluted by immigration.

The OfS said Wharton was not speaking in his capacity as chair as the OfS.

When questioned about Wharton’s endorsement of Orbán in parliament on Monday, the education secretary, Nadhim Zahawi, said: “I support the chair of the Office for Students for all his work that he’s doing in improving outcomes for students in our universities.”