Labour is facing calls to stop Keith Vaz, the former Labour minister reprimanded last week for “sustained and unpleasant bullying”, from being involved in the party.

The demands follow disclosures that he was expected to speak at a conference fringe meeting alongside shadow ministers and has taken up a new party role in Leicester.

The ex-MP for Leicester East was scheduled to speak at a Hindus for Labour event on Wednesday. Other guests, according to the party’s fringe listings, include Preet Gill, the shadow international development secretary, and Steve Reed, the shadow communities secretary.

Several Leicester party members have told the Guardian that Vaz should no longer be involved in the party.

Helen Lentell, a local party member at the conference, said: “Keith Vaz should not be involved with the party, full stop. He has been shown to be a bully. The party should exclude him from the party. He should not be able or allowed to represent the party, hold office in the party, or speak at party events. The party should keep him out of party politics, because he has damaged the party’s reputation and standing.”

An online listing for the Hindus for Labour event with Keith Vaz listed as one of the speakers.

Andrew Bridgen, the Conservative MP for North West Leicestershire, said: “Why is Keith Vaz still a member of the Labour party after his litany of much publicised transgressions?”

Labour sources confirmed that Vaz had also taken up a role on the party’s local government committee in his home city of Leicester, provoking anger among some local activists.

Last week, an independent expert panel said Vaz was guilty of “sustained and unpleasant bullying, with a real and enduring psychological impact” that had ended the career of a parliamentary staff member, Jenny McCullough.

“[Vaz’s] conduct to the complainant was hostile, sustained, harmful and unworthy of a member of parliament. He should be ashamed of his behaviour,” said the panel chair, Sir Stephen Irwin.

Vaz, 64, a former Europe minister, who was in the Commons for more than 30 years, declined to cooperate with the investigation, saying he was too ill.

Vaz stood down from parliament at the 2019 election, having been handed a six-month suspension from the Commons after he was caught offering to buy class A drugs for male sex workers.

Sources confirmed that Vaz was elected in August to Labour’s local government committee in Leicester as a union representative. The 36-strong body scrutinises the work of Leicester council.

One witness said: “It was a big shock to see him back in the Labour fold, given his previous behaviour.”

Eyebrows were raised in the city over the weekend when a leaflet distributed among Leicester East constituents by a campaign against boundary changes included a large photo and quote from Vaz. The leaflet also included a letter written by Vaz calling for a halt to road changes.

The MP who succeeded him in Leicester East, Claudia Webbe, is due to face trial for harassment on Monday, a charge she vehemently denies. Webbe was a Labour MP but is now independent after having the whip withdrawn. Speculation is rife that Vaz would consider standing in the seat again if she were to face a recall petition.

Asked to comment by the Guardian, Vaz said he would not consider standing as an MP in future elections but defended his right to still be involved in local campaigns.

“I and family members have lived in Leicester for 34 years. As a private citizen I am happy to support local and national causes I feel passionate about. My various interests are well known. I write letters. I sign petitions. I have retired, I am not dead.

“I have recently been hospitalised and diagnosed with Bell’s palsy and I am undergoing treatment for this and other conditions. I try not to be bothered by what is said by others about me it does not help my recovery.”

After the Guardian’s story was published, Vaz wrote to say he did not plan to speak at a fringe event. “I am not speaking at any fringe meeting anywhere and have not done so for several years,” he said.

Labour has been approached for a comment.