Ed Davey, the new Lib Dem leader, has warned his party that it was “diverted and distracted” by Brexit and must now refocus on voters’ more urgent concerns, as he faces mounting internal demands to formally back rejoining the European Union.

Davey said that three poor election results in the last five years demonstrated the party needed to listen to voters again, warning that the Lib Dems had “forgotten that we serve the community, we serve the people”.

In an interview with the Observer during the Lib Dem virtual conference, he said the debate around EU membership was over “for the next few years”, adding that it was not the issue being raised by voters he had met during a “listening tour” he began after winning the leadership.

Davey lamented the course of the party since its huge loss of support in the wake of joining the coalition government with the Conservatives from 2010-15. He called for a return of the kind of party led by Paddy Ashdown or Charles Kennedy, which would listen to voters’ concerns and “then give a liberal response”.

“I think somehow, whether it was the coalition years or recovering from the coalition or because we got – not sidetracked – but we got distracted by the Brexit battle, we’ve forgotten that we serve the community, we serve people. And we have to start with them and their problems, and not start with our particular interests.

“We fought the good fight. I was marching on the streets. I was there in the lobbies. I gave it everything. But it’s tragic, we lost that fight.”

Asked what he would tell ardently pro-EU members concerned about Brexit, he said: “I’d say that I’m a pro-European. And I’d say that we’re a pro-European party. That isn’t changing. And by the way, the other thing we can’t change is parliamentary arithmetic.

“And so what is the challenge now? The challenge is to show that the progressives in British politics are on people’s side.

“We are in a bit of a bind. We’ve had three very poor election results, very disappointing, in five years. I’ve got to rebuild the Liberal Democrats. And I’m not going to be diverted and distracted like we were on Brexit. I’m going to focus on rebuilding the Liberal Democrats.”

However, his stance is already being challenged by members. About 700 are said to have signed up to a conference motion to back rejoining the EU “preferably within 10 years”. A similar motion calling on the party to back EU membership in principle will be voted on at the virtual gathering.

Davey, who has set out to make the Lib Dems the voice of Britain’s carers, accused the government of damaging the rights of disabled people with its coronavirus laws. He revealed his MPs would vote against the renewal of the coronavirus legislation this week unless the rights of the disabled were protected. The Coronavirus Act relieves councils of some of their legal obligations to provide care should they feel the need to do so in order to cope with the pandemic.

“There’s some evidence showing that the pandemic has been hitting disabled people disproportionately,” said Davey, who has a disabled son and cared for his mother as a teenager.

“This is a time when they need more care, not less. I can’t stand by, the Liberal Democrats will not stand by, to see disabled people lose their rights, lose the care they need, when they need it the most.”

Asked how he now wanted voters to regard the Lib Dems, Davey repeated his leadership election priorities of “a greener economy and a fairer society in a more caring country”. However, the party had to “take some time” in trying to work out what voters are saying. “We’ve got to be humble enough to realise that we weren’t getting our message over.”

Davey said the departure of Jeremy Corbyn as Labour leader would help the Lib Dem revival in seats where it was competing against the Tories. He said the fear of a Corbyn government had hindered the party at the last election. “Corbyn was not helpful to us in our key marginals with the Tories. And the only way we’re going to get Johnson out of No 10 is if Liberal Democrats win more Tory seats, and I’m determined we win many more seats off the Tories.”