After a decade of reinvention, Labour has again fallen short electorally. Thursday’s dramatic loss in Hartlepool and in councils across the country has raised the perennial question for parties on the left: “What is to be done?”

Much of the discussion of Labour’s woes concentrates on British particularities, its Brexit strategy and the relative merits of the Blairite and Corbynite reinventions. However, the dilemmas Labour faces are far from particular. The last decade has not been kind to social democratic parties across Europe. The centre-left parties that dominated European politics for the second half of the 20th century have suffered a string of losses.

In France, the Parti Socialiste fell to under 8% of the vote in the last legislative elections, with no signs of recovery. In 2017, the German SPD experienced its worst postwar performance, a showing likely to worsen in September’s election. Even where social democrats are in power, their position is tenuous. The Swedish Social Democrats, the most electorally successful socialist party in Europe, struggled to form a government after the 2018 election.

Too often, the debate about the failures of the left focuses on the past, asking why social democrats have lost traditional working-class voters. This kind of argument claims that social democrats have lost their base because they “lost their way” – moving too far to the left on new “woke” social issues, while at the same time moving too far to the right on economic issues. This perspective is at least incomplete and often misleading.

Deep changes in Europe’s class structure mean that the appeals of the social democratic heyday are increasingly electorally limited. As recently as the 1980s, many children left school without a qualification, fewer than half of women worked outside of the home and working-class jobs were concentrated in heavy industry and manufacturing. Today, the population is hugely more educated. In the UK, as elsewhere, the share of 25- to 34-year-olds with a higher education qualification has risen dramatically, from 28% to 52% in the last 20 years. A majority of women are in paid work and much of the working class is employed in retail, hospitality and care services. These structural transformations have created new groups of voters with new economic and social concerns.

In looking at what these changing voter groups want, it is not at all the case that social democratic policies are unpopular, especially among the new middle classes. Survey after survey reaffirms that large portions of European voters support a robust welfare state, protections for workers and high-quality public services. So, given the popularity of many social democratic policies, what, then, explains their near universal decline?

Contra to the dominant narrative, this decline is not solely attributable to the loss of working-class voters. In Europe’s proportional electoral systems, highly educated voters are overrepresented among those who turned their backs on social democratic parties. Here, social democrats have often been outcompeted by moderate right and progressive left parties. Importantly, most social democratic parties have only lost a small share of their supporters to the radical right.

Instead, social democrats have largely failed to construct an agenda that both communicates a clear vision of economic policy, but is not only focused on economics.

In the arena of economic policy, social democratic parties remain a central progressive force in Europe. In Britain, through the 00s, Labour’s policies dramatically improved public services, cut child poverty and improved the fates of millions of low-income families. After a decade of labour market reform, the German SPD spent the 2010s fighting to protect pensions and income security in coalition with the CDU. And in the Nordic countries, the social democrats are robust defenders of the welfare state. However, social democrats in the post financial-crash era clearly did offer too little to voters suffering from economic austerity, focusing on economic competence rather than a vision of a fairer future. As European centre-right parties moderated their position on economic and social policies, these parties were able to attract centrist former social democrats.

If the economic vision was too undefined, what about the social vision? A commonplace argument is that a shift away from progressive policies on immigration, European integration and gender equality could help social democratic parties (and Labour) win back working-class voters. The rise of both new working and middle classes means that this strategy is bound to fail in the long run. Moreover, political science research shows that this approach does little to actually attract these groups in the short run. Voters who support left economic policies also tend to favour more equitable gender relations, racial equity and a greener future. New left and green parties have often picked up voters with these demands, further squeezing social democrats.

What, then, is to be done? Social democratic parties need to ask themselves how they can build a broad and durable electoral coalition. They need to overcome an image of the working class as white men in the production sector with conservative cultural attitudes. The truth is that today’s working class is ethnically diverse and often holds progressive positions, even on the hotly contested issue of immigration.

Here, Labour is in fact in a stronger position than many of its European sister parties. Labour’s mobilisation through the 2010s attracted a swath of new voters to the party. In recent elections, the average age of social democrats in France and Germany was 58 and 57, but for Labour it was 45. The challenge is bridging this younger base with a broader appeal. The experience of the Biden administration suggests that articulating a more visible, progressive strategy on macro-economic policy, while supporting both organised labour and community organisations, could be a winning way forward.