They have seen the passing of the American Indian and the buffalo; and now they query as to how long the Anglo-Saxon may be able to survive.” So wrote William Ripley in an essay for the Atlantic Monthly in 1908.

Ripley was one of America’s most renowned academics, professor of political economy at Columbia University and of economics at MIT. By the early 20th century, his chief preoccupation had become the fear that white people, which, to the American elite at least, meant “Anglo-Saxon”, were vanishing from the population.

“Whereas, until about 20 years ago, our immigrants were drawn from the Anglo-Saxon or Teutonic populations of north-western Europe,” Ripley wrote, “they have swarmed over here in rapidly growing proportions since that time from Mediterranean, Slavic, and Oriental sources.” He was worried that Anglo-Saxons were showing a “low and declining birthrate” while “the immigrant horde… has continued to reproduce upon our soil with well-sustained energy”.

A century on, such fears of “white decline” have returned in certain circles. Until recently, issues of “white identity” and of “white decline” were confined to the far-right fringes, where notions such as that of the “great replacement”, the belief that white people are being driven out of their “homelands”, flourished.

Increasingly, though, mainstream conservatives – academics, politicians, writers – are taking up the call. The latest comes in an essay by the novelist Lionel Shriver in the Spectator. It’s entitled “Would you want London to be overrun with Americans like me?” But it’s not Americans like her that worries Shriver. Her fear, rather, is that “white Britons” are “becoming a minority in the UK”. “The lineages of white Britons in their homeland commonly go back hundreds of years,” she writes, and yet they have to “submissively accept” the “ethnic transformation of the UK… without a peep of protest”.

For Britain to remain Britain, it has to remain predominantly white. To say so, Shriver insists, is not racist. It is difficult, though, to know what else it could be. “For westerners to passively accept and even abet incursions by foreigners so massive that the native-born are effectively surrendering their territory without a shot fired,” Shriver claims, “is biologically perverse”.

This is the language of the British National party, of the AfD in Germany, of Marine Le Pen in France. To describe immigration as “incursions by foreigners”, to view black or brown people moving into your town as “surrendering one’s territory” and to regard non-white immigration as “biologically perverse” is not just to stray into racist territory, it is to jump head first into the swamp.

There are legitimate arguments to be had about immigration. I am more liberal about immigration than most, but I have long argued that we should not, as many do, dismiss those calling for tighter controls as “racist”; some are, most probably not. Conservative proponents of tighter controls react with fury at the charge that their arguments may be racist, insisting they are concerned purely about numbers. Increasingly, though, the question of race has become explicit. Shriver’s is but the latest in a series of arguments by prominent conservatives bemoaning the decline of the white population or defending the legitimacy of white “racial self-interest”.

Many conservatives argue that in their obsession with white decline, they are speaking for working-class white people, whose views have been neglected by the metropolitan elite. Yet nine out of 10 Britons disagree with the claim that “to be truly British you have to be white”. Two-thirds of Americans think that the declining share of white people in the US population is “neither good nor bad”. On both sides of the Atlantic, ordinary people seem more rational about whiteness than does Shriver. “White declinists” are using working-class white people as alibis for their own prejudices.

One of the ironies in all this is that many of those who most worry about “white decline” are also among the most strident critics of identity politics. Taking part in a debate in defence of the proposition that “identity politics is tearing society apart”, Shriver argued that she had been a “fierce advocate” of the US civil rights movement because its goal was “to break down the artificial barriers between us” and “to release us into seeing each other not as black or white… but as individual people”. “The colour of my skin,” she added, “is an arbitrary accident... the boxes into which I have been born are confinements I have struggled to get out of and I would wish that liberation to everyone else.”

Except, it seems, if you are a non-white immigrant. Then, the “arbitrary accident” of birth becomes an essential feature of one’s identity, the “artificial barriers between us” need to be emphasised, the “confinements” of ethnic boxes maintained and people seen not as “individuals” but as “black or white”.

Such double standards are not peculiar to Shriver. The writer Douglas Murray, for instance, is similarly hostile to identity politics, arguing “skin colour is of no significance, which is what I think and I hope society can end up thinking” while also denouncing the fact that “white Britons [are] a minority in their own capital city” and asking: “Were your derided average white voters not correct when they said that they were losing their country?” It’s a common thread running through much conservative criticism of identity politics.

In truth, rightwing critics have never been hostile to identity politics. They just want to push their form of identity politics. The critique of identitarianism is, for them, a useful weapon with which to attack the left, while promoting their own insidious notions of identity. Meanwhile, many on the left, who rightly condemn the racism of the “white declinists”, are themselves drawn to defend their own version of identity politics, not recognising that in doing so they make it easier to rebrand racism as white identity politics.

For both right and left, whiteness has come to acquire an almost magical quality. On the one side, whiteness is something to be protected, something too little of which transforms British communities, and mysteriously makes them less British. On the other, whiteness has become an embodiment of privilege or wickedness and racism seen not in social or structural terms but in the inherent qualities of being white.

It’s an obsession that replaces political argument with magical thinking and gives new legitimacy to bigotry. Racism matters. Whiteness does not.