Top websites block Google from training AI models on their data

Top websites are increasingly denying Google the use of their content for AI model training. Although the code 'robots.txt' has allowed them to control such access for years, the rise of powerful AI, such as OpenAI's platforms, has changed the dynamic.

These AIs can often directly answer queries, potentially reducing the traffic websites receive from Google.

In response, Google introduced a new tool called Google-Extended in September, allowing sites to specifically block their data from being used to train AI. Roughly 10% of the top 1,000 websites have activated this feature.

Notably, The New York Times, which is embroiled in an AI copyright dispute with OpenAI, has applied this block, as well as prohibiting any scraping of its content without express permission.

Interestingly, this move by Google has seen less adoption compared to OpenAI's GPTBot, which is being used by about 32% of the top sites. When queried, Originality.ai CEO Jonathan Gillham suggested that Google's less dominant position could be due to concerns that blocking Google might result in being excluded from future AI-generated search results.

Google is currently experimenting with an advanced AI-powered search feature, but it remains to be seen how this will impact the traditional search model and the broader web landscape.