Switch

A bit has been flipped on Google Search.

Previously, the Googlebot would index any web page it came across, unless a robots.txt file said otherwise.

Now, a robots.txt file is required in order for the Googlebot to index a website.

This puzzles me. Until now, Google was all about “organising the world’s information and making it accessible.” This switch-up will limit “the world’s information” to “the information on websites that have a robots.txt file.”

They’re free to do this. Despite what some people think, Google isn’t a utility. It’s a business. Other search engines are available, with different business models. Kagi. Duck Duck Go. Google != the World Wide Web.

I am curious about this latest move with Google Search though. I’d love to know if it only applies to Google’s search bot. Google has other bots out crawling the web: Adsbot-Google, Google-Extended, Googlebot-Image, GoogleOther, Mediapartners-Google. I’m probably missing a few.

If the new default only applies to the searchbot and doesn’t include say, the crawler that’s fracking the web in order train Google’s large language model, then this is how things work now:

  • Your website won’t appear in search results unless you explicitly opt in.
  • Your website will be used as training data unless you explicitly opt out.

It would be good to get some clarity on this. Alas, the Google Search team are notoriously tight-lipped so I’m not holding my breath.

Have you published a response to this? :

Responses

KB

@adactio.com If I understand Google’s video correctly, Google now looks first for a robots.txt file, and if it can’t find/reach one, it will not index your site at all. But what happens if it *does* find a robots.txt file - does it still respect the instructions in the traditional way? E.g. if you block Google from accessing a directory, does it still respect that instruction? That’s how I’m interpreting it; if you want Google to index at all, you must have a robots.txt. Weird. 🤷‍♂️

# Posted by KB on Wednesday, January 14th, 2026 at 6:19pm

Thomas

@adactio Are you sure this “Community” video is to be trusted? The whole thing makes no sense to me.

# Posted by Thomas on Wednesday, January 14th, 2026 at 7:01pm

vanderwal

@adactio I this this a few weeks back when trying to sort out why Google no longer was coming by vanderwal.net and bringing in new content, nor surfacing older content there.

In digging through how to get it kickstarted it noted I didn’t (or no longer had) a robot.txt file and I needed one.

# Posted by vanderwal on Thursday, January 15th, 2026 at 12:03am

Martin Splitt

Hey Jeremy, this isn’t quite right. Nothing changed here. If there is no robots.txt, crawling is assumed to be allowed. If there is one, Googlebot follows the rules in it. If your server gives a 5xx, Googlebot doesn’t know if there’s a robots.txt or not and then failsafes to “no crawling now”

2 Shares

# Shared by KB on Wednesday, January 14th, 2026 at 5:27pm

# Shared by Sue on Wednesday, January 14th, 2026 at 7:01pm

2 Likes

# Liked by https://feed.city/c/dan on Wednesday, January 14th, 2026 at 5:21pm

# Liked by Jordi Sánchez on Friday, January 16th, 2026 at 2:33am

Related posts

Permission

You have the power, not Google.

Related links

Pulling my site from Google over AI training – Tracy Durnell

I’m not down with Google swallowing everything posted on the internet to train their generative AI models.

Tagged with

A principled approach to evolving choice and control for web content

This would mean a lot more if it happened before the wholesale harvesting of everyone’s work.

But I’m sure Google will put a mighty fine lock on that stable door that the horse bolted from.

Tagged with

Previously on this day

1 year ago I wrote A long-awaited talk

Remy and I gave a talk at Brighton’s Async meetup …five years after we were originally booked in.

7 years ago I wrote Writing for hiring

Robot-free recruiting.

10 years ago I wrote Separated at death

Farewell, doppelgänger.

12 years ago I wrote Chüne

Churn on, Chüne in, chrop out.

16 years ago I wrote Making Workshops for the Web

Behind the scenes of the latest Clearleft site.

17 years ago I wrote Authors On Tour — Live!

For your huffduffing pleasure.

18 years ago I wrote Ten songs titles that could be Twitter updates

The kind of list that’s too geeky for McSweeney’s.

19 years ago I wrote The Best Songs I Acquired in 2006 Ever

Following Richard’s lead.

21 years ago I wrote DHTML is dead. Long live DOM Scripting.

Just in case I haven’t completely hammered the point home lately, I have a feeling that 2005 is going to see a big surge in the use of the Document Object Model with JavaScript.

23 years ago I wrote I've seen fire and I've seen rain

…but mostly rain.

24 years ago I wrote Which Kevin Smith character are you?

Excellent! I am Silent Bob, apparently:

24 years ago I wrote Biz Stone: Wrong Font

Take a look at this picture of a storefront, it’s a great example of how not to choose a font.