The other day I reported again how blocking US based users would require you to also block Googlebot, Google’s crawler. And that would be devastating to a site’s search rankings in Google. But John Mueller of Google said it is not out of the question that Google one day may decide to crawl from outside of the US.
John said as a follow up on Twitter “I could see this [where Google crawls from] expanding over time if we see a broader need.” He did add that he would be sad if this had to happen because “it would mean the web is growing in isolated bubbles, rather than being “world-wide”.” But he understands that “what we’d like to see is sometimes not what we get,” so it one day may be inevitable.
Here are those tweets:
We primarily crawl from the US (at least for the moment), so I wouldn’t count on Google trying random other countries to see where things are accessible. There are very few exceptions. That hasn’t changed.
— 🐐 John 🐐 (@JohnMu) March 2, 2022
I could see this expanding over time if we see a broader need. I think that would suck, because it would mean the web is growing in isolated bubbles, rather than being “world-wide”. That said, what we’d like to see is sometimes not what we get, & there are so many folks involved.
— 🐐 John 🐐 (@JohnMu) March 2, 2022
Now, Google previously said that it did not find any benefit to crawling from outside of the US back in 2017. There are however times Google will crawl from a specific region but it is rare.
I don’t know if Google will ever crawl from outside of the US regularly but never say never with Google.
Forum discussion at Twitter.
