As you know, last week, on April 26th, Google disabled the URL parameter tool within Google Search Console and stopped respecting any of the rules added to that tool. Some are asking if the recent crawl spikes they saw with Googlebot was related to this change.
Dave Smart posted on Twitter charts showing his crawl activity spiking around this time. Of course, this can easily be unrelated to the URL parameter tool. It is almost impossible to isolate this crawl spike to the URL parameter tool. But Dave did say there were no other changes around this time.
Here are his tweets:
1/3 Gone, but not forgotten! Now might be a good time to check in to see what’s being crawled.
I suspect they depreciated using what was configured here a few days back, here’s the crawl stats for a client’s sitehttps://t.co/FbBvbCWjct pic.twitter.com/yAPhONATGU— Dave Smart (@davewsmart) April 26, 2022
3/3 Judicious use of robots.txt quickly brought this back down from 9 hits per second from gbot to 1.5 / 2 per second, & (a lot more of) those actually valuable URLs. A temporary fix, the site’s re-platforming soon anyway. If your site relied on the param tools, keep an eye open.
— Dave Smart (@davewsmart) April 26, 2022
Glenn Gabe jumped in to say it is likely unrelated:
Always a strong possibility they picked up and started crawling randomly. I agree! This started ramping up on the 13th, so nearly 2 weeks ago, no other real changes, but then nothing is ever static.
— Dave Smart (@davewsmart) April 26, 2022
And John Mueller from Google agreed:
That would be my take too. The systems automatically adjust over time too, so I wouldn’t ever expect this to be completely static.
— 🦙 johnmu.xml (personal) 🦙 (@JohnMu) April 26, 2022
Of course, anything is possible. I have heard from some others privately that they saw significant crawl spikes after the URL parameter tool went away – and yes, they used that tool heavily. But again, it is impossible to confirm without having access to Google’s end.
Have you noticed this as well? What do you all think?
Forum discussion at Twitter.
