I’m talking to you today about the Google index. We know mobile-first is coming, and it’s being rolled out as we speak. Watch for an announcement about it in your Google Search Console. This, by the way, is just awesome for finding out things about the index that you didn’t previously know. We were doing a search for one particular client and found a whole bunch of issues that were previously hidden.
One of the main problems we had was trying to get their site indexed properly. When we looked at their Search Console we could see that they had thousands of errors popping up, including over 24,000 soft 404s. Now, a soft 404 is basically when Google is thinking you want your server to respond with a 404 error page, but what you’ve actually done is a redirect.
In this situation, the Googlebot was being redirected and interpreting that as a soft 404, which is a big problem. After some digging we found that people weren’t being redirected, just the Googlebot, and only the main Googlebot that’s attached to the main desktop. To make matters worse, this is exclusively a Magento problem. We worked on isolating the trouble (which was a bit of a trick) and found that it’s a user validation issue. We tried using the mobile Googlebot on our Magento site, and it was redirected just as it should have been. Even more surprising, that seemed to fix the problem, and the desktop Googlebot is now behaving itself!
Now, this is an intermittent problem that’s not even hitting all Magento sites all the time. But for now, if you’ve got a site that’s having indexing problems, switch off user agent verification. That should get things back where they should be, and your traffic and revenue should rise quickly after that.
This article originally published on stewartmedia.com.au.
Jim Stewart is a leading expert in search engine optimisation. His business StewArt Media has worked with clients including Mars, M2 and the City of Melbourne.
