Seen my personal blog lately? Probably not, if you were searching via Google. Major sections of my site have been disappearing from the search index over the past three weeks. My homepage, my blog and many of the most recent articles on it no longer showed up in result pages. I’m no Matt Cutts, but I get a fair number of people coming to my site when searching for info about Google search, avoiding scams, and how to name their baby. All that traffic has been slipping away.
You can probably imagine how you would feel if this was happening to you. Does Google hate me? Was my site hacked? What do I do, and how much will it cost to get this fixed?
I will answer all of those questions, starting with the first:
My site is falling out of the index, does Google hate me?
Probably not. My situation is actually pretty illustrative – I’m pretty sure Google doesn’t hate me and isn’t unfairly slapping my site down because, well, I work at Google.
That’s right, Google was kicking pages from one of its own employees out of search results. I’m sure I’m not the first. Google doesn’t treat my site any differently than anyone else’s. BTW, standard disclaimers apply to this post.
So I knew there was probably a logical reason for the dropped pages, which brings me to the next question:
Is Google dropping my pages from search results because my site got hacked?
This is a very, very good question to ask – hacking is unfortunately common. This very site has been hacked before. I don’t want to go into a lot of detail on how to tell if your site was hacked in this post, but the Webmaster Central Blog has some good pointers.
Really, this is part of a broader question – what has changed with my site? In my case it wasn’t hacking – it turns out out that Google was getting tons of crawl errors over the past couple weeks. As Googlebot tried to recrawl my site looking for new content, it kept getting network unreachable errors. After days and days of this, Googlebot figured I had closed up shop. We don’t want to send searchers to pages that have disappeared.
Why all the errors? Here’s where I had to do some digging – I have a multi-site account with a web host. It turns out my site had exceeded my bandwidth quota. The worst thing is it’s not the max quota I paid for, but just the amount I had portioned off for my blog. I didn’t think I’d ever get enough visitors to worry about it. D’oh.
What do I do, and how much will it cost to get this fixed?
If your site starts disappearing from Google search results, how can you figure out what’s going on without access to all the uber-powerful, super-secret Google tools that I used?
It turns out I only needed one Google tool to diagnose the problem, and I’ll let you in on the secret. In fact, I’ll leak the url (SEO bloggers get ready to tweet!):
http://www.google.com/webmasters/tools/
Yep, good old Webmaster Tools. Between the reports there and my own host’s dashboard, I was able to figure out and fix the problem in less than 20 minutes. Googlebot will take longer than that to reindex everything but I noticed progress almost immediately:
I looked under Site Diagnostics -> Crawl Errors to see all the pages that Google couldn’t reach. After I fixed the bandwidth cap on my side, I noticed that under Sitemaps, my sitemap status had a little red “X” as well. I logged into to my blog and regenerated the sitemap, checked that it was there, and clicked “Resubmit” in Webmaster Tools.
Voila! Total cost: $0.
Bonus Question: This sounds embarrassing, why are you sharing it with everyone?
Having my own site start disappearing from my employer’s search engine isn’t exactly something to brag about. It’s doubly embarrassing to admit how long it took for me to notice what was going on, though I bet a lot of other site owners are in the same boat, to busy doing their day jobs to constantly check search rankings.
I wanted to share my story because #1, it might help someone figure out what to do if they have a similar issue, and #2, it illustrates a bit about how Google tries to do business.
Everything is set up to give users the best results we can, and to cope with all the spam and abuse on the web. Sites that are irrelevant, unresponsive, or violate the webmaster guidelines might not end up in front of searchers, even if it’s my site. Heck, even if it’s Google Japan.
Thank you so much! This was helpful.
Hi
First of all thanks a lot for the great info. I have been struggling out in forums and google forums to figure this out. I really couldn’t find the right answer.
Well, after coming upon your blog i have now a new hope to get back everything that i lost.
I will keep my progress updated if any good news comes up.
Regards