Cut Search Robots Drain on Your Bandwidth

I’m not fool enough to pretend that I have any inkling why Google does things or if there was a correlation between my actions and what followed.

I saw picsearch.com’s psbot gobbling up huge amounts of bandwith on some of my sites. Sites that host no images. No pictures, not a one. Unless you followed a few affiliate links. Picsearch was grabbing them as if they were hosted on my own site. Stupid ‘bot.

So I blocked psbot in my robots.txt.

Insane bandwidth consumption dropped massively.

Oddly a week later Google’s indexing of those sites jumped back up to prior levels.

I also did a couple of other things:

Asked become.com to stop indexing a site. They were yanking more bandwidth than even greedy Jeeves. But giving me no referrals. Really when I last looked at Become.com it looked like little more than the world’s largest Amazon affiliate.

And I used the crawl-delay to slow down Yahoo’s slurp and Jeeves.

Some of the sites get a larges volumes of traffic from Yahoo but it was as if the search engine needed to keep knocking at my door like a maniac from a slasher movie.

Instead of having to think of moving the sites to a more expensive server now I can let them just stay where they are.

Ban bad search ‘bots. Do it today.

Originally posted 2005-09-20 18:57:37. Republished by Old Post Promoter

Related posts:

1. How To Control Search Engine Robots ...
2. Bandwidth and hosting implications. ...
3. Bandwidth saving from gzip ...
4. Alexa as a source of referral & search engine spam ...
5. Yahoo Search vs. Google ...

Related posts brought to you by Yet Another Related Posts Plugin.