URL Redirection and Robots.txt

Someone at Picsearch replied to my complaint about PSBot’s naughty behavior. He apologized and said they’d fixed it so this site wouldn’t be indexed again.

There was this curious remark:

A general note however: computertoaster.com/robots.txt redirects to www.computertoaster.com/robots.txt, which would prevent bots from stopping to request URL:s on computertoaster.com (I do not know if this is what you are seeing from us or not).

Eh? I want PSBot to stop requesting for sure. Or am I being told the redirection would prevent the bot from being stopped?

Related posts:

  1. Robots.txt : A Webmaster’s Guide ...
  2. Robots.Txt Explained How to write your site's robots text file. ...
  3. Cut Search Robots Drain on Your Bandwidth ...
  4. How To Control Search Engine Robots ...
  5. Playing in Googlebot’s Sandbox with Slurp, Teoma & MSNbot ...

Related posts brought to you by Yet Another Related Posts Plugin.

0 comments ↓

There are no comments yet...Kick things off by filling out the form below.

Leave a Comment