Entries Tagged 'Search Engines' ↓

My main website zapped by Google’s Jagger updates I do something I’ve often wanted to do: move ‘content’ (quotes because this content is strictly personal writings that do have meaning to their author) to a new domain.

Last Sunday I registered the new domain. DNS information propagates quickly nowadays. Within three hours I had the 900+ pages transferred and setup my 301 redirect.

And I added a sitemap.

As of today a site:mydomain.org shows half of my pages. Not bad for five days I’d say.

Page Rank? I don’t care about Google PR that much. It’ll show up in due time.

The site is getting indexed quickly and the traffic is fine.

So you heard about someone stressing the importance of the robots.txt file, or noticed in your website’s logs that the robots.txt file is causing an error, or somehow it is on the very top of the top visited pages, or, you read some article about the death of the robots.txt file and about how you should not bother with it ever again. Or maybe you never heard of the robots.txt file but are intrigued by all that talk about spiders, robots and crawlers. In this article, I will hopefully make some sense out of all of the above.

There are many folks out there who vehemently insist on the uselessness of the robots.txt file, proclaiming it obsolete, a thing of the past, plain dead. I disagree. The robots.txt file is probably not in the top ten methods to promote your get-rich-fast affiliate website in 24 hours or less, but still plays a major role in the long run.

First of all, the robots.txt file is still a very important factor in promoting and maintaining a site, and I will show you why. Second, the robots.txt file is one of the simple means by which you can protect your privacy and/or intellectual property. I will show you how.

Let’s try to figure out some of the lingo.

Continue reading →

Given that my main site is getting fewer hits than it did when it had maybe a couple of hundred entries I’m certainly unhappy with the progress of Google’s Jagger Updates so far.

Will you hand me a crying towel? Thanks.

The mysterious entity known as Google Guy who posts on WebmasterWorld and new A List Blogger Matt Cutts have both indicated that the Jagger Update will be a three part process.

The update isn’t done until the third part careens through Google’s datacenters.

I watch with the same avid interest as the webmasters whose incomes depend on their ranking in Google’s results.

Given that I’ve owned my own brick and mortar business for over twenty years it isn’t quite the same for me. I don’t sell through search engine results.

But I do miss the readers. More properly the folks who leave comments. Some evolve into conversations. Even friendships of a virtual sort. While I’d rather not I can live without them. And who knows what the next Google algorithm change may bring.

Calling the thread on WMW a soap opera is implying that it is more thrilling reading than it is.

You have a Google fanboy waiting for the search engine to bring truth and light with the third Jagger Update. Acting almost like a knight of old hoping to defend the honor of his lady fair.

There are the conspiracy theorists: Google is doing this to force them to spend money on AdWords. I don’t think the search engine is stupid enough to court congressional investigations, or the sort of publicity that would actually send people to Yahoo! and MSN for their searches.

And lots and lots of webmasters beating their breasts: “Oh why, Oh why is Google being so faithless!” Think of it of Google as a cruel cocktease, offering then withdrawing her favor so she can cruelly mock the submissive webmaster who only offered his all.

If he got hits and income that is.

I don’t like what has happened to my main site (this one hardly matters). I’ll be patient. If the last stage of Jagger doesn’t leave me better off maybe I’ll learn something and my search engine results will be better next year.

Someone at Picsearch replied to my complaint about PSBot’s naughty behavior. He apologized and said they’d fixed it so this site wouldn’t be indexed again.

There was this curious remark:

A general note however: computertoaster.com/robots.txt redirects to www.computertoaster.com/robots.txt, which would prevent bots from stopping to request URL:s on computertoaster.com (I do not know if this is what you are seeing from us or not).

Eh? I want PSBot to stop requesting for sure. Or am I being told the redirection would prevent the bot from being stopped?

I don’t want most of my sites indexed by Picsearch. All they are indexing is affiliate and advertising images that can be found anywhere. No need for them to clutter up my server logs or add to my bandwidth. Obeying Picsearch’s instructions for telling PSBot to go away I added the following to my robots.txt file: User-agent: psbot Disallow: / But I keep having to write to Picsearch because PSBot is ignoring the exclusion. It’ll stop for a few days. Then come back. Nuisance.

A couple of months ago on one of my sites I took out a couple of 301 redirects that I’d had in place for about a year and half. No real reason to do it, just seemed tidy.

Suddenly Google and Yahoo were both looking for the original pages on another site. Including Computer Toaster, which for a long time was just a subsection of another site.

Probably old links on other sites.

Once you do a 301 you should probably never remove it unless no one has ever linked to your prior pages.

Another one for the Mysteries of Search Engines Dept:

• Inktomi Slurp 12228+1177 93.13 MB
• Googlebot 9916+74 161.85 MB

Yahoo!’s slurp is visiting more pages than Googlebot. But Googlebot is using up more bandwidth.

Maybe Slurp is merely tasting the top of the pages while Googlebot is gulping them down whole.

My main website is about five years old. Never paid much attention to traffic. I created it to amuse myself.

Well, I never paid attention to traffic until the first time I ran out of bandwidth.

To my surprise a joke reference to a very famous young female pop singer undraped had somehow made me Google’s top two results for the search.

Having no ideas about “monetizing traffic” or the like I deleted the entries to make the hits go away. But I learned to watch my referral logs.

Bemusedly I watch Google (and sometimes Yahoo and MSN as well) embrace my site, then push it away. Up and down, up and down.

A few days ago Google fell passionately in love with that site again. Traffic jumped 250%. I did nothing to cause that surge.

My site will probably fall out of favor again. And perhaps return to search engine favor.

I can’t help but feel sorry for folks whose lifeblood is search engine optimization.

Otherwise there is no moral to this story.

There are many articles out there claiming to explain the Google sandbox. At least one is relevant and worth reading.

Google’s infamously and arguably mis-titled”Sandbox Effect” has been an observed phenomenon since early 2004. Although many continue to argue and debate the causes and effects of this unusual algorithmic element, there is virtually no debate on its existence. At one time, the best explanation of the sandbox was:

Read 2005 Analysis of Google’s Sandbox

That Alexa rankings don’t have much value is a given with experienced webmasters.

With some sites I like to see how the ranking shifts. Just a minor supplement to what I get from AWStats and Webalizer.

People who are thinking about advertising on websites often take the Alexa ranking very seriously. And some of the online ad services incorporate Alexa’s stats in their summary of the sites for whom they are selling ads.

That makes it even easier for Google and other search engines to gulp up the statistics and links.

Referral spammers have caught on to this.

Have you ever wondered why all those poker sites would send so many false referrers when your site’s visitor statistics aren’t visible to the search engines.

The bogus referrals show up in Alexa’s results. Yet another way to get Google to pay attention to your URL.

Some of my sites list a huge number of visitors from gambling sites that don’t link to me.

More search engine spam for the Google, Yahoo, Jeeves and MSN to eventually find a way to filter out.

Surely makes it hard for potential website advertisers to make a decision.