Search Engine Robots - How They Work, What They Do (Part II)

If your site isn't found in the search engines, it is probably because the robots couldn't deal with it. It could be something as simple as not being able to find the site, or it may be more complicated issues involving the robot's not being able to crawl the site or figure out what your pages are all about.

Submitting your site to the major search engines: that will help with the "can't find it" problem. Even having links pointing back to your site can be enough to attract the search engine robots. Google, for example, suggests that you may not have to submit your pages; they will find your site if you have a link pointing back to it from at least one other site on the web.

If the robots can find your site but can't make sense of it, then you may need to look at the content and technology used on your pages. Frames, Flash, dynamically generated pages, and invalid HTML source code can cause problems when the search engine robot tries to access your web pages. While some search engines are beginning to be able to index dynamically generated pages and Flash (e.g. Google and AllTheWeb), use of some of these technologies can hinder your ability to be indexed by the search engine robots.

Text in images cannot be read by the search engine robots. Using ALT image text is an important way to help the robots "read" your images. Websites with extensive images rely heavily on ALT text to present their content. How Do I Get The Most Out Of Indexing?

If you know what to "feed" the spidering robots you will help yourself with search engine ranking.

Having a website full of good content is the major factor. Search engines exist to serve their visitors, not to rank your website. You need to be sure to present yourself in your site in the way that will be most useful to the search engine visitor. Each search engine has its own idea of what is important in a page, but they all value text highly. Making sure that the text on your pages includes your most important keyword phrases will help the search engine evaluate the content of those pages.

Making sure that you have good title and meta tags will further assist the search engines in understanding what your page is about. If the text on the page is about widgets, the title is about widgets, and the meta tags are about widgets, the search engine will have a pretty good idea that you are all about widgets. When their visitors search for widgets, the search engines know to list your site in the results.

A sitemap page is a very good way of giving the search engine robot every opportunity to reach your website pages. Since robots click through the links of your web pages, make sure that at least your most important pages are included in the sitemap; you may even want to include all your pages there, depending on the size of your site. Be sure to add a link to the sitemap page from each page on your site.

Another important consideration is that of keeping all of your pages within a small number of "clicks" from your top page. Many robots will not follow links more than two or three levels deep, so if your "widgets" page can only be reached from your home page by following multiple links (e.g. home page >> about us page >> products page >> widgets page), the robot may not crawl deep enough to get to the widgets page. Testing Your Website For Search Engine Robot Accessibility

To get an idea just what the search engine robot "sees" on your page, you can look at the Sim Spider tool. You may be surprised at how different your site looks to the robot. You can find this tool at http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

You will see text and ALT image text show up in the results. If your entire website is built in Flash, you will see nothing at all because robots don't understand Flash movies. The Bottom Line

When it comes to search engine robots, think simply. Lots of good content and text, hyperlinks the robots can follow, optimization of your pages, topical links pointing back to your site and a sitemap will help insure the best results when the robots come visiting.

Resources

*SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html

*Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/

*Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/

*Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/

*Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company.

Copyright © 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.

Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.

More SEO Articles

Search Engine Robots - How They Work, What They Do (Part I)
Automated search engine robots, sometimes called "spiders" or "crawlers", are the seekers of web pages. How do they work? What is it they really do? Why are they important?You'd think with all the fuss about indexing web pages to add to search engine databases, that robots would be great and powerful beings.

Organic SEO: Patience For Long Term Ranking Results
When does long term SEO show ranking results? It takes time for optimization to produce targeted traffic to your website. Organic SEO requires time to take effect, just as it takes time for your web pages to start showing up in the search engine results.

Top 10 Little Used SEO Strategies
There are millions of web sites trying to get listed in the top 20 to 30 spots on the major search engines. That's a lot of competition! So if you're having a hard time getting those spots, maybe you should try some little used but effective seo strategies to get those prized listings.

Local Search Optimization - A Guide to Getting Started
While searching the web these days, it's hard not to notice all those little Local tabs sprouting up in the vicinity of the search field on virtually every major search engine. Within the past year, the race has been to integrate a plethora of advanced features into local search capabilities.

So, Where Has Your Search Engine Been Today?
Visit Google, Yahoo, MSN or one of the lesser search engines, and you get a few million results for just about any search term. Despite this impressive depth of results, most users consider only a few of the WebPages being pointed to.

How to Get the Ranking You Always Wanted!
Is your web site well ranked (In the top ten search results) in the results? If not, you need to read this and get the ranking you always dreamed of getting with your web site! I will show you how to get your web site a top ten ranking in the search engines with these few easy-to-do steps.Site Design - The web site's structure will decide how the spiders read your web site and the speed of them indexing the web pages.

Its Not Just All About Google Anymore
Those webmasters that stick to the old ways and focus entirely on Google are missing out on a lot of search traffic these days if they are not also well ranked by Yahoo and MSN.For the first few months after Yahoo decided to go their own way with natural search (and MSN decided to get serious about the search business), the search results provided by those two could only be described as bizarre.

DIY SEO - Part 2 Tags n Things
Part 2 Tips 4 tagsMeta tags are the descriptive tags used by website designers to explain to the search engines exactly what your website is all about. They look like this,title Put title of your page here, include important keywords /titlemeta name="Description" content="Put a short keyword rich description of your site here" /meta name="Keywords" content="Put your keywords in here" /meta name="AUTHOR" content="Your name or company name here" /meta name="COPYRIGHT" content=" Your name or company name here " /meta name="DISTRIBUTION" content="Global" /meta name="RATING" content="General" /meta name="robots" content="index, follow" /meta name="revisit-after" content="1 Week" /meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /Note: the opening and closing brackets have been missed off to make this article compatable for publication.

Easy Steps to Get Onto Google Top Search Pages
To get on Google?s top pages can be accomplished by following a few simple steps. The following procedure works well for companies working within a local region i.

Basic Search Engine Promotion
I can't tell you how many times people have asked me, "How to do I get my site to come up in the top ten of the search engines so I can get hits and sell my product?"What they really want to know is, "How can I get people to show up to my website without spending any money on advertising my site!"People usually don't like my answer because accomplishing a "top ten ranking" on any search engine always requires careful thought along with a significant amount of work.Three types of "Search engines" abound on the Internet.