Search Bots, Crawlers, and Spiders |
If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time? Not quite sure what I am talking about? Here is a few examples of various bots searching my website: 207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot. 207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot. 66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense. What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites. What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered. Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard. How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do. How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website. What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days. Ken Dennis
http://KenDennis-RSS.homeip.net/
SEO Tips for Google
Getting a high ranking on Google is a big achievement. There are many factors that go into pulling a high page rank.
1 Simple SEO Strategy To Get More Visitors To Your Site From Google
Did you know that you can dramatically increase the number of
visitors that come to your site on a daily basis from Google?
And it's not constantly improving your position in Google search
engine result pages(SERPs) for your competitive keywords which
can take some time after working on your search engine marketing
campaigns.I take this example from Google because I've experienced it some
time back now.
Get Listed in Google Without Submitting Your Site
With Google delivering so much traffic, it is only normal to be eager to submit your page and have it indexed as soon as possible. However, submitting your page is not your only option, and it's not the best one.
A Way for Search Engines to Improve
Wouldn't it be nice if the search engines could comprehend our impressions of search results and adjust their databases accordingly? Properly optimized web pages would show up well in contextual searches and be rewarded with favorable reviews and listings. Pages which were spam or which had content that did not properly match the query would get negative responses and be pushed down in the search results.
Search Engines: Tips and Strategies on Getting Listed and Ranking High for Newbies
You?ve got a website. You?ve put countless hours into it, tweaking the look and feel and making sure all the links work.
Google Patent Application - SEO Highlights
The recent patent application filed by Google details
numerous items the search engine uses to rank web pages. The
specific application is summarized as:?A method for scoring a document, comprising: identifying a
document; obtaining one or more types of history data
associated with the document; and generating a score for the
document based on the one or more types of history data.
Ten Steps To A Well Optimized Website - Step 3: Site Structure
Welcome to part three in this search engine positioning series. Last week we discussed the importance and considerations that much be made while creating the content that will provide the highest ROI for your optimization efforts.
Ten Steps To A Well Optimized Website - Step 4: Content Optimization
Welcome to part four in this search engine positioning series. Last week we discussed the importance of the structure of your website and the best practices for creating an easily spidered and easily read site.
King + Queen = Royal Success
Ever wonder how or why your competitor gets better search engine rankings than you do? Does he know something you don't? Well, maybe he actually does?And that's where I come in. I'm here to unveil the secret tricks of the trade that may be giving your competitor that much-needed edge in search engine rankings and to show you how you can reclaim your position without skipping a beat.
Ten Steps To A Well Optimized Website - Step 5: Internal Linking
Welcome to part five in this search engine positioning series. Last week we discussed the importance of content optimization.
|
