Search Engine Tricks

  Homes arrow Search Engine Tricks arrow Robots who read
 Webmaster Tools
Base64 Encoding 
Browser Settings 
CSS Coder 
CSS Navigation Menu 
Datetime Converter 
DHTML Tooltip 
Dig Utility 
DNS Utility 
Dropdown Menu 
Fetch Content 
Fetch Header 
Floating Layer 
htaccess Generator 
HTML Encoder 
HTML Entities 
IP Convert 
Meta Tags 
Password Encryption
Password Strength
Pattern Extractor 
Ping Utility 
Pop-Up Window 
Regex Extractor 
Regex Match 
Scrollbar Color 
Source Viewer 
Syntax Highlighting 
URL Encoding 
Web Safe Colors 
Forums Sitemap 
Weekly Newsletter
Developer Updates  
Free Website Content 
 RSS  Articles
 RSS  Forums
 RSS  All Feeds
Write For Us 
Contact Us 
Site Map 
Privacy Policy 
  >>> SIGN UP!  
  Lost Password? 

Robots who read
By: Developer Shed
  • Search For More Articles!
  • Disclaimer
  • Author Terms
  • Rating:  stars stars stars stars stars / 0

    Table of Contents:

    Rate this Article: Poor Best 
      Del.ici.ous Digg
      Blink Simpy
      Google Spurl
      Y! MyWeb Furl
    Email Me Similar Content When Posted
    Add Developer Shed Article Feed To Your Site
    Email Article To Friend
    Print Version Of Article
    PDF Version Of Article




    Robots who Read

    Before you submit your site to a search engine, you may want to consider what you want the search engine "bot" (the program that indexes your site) to "spider" (follow). You may have pages with sensitive information, or a scrap directory full of pages in progress that you would not like to see listed.

    This can be achieved 2 ways. The first way is with a robots.txt file placed in the root directory of your web, but you must have full domain privileges in order for this to work. I will cover the robots.txt file configuration in a later article.

    - A quick note on the robots.txt file - do not leave it empty. This will indicate to some search engines that you do not want any part of the site indexed.

    The other way to stop a number of bots from searching a page is to use META exclusion tags.

    The following META tags can be used:


    Putting this line between your <header> and </header> tags in your HTML will prevent the bot from indexing that page.

    An alternative is:


    The page will be indexed, but any hyperlinks in that page will not be spidered by the bot.

    Or a combination of the two:


    Page will not be indexed, and other links will not be followed by the bot.

    Michael Bloch
    Tutorials, web content and tools, software and community.
    Web Marketing, eCommerce & Development solutions.

    DISCLAIMER: The content provided in this article is not warranted or guaranteed by Developer Shed, Inc. The content provided is intended for entertainment and/or educational purposes in order to introduce to the reader key ideas, concepts, and/or product reviews. As such it is incumbent upon the reader to employ real-world tactics for security and implementation of best practices. We are not liable for any negative consequences that may result from implementing any information covered in our articles or tutorials. If this is a hardware review, it is not recommended to open and/or modify your hardware.

    More Search Engine Tricks Articles
    More By Developer Shed



    - Time for Your Site`s SEO Tune-up?
    - The Basic Parts of an SEO Campaign
    - Dynamic Content Still Challenging for Search...
    - Google`s Panda Update at One Year Old
    - Why Links Don`t Count Instantly
    - Check Your Code for SEO Issues
    - To Be an SEO, Start With What You Know
    - Don`t Worry About Bad Inbound Links
    - Guard Your Google Places Listing
    - Overlooked SEO Tricks
    - A Simple Long Tail Keyword Strategy
    - Writing for the Long Tail
    - Choosing and Using Keywords
    - Seven More SEO Myths That Can Hurt Your Sit...
    - Google Tips and Tricks

    Developer Shed Affiliates


    © 2003-2019 by Developer Shed. All rights reserved. DS Cluster - Follow our Sitemap