Robots who Read
Before you submit your site to a search engine, you may want to consider what you want the search engine "bot" (the program that indexes your site) to "spider" (follow). You may have pages with sensitive information, or a scrap directory full of pages in progress that you would not like to see listed.
This can be achieved 2 ways. The first way is with a robots.txt file placed in the root directory of your web, but you must have full domain privileges in order for this to work. I will cover the robots.txt file configuration in a later article.
- A quick note on the robots.txt file - do not leave it empty. This will indicate to some search engines that you do not want any part of the site indexed.
The other way to stop a number of bots from searching a page is to use META exclusion tags.
The following META tags can be used:
<META NAME="ROBOTS" CONTENT="NOINDEX">
Putting this line between your <header> and </header> tags in your HTML will prevent the bot from indexing that page.
An alternative is:
<META NAME="ROBOTS" CONTENT="NOFOLLOW">
The page will be indexed, but any hyperlinks in that page will not be spidered by the bot.
Or a combination of the two:
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
Page will not be indexed, and other links will not be followed by the bot.
Tutorials, web content and tools, software and community.
Web Marketing, eCommerce & Development solutions.
| DISCLAIMER: The content provided in this article is not warranted or guaranteed by Developer Shed, Inc. The content provided is intended for entertainment and/or educational purposes in order to introduce to the reader key ideas, concepts, and/or product reviews. As such it is incumbent upon the reader to employ real-world tactics for security and implementation of best practices. We are not liable for any negative consequences that may result from implementing any information covered in our articles or tutorials. If this is a hardware review, it is not recommended to open and/or modify your hardware. |
More Search Engine Tricks Articles
More By Developer Shed