How to Control Search Engine Robots - Meta tags are...
(Page 2 of 4 )
Meta Tag - Meta tags are used to provide structured data about data.
In the early 2000s, search engines veered away from reliance on Meta tags, as many web sites used inappropriate keywords, or were keyword stuffing to obtain any and all traffic possible.
Some search engines, however, still take Meta tags into some consideration when delivering results. In recent years, search engines have become smarter, penalizing websites that are cheating (by repeating the same keyword several times to get a boost in the search ranking). Instead of going up rankings, these websites will go down in rankings or, on some search engines, will be kicked off of the search engine completely.
Index a site - The act of crawling your site and gathering information.
How can the robots.txt file and meta tag help you?
In the robots.txt you can tell the harmful 'web crawlers' to leave your web site alone, and give helpful hints to the ones you want to crawl your site. Here is an example on how to disallow a web crawler to search your site:
# this identifies the wayback machine
User-agent: ia_archiver
Disallow: /
ia_archiver is the crawler name for the wayback machine that you may have heard of, and the / after disallow tells ai_archiver not to index any of your site. The #message here allows you to write comments to yourself so you can keep track of what you typed.
Type the above three lines into notepad from your computer and save it to the root directory of your web site as robots.txt. Web crawlers look for this document first at a web site before doing anything else. This helps the crawler to do its job, and helps the web site owner tell the spider what to do. Say for instance you have some data that you don't want the crawlers to see. (Like duplicate content for other browser referrer pages)
Next: You can deter... >>
More Search Engine Tricks Articles
More By Jase Dow
