The META tag: Search Engines

Robotic search engines (such as AltaVista, Lycos, and Google) use META tags to help them catalog the HTML pages on your website (although this is not the only thing they use). These META tag features are specific to individual search engines, and are not officially standardized. There are, however, two META tag variants which are generally accepted as standard, which I will discuss here.

Before I go into details of these semi-standardized variants of the META tag, I need to talk about the manner in which robotic search engines catalog your HTML pages, in general.

First of all, when I speak of robotic search engines, I am NOT talking about a search engine such as Yahoo, which catalogs websites based on a web developer's or site creator's notion of where on Yahoo the site should be catalogued.

Robotic search engines, such as AltaVista, use robotic "agents" (or programs) to search out every URL known to them and retrieve every HTML page found at those URLs to store in their own database. These robotic agents then follow every hyper-reference on every page that they find, and retrieve and catalog THOSE pages; then they follow every hyper-reference on THOSE pages, and so on and so on, until they have retrieved and catalogued every HTML page that they can find. These HTML pages are then sorted and prioritized in the search engine database based on a number of different criteria (including the META tags).

When a user accesses a robotic web search engine, he/she types one or more keywords into some sort of form text field and submits the form for processing. Based on the individual search engine's sorting criteria, a number of suitable matches to the submitted keywords are collected in an HTML page and displayed to the user.

For instance, I could type in the keywords "blue" and "jeans" into a robotic search engine, and find all of the web pages which deal with, talk about, or sell "blue jeans" (probably tens of thousands of choices). I could narrow the search by adding "levi" and "501" to the aforementioned search, and get more specific matches, etc.

How does the search engine figure out which of the millions of HTML pages in its database you want to see? And how does it prioritize those pages? Which HTML pages will come to pages 1 and 2 of the search results, and which HTML pages will be relegated to page 200 of the search results?

AltaVista, of all the robotic search engines, is the most forthcoming with information regarding how it catalogs, sorts, and prioritizes the HTML pages it finds; there are extensive FAQs and other documents published by AltaVista on this topic which you may review at http://www.altavista.com/. I will cover the most salient points here, which I believe to have been, to different degrees, adopted by the robotic search engine community at large.

The MOST important keywords on an HTML page, as far as a search engine is concerned, are the words marked by the TITLE tag in the HEAD of an HTML document. If a user types in keywords which are contained in the TITLE of one of your HTML pages, that page will probably float up to near the top of the search results. It is extremely important, therefore, that you ALWAYS include the most important keywords relating to your website in the TITLEs of pertinent HTML pages on your site.

Example (abbreviated):

<title>Michael Corporation: Online Pastry, Cheesecake, Trifle, and Doughnuts</title>

In the above example, I have included my company name in the TITLE, as well as the most important keywords or products that I am selling.

You obviously can't put EVERY keyword into your TITLE, only the MOST important ones. The remaining keywords will be placed in a META tag, which we will talk about momentarily.

Search engines are also affected by how close to the root directory of your website the HTML page is located. An HTML page located at the root directory of a web server for a domain is going to come up much higher in the search engine hierarchy than an HTML page nested several folders deep in the website.

Main Menu