XML Reference Guide

So you've got a web site. Naturally, you want people to be able to find it. Specifically, you want it indexed, or included in, Google. You could go ahead and submit your main page, and then wait for Google to visit. Google's spider, Googlebot, then indexes the main page, every page linked from that page, every page linked from those pages, and so on.

Now that's all well and good, but it does leave you with a couple of problems. First off, what if you have content that's not linked from any of your main pages? Or content that's changed, and that you want Google to revisit?

On the other hand, look at it from Google's point of view. Google wants to index, well, everything on the web, whether it's linked or not. And they want to do it as efficiently as possible, picking up changes quickly without having to spider millions of pages that haven't changed.

Obviously, there's room for some collaboration here, in the form of a way to tell Google what you've got and just how fresh it really is. So Google formulated the Google Sitemap Protocol, a way for webmasters (and webmistresses) to do just that.

Google provides two different ways to create a sitemap, or a list of content to index, and two different ways to submit that sitemap for processing. Let's start with the simple way.

The easiest way to create a sitemap is to list all of your URLs in a text file, with one URL per line. For example, I can create a simple sitemap for my blog:

http://www.chaosmagnet.com/
http://www.chaosmagnet.com/blog/archives/cat_personal.html
http://www.chaosmagnet.com/blog/archives/cat_nicks_publications.html
http://www.chaosmagnet.com/blog/archives/cat_politics.html
http://www.chaosmagnet.com/blog/archives/cat_programming_tips.html
http://www.chaosmagnet.com/blog/archives/cat_rants.html
http://www.chaosmagnet.com/blog/archives/cat_reader_questions.html
http://www.chaosmagnet.com/blog/archives/cat_technology.html
http://www.chaosmagnet.com/blog/archives/cat_web_development.html
http://www.chaosmagnet.com/blog/archives/cat_web_services.html
http://www.chaosmagnet.com/blog/archives/cat_xml.html
http://www.chaosmagnet.com/blog/archives/cat_fun_stuff.html
http://www.chaosmagnet.com/blog/archives/000650.html
http://www.chaosmagnet.com/blog/archives/000649.html
http://www.chaosmagnet.com/blog/archives/000648.html
http://www.chaosmagnet.com/blog/archives/000647.html
http://www.chaosmagnet.com/blog/archives/000646.html
http://www.chaosmagnet.com/blog/archives/000645.html
http://www.chaosmagnet.com/blog/archives/000644.html
http://www.chaosmagnet.com/blog/archives/000643.html
http://www.chaosmagnet.com/blog/archives/000642.html
http://www.chaosmagnet.com/blog/archives/000641.html
http://www.chaosmagnet.com/blog/archives/000640.html

I can then save this information in a file, plainsitemap.txt, and upload it to my site so that it's available at

http://www.chaosmagnet.com/plainsitemap.txt

Note that the actual location of the file doesn't matter, except in relation to its content. For example, if I place the file at

http://www.chaosmagnet.com/blog/simplesitemap.txt

I can only list URLs in the blog directory and it's subdirectories. I couldn't, for example, list

http://www.chaosmagnet.com/howto/index.php/Upload_a_file

because Google presumes that if you have write access to a directory, you have control (or at least authority) over its contents. Otherwise, all bets are off. So if you place the sitemap in the server's root directory, you can list any URL for that server.

To submit the sitemap using a simple HTTP request, I can point my browser to

http://www.google.com/webmasters/sitemaps/ping?sitemap=http%3A%2F%2Fwww.chaosmagnet.com%2Fplainsitemap.txt

(Notice that everything after the question mark is actually data being submitted to a script, so you have to URL encode it. That means that the colon (:) becomes %3A, and slashes (/) become %2F. For more information on URL encoding, check out the reference and converter at http://www.blooberry.com/indexdot/html/topics/urlencoding.htm.)

The resulting message tells me the file was accepted:

That takes care of the basic issue of letting Google know what files are out there, but doesn't address the issues of recently updated or frequently changing files. To solve those problems, we need to use the Google Sitemap Protocol's XML format.

Consider, for example, this file, showing pages from my web site:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.google.com/schemas/sitemap/0.84"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84
	http://www.google.com/schemas/sitemap/0.84/sitemap.xsd">
   <url>

      <loc>http://www.nicholaschase.com/</loc>
      <lastmod>2005-01-05</lastmod>
      <changefreq>yearly</changefreq>
   </url>

   <url>
      <loc>http://www.nicholaschase.com/blog/</loc>
      <lastmod>2005-06-24</lastmod>
      <changefreq>daily</changefreq>

      <priority>1.0</priority>
   </url>
   <url>
      <loc>http://www.nicholaschase.com/blog/archives/cat_personal.html</loc>
      <lastmod>2005-06-17</lastmod>

      <changefreq>weekly</changefreq>
   </url>
   <url>
      <loc>http://www.nicholaschase.com/blog/archives/cat_fun_stuff.html</loc>
      <lastmod>2005-05-25</lastmod>

      <changefreq>weekly</changefreq>
   </url>
   <url>
      <loc>http://www.nicholaschase.com/blog/archives/000650.html</loc>
      <lastmod>2005-06-17</lastmod>

      <changefreq>never</changefreq>
   </url>
   <url>
      <loc>http://www.nicholaschase.com/blog/archives/000649.html</loc>
      <lastmod>2005-05-28</lastmod>

      <changefreq>never</changefreq>
   </url>
   <url>
      <loc>http://www.nicholaschase.com/howto/index.php/Main_Page</loc>
      <lastmod>2005-04-13</lastmod>

      <changefreq>weekly</changefreq>
   </url>
   <url>
      <loc>http://www.nicholaschase.com/howto/index.php/Get_submitted_form_values</loc>
      <lastmod>2005-04-24</lastmod>

      <changefreq>weekly</changefreq>
   </url>
   <url>
      <loc>http://www.nicholaschase.com/howto/index.php/Upload_a_file</loc>
      <lastmod>2005-04-17</lastmod>

   </url>
   <url>
      <loc>http://www.nicholaschase.com/howto/index.php/Get_the_current_date</loc>
      <lastmod>2005-01-30</lastmod>
   </url>

</urlset>

Let's look at what each of these elements means. First off, the urlset element encloses all of the data, which is in the http://www.google.com/schemas/sitemap/0.84 namespace. (The other urlset attributes are optional, but make it easier to check the file with an XML validator.)

A url element represents each page that you want Google to index. That url element has one required child element, the loc element. That element contains the URL for the page. This URL can be anything, including dynamically generated URLs such as

http://www.informit.com/guides/content.asp?g=xml&seqNum=231

but because this is an XML file, you'll need to encode characters such as the ampersand (&), so this URL becomes:

http://www.informit.com/guides/content.asp?g=xml&seqNum=231

That takes care of the requirements, but it's the optional information that's really exciting.

First off, the lastmod element, which contains a date (and optionally, a time) in ISO 8601 format, tells Google when the content was last changed, so you can alert them to the fact that a page they've already spidered has new information and needs to be reindexed.

The changefreq element simplifies this process by telling Google how frequently, in general, a page changes. Possible values are "always" (for pages that are different every time you access them), "hourly", "daily", "weekly", "monthly", "yearly" and "never". This enables Google to more intelligently allocate its resources, indexing more frequently changing pages, well, more frequently, while leaving more static pages for later. Note that this is just advice to Google. Just because you mark a page as hourly doesn't mean the search engine will look at it that frequently, just as they may index yearly or never pages periodically, just in case.

Note also that it's not a good idea to artificially inflate your change frequencies. The whole point of this exercise is to get better information, so we don't need Google taking this information with a grain of salt. If you have a change you want picked up, you can always use the lastmod element to let Google know, so there's no reason to abuse the system.

The final element in this set is the priority element, a number from 0.0 to 1.0. Now, before you get all excited, this number has no effect on your site's overall ranking in Google's search results. Instead, Google uses it in a situation in which multiple pages from your site come up in a search. You know how Google adds a "Similar pages" link to certain results? The priority value lets you help choose which page gets shown in the results page. If you don't specify a value, Google uses the default value of 0.5.

Google allows you to have up to 50,000 urls in a single sitemap, but what if you have more than that? Google has created a second structure, the sitemap index document. This enables you to break up your data into multiple files, and then submit them using a single index, as follows:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.google.com/schemas/sitemap/0.84"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84
	http://www.google.com/schemas/sitemap/0.84/sitemap.xsd">
   <sitemap>
      <loc>http://www.nicholaschase.com/blogsitemap.xml</loc>
      <lastmod>2005-06-25</lastmod>

   </sitemap>
   <sitemap>
      <loc>http://www.nicholaschase.com/howtositemap.xml</loc>
      <lastmod>2005-06-25</lastmod>
   </sitemap>

</sitemapindex>

Using a sitemap index also enables you to control what information Google pulls, and how often, because only more recently changed sitemaps need to be downloaded. It's also handy when sitemap files need to generated by different means. For example, I might have Movable Type generate a sitemap for my blog, but use a separate application to generate one for the pages in my Programmer's How-To wiki.

Just as for the url element, the loc and lastmod elements specify the URL and last modification date, respectively, but for the sitemap file rather than the actual content page.

OK, so now you've got your sitemap or sitemap index file, and you've uploaded it to your server.

Now what?

Well, you can submit the file via HTTP request, but there's an even better way. If you submit your sitemap through a (free) Google account, you can see if there are errors, and you can monitor Google's progress in picking up the file.

To start, point your browser to

http://www.google.com/webmasters/sitemaps/login

and if you don't already have one, sign up for an account. Once you have one, you can log in and click Add a Sitemap to enter a URL for the system to evaluate. If the system detects any problems, the error will show under Status so you can fix it and resubmit.

At the moment, Google Sitemaps are in beta, so they may change, but from the way Google has phrased its documentation, it appears the intent is to create a general standard for all search engines, so now that it's public, it's not likely to change much between the current version, 0.84, and the final version.

And it's definitely going to be popular...