Export to GitHub

perlsitemapgenerator - Create.wiki

[<< 1. Downloading the program files] | [3. Uploading the necessary files >>]

2. Creating a configuration file

This section provides step-by-step instructions for creating a configuration file. It also provides a complete reference of the options available. If you are creating Mobile Sitemaps, see the additional mobile guidelines.

In order to create a configuration file for your site, you must have the following information:

  • The base URL for your site (such as http://www.example.com/). Only URLs that begin with this base URL can be included in the Sitemap. Ensure that you include the protocol (such as http://). For instance, http://www.google.com is a valid base url, but www.google.com is not.
  • The web server path to the location where you want to store the Sitemap. Generally, this is the path to the base URL as the Sitemap cannot contain URLs that are in a higher-level directory from the location of the Sitemap. When you run the Google Sitemap Generator, it creates the Sitemap and places it in the location you specify.
  • The methods you want the Sitemap Generator to use to create your Sitemap. You can use any combination of methods. The following methods are available:
    • URL —list individual URLs in this section of the configuration file, along with information about each URL. You would generally use this method in conjunction with another method to manually include additional URLs that other methods wouldn't pick up.
    • URL list —point the configuration file to a text file that contains a list of URLs. You might want to use this method if this text file already exists or if you use a script to generate a list of URLs.
    • Directory paths —specify the directory paths for your site and corresponding URLs to those paths. The Sitemap Generator will create a list of URLs based on the contents of those directories. You might want to use this method if your site consists of static HTML files.
    • Access logs —point to the path to your log files. The Sitemap Generator will create a list of URLs based on the URLs included in the logs. You might want to use this method if your site consists of dynamic pages.
    • Sitemap —point to existing Sitemaps that you have created with the Sitemap Generator. The Sitemap Generator will create a single Sitemap that includes the URLs contained in each Sitemap. You could use this method if you have already created several smaller Sitemaps that you want to combine into one larger Sitemap.

Create the configuration file as follows:

  1. Open the example_config.xml file in a text editor. Save it as a new file (such as config.xml or mysite_config.xml).
  2. Locate the site definition section: <site base_url="http://www.example.com/" store_into="/var/www/docroot/sitemap.xml.gz" verbose="1">
  3. Change the base_url value to the URL for your site.
  4. Change the store_into value to the path on your web server where you want to store the Sitemap and the filename you want to use for the Sitemap. Generally, this is the path to the base URL since Google can only accept URLs that are at the same level as or subdirectories of the directory that holds the Sitemap. You can specify a relative path from the directory where you upload the script or a complete path from the root of your web server. If you upload the script to your base URL directory, you can simply specify the filename.
  5. Locate the generation method sections that begin with MODIFY or DELETE . Each of these sections corresponds to a method for generating a Sitemap.
  6. Delete the sections for the methods you aren’t going to use.

  7. Follow the instructions below for the methods you are going to use.

    • URL

Locate the following section: ```

Required attributes: href - the URL

Optional attributes: lastmod - timestamp of last modification (ISO8601 format) changefreq - how often content at this URL is usually updated priority - value 0.0 to 1.0 of relative importance in your site -->

```

This section gives two examples: the first includes only the required attribute and the second includes the required attribute as well as the optional attributes.

Use this format for each of the URLs you want to include. The changefreq attribute gives Google a general idea of how often the URL is updated. This helps Google know how often to visit the page for new content. The priority attribute gives Google information about the relative importance of this page compared to the other pages of your site. This attribute has no effect on how Google compares your page to pages on other sites, it just helps Google know which pages of your site that you think are most important.

  • URL List

Locate the following section: ```

Required attributes: path - path to the file

Optional attributes: encoding - encoding of the file if not US-ASCII --> ```

Use this format to point to the path and name of the text file that contains your list of URLs. You can use the provided example_urllist.txt file as a template for that text file. You can specify either a relative or complete path to your web server. For instance, if the Sitemap Generator and urlist.txt file are located in the same directory, you can simply specify the filename of the .txt file, If you create a text file with an encoding other than UTF-8, you can use the encoding attribute to indicate this encoding. If you have multiple .txt files, you can use wildcards. For instance: <urllist path="example_urllist*.txt" encoding="UTF-8" /> For each URL you include in the text file, you can specify the last modification date, change frequency, and priority. See the URLlist text file reference section for complete information about the structure of this file.

  • Directory paths

Locate the following section: ```

Required attributes: path - path to begin walking from url - URL equivalent of that path

Optional attributes: default_file - name of the index or default file for directory URLs

--> ```

This section gives two examples. If all of your pages are contained in subdirectories of one path, then you only need to include one entry. However, if you have multiple paths to pages on your site, include an entry for each.

Remember that each URL must begin with the base URL you specified in step 3. For instance, the examples given in the example_config.xml file both have URLs that begin with http://www.example.com/. Therefore, both URLs are valid.

Replace the example entries with entries for your site. Many sites will only have one entry that points to the base URL. Ensure that path value is the complete path to the directory on your web server. Ensure that the url value is the complete URL, including the protocol (such as http) and a trailing slash, if required.

You can use the default_file parameter to specify the filename that your server uses as the default page for a directory. In the above example, /var/www/docroot resolves to http://www.example.com/index.html. You are not required to specify this. However, if you do, the Sitemap Generator will include the page that maps to each subdirectory only once (rather than list both the directory URL and filename URL) and will use the last modified date of the file (rather than the directory) to extract the lastmod attribute for that page.

  • Access logs

Locate the following section: ```

Required attributes: path - path to the file Optional attributes: encoding - encoding of the file if not US-ASCII --> ```

This section gives three examples. You should replace these entries and include an entry for each log file. Ensure that the path value is the complete path and filename on your web server. If the log files are not encoded as US-ASCII or UTF-8, then use the optional encoding attribute to specify the encoding. Rather than list each log file, you can use wildcards. For instance, in the above example, you could include the following entry that would include all three log files: <accesslog path="/etc/httpd/logs/access.log*" encoding="UTF-8" /> The Sitemap Generator assigns priority to URLs it finds in the logs based on how often each URL is accessed. For instance, a URL that has been accessed 100 times will be given a higher priority than a URL that has been accessed twice. The actual priority assignment is relative and depends on each URL as compared to other URLs in the site.

  • Sitemap

Locate the following section: ```

"sitemap" nodes tell the script to scan other Sitemap files. This can
be useful to aggregate the results of multiple runs of this script into a single Sitemap.

Required attributes: path - path to the file --> ```

This section gives one example. You should replace this entry and include an entry for each Sitemap you want to include. Ensure that the path value is the complete path and filename on your web server. You can list gzipped Sitemaps as well, as long as they have a .gz extension. Rather than list each Sitemap, you can use wildcards. For instance, the following entry would include any Sitemaps that begin with the word "sitemap" and have an .xml extension: <sitemap path="/var/www/docroot/subpath/sitemap*.xml" /> The Sitemap Generator extracts all URLs and the optional data listed for each URL for every Sitemap you list and creates one Sitemap with this information. At this time, we can't guarantee that this method will work Sitemaps created with tools other than the Sitemap Generator.

  1. Locate the filter definition section: ``` ****************************************************
    FILTERS

Filters specify wild-card patterns that the script compares against all URLs it finds. Filters can be used to exclude certain URLs from your Sitemap, for instance if you have hidden content that you hope the search engines don't find.

Filters can be either type="wildcard", which means standard path wildcards (* and ?) are used to compare against URLs, or type="regexp", which means regular expressions are used to compare.

Filters are applied in the order specified in this file. An action="drop" filter causes exclusion of matching URLs. An action="pass" filter causes inclusion of matching URLs, shortcutting any other later filters that might also match. If no filter at all matches a URL, the URL will be included. Together you can build up fairly complex rules.

The default action is "drop". The default type is "wildcard".

You can MODIFY or DELETE these entries as appropriate for your site. However, unlike above, the example entries in this section are not contrived and may be useful to you as they are. ***************************************************** -->

```

You can use filtering to exclude specific URLs from the generated Sitemap. You might want to do this to create a cleaner list, to reduce redundant listings, or to keep certain URLs from being indexed. Note that if you use a robots.txt file to keep URLs from being indexed, then even if the URLs are included in your Sitemap, Google will not search or index them.

You can use any or all of the filtering methods. You can delete the entries you don’t need and can create additional entries, if desired. Below are sample usages. <filter action="drop" type="wildcard" pattern="*.jpg" /> This filter excludes URLs that end in .jpg. You might want to include a similar filter if all of your site’s images are embedded within HTML pages and should not be accessed as standalone URLs. <filter action="pass" type="wildcard" pattern="*.htm*" /> <filter action="drop" type="wildcard" pattern="*" /> This filter includes all .htm* files but excludes everything else.

  1. Once you have made all the changes for your site, save the file.

Config File Syntax Reference

A complete explanation of the config file syntax is below. Each tag begins with a code sample, followed by a description of the attributes.

site

Required tag at the beginning of each config file. <site base_url="http://www.example.com/" store_into="/var/www/html/sitemap.xml.gz" verbose="1" supress_search_engine_notify="1" default_encoding="UTF-8">

|base_url|required|The HTTP path of the base of your website - only URLs that begin with this base can be included in the Sitemap| |:--------|:-------|:-------------------------------------------------------------------------------------------------------------| |store_into|required|The web server path to the desired output file. The script will create this file - there's no need to create the file before running the script.| |verbose |optional|Enter a number from 0-3, with higher numbers corresponding to increased debug information | |suppress_search_engine_notify|optional|Disable search engine notification by entering "1" for testing purposes | |default_encoding|optional|Specify a character encoding to be applied to file system paths and URLs |

url

Optional tag that you can use to list each URL in your site. <url href="http://www.example.com/stats?q=age" lastmod="2004-11-14T01:00:00-07:00" changefreq="yearly" priority="0.3" />

|href|required|The HTTP path of the base of your website - only URLs that begin with this base can be included in the Sitemap| |:---|:-------|:-------------------------------------------------------------------------------------------------------------| |lastmod|optional|The time the URL was last modified in W3C Datetime format (YYYY-MM-DDThh:mm:ss+00:00). You may omit the time portion. Examples: "2005-02-21T18:00:15+00:00", "2005-02-21"| |changefreq|optional|The frequency with which the URL is likely to change. This is considered a hint and not a command. The value must be one of "always", "hourly", "daily", "weekly", "monthly", "yearly", or "never".| |priority|optional|The priority of this page relative to other pages on the same site. The value is a number between 0.0 and 1.0, where 0.0 is the lowest priority and 1.0 is the highest priority. The priority can affect the order that search engines select URLs to explore on your site. Since the priority is relative, it is only used to select between URLs within your own site; the priority of your pages will not be compared to the priority of pages on other sites.|

urllist

Optional tag that you can use to point to a text file that contains a list of the URLs in your site. <urllist path="/var/www/html/urllist.txt" encoding="UTF-8" />

|path|required|The path and filename of the .txt file. You can specify either a relative or complete path.| |:---|:-------|:------------------------------------------------------------------------------------------| |encoding|optional|The encoding of the file, if not UTF-8. |

The urllist.txt file is a simple text file containing a list of URLs to map. You can also include optional attributes for each URL. Attributes are entered on the same line as the URL and are separated by a single space. For example: http://www.example.com/abc/something http://www.example.com/abc/xyy.pdf lastmod=2001-12-31T14:05:06+00:00 http://www.example.com/abc/def?x=12&y=23 changefreq=weekly priority=0.3

|lastmod|optional|The time the URL was last modified in W3C Datetime format (YYYY-MM-DDThh:mm:ss+00:00). You may omit the time portion. Examples: "2005-02-21T18:00:15+00:00", "2005-02-21"| |:------|:-------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |changefreq|optional|The frequency with which the URL is likely to change. This is considered a hint and not a command. The value must be one of "always", "hourly", "daily", "weekly", "monthly", "yearly", or "never".| |priority|optional|The priority of this page relative to other pages on the same site. The value is a number between 0.0 and 1.0, where 0.0 is the lowest priority and 1.0 is the highest priority. The priority can affect the order that search engines select URLs to explore on your site. Since the priority is relative, it is only used to select between URLs within your own site; the priority of your pages will not be compared to the priority of pages on other sites.|

directory

Optional tag that you can use to specify directories in your site so the Sitemap Generator can create a list of URLs from the files found in those directories. <directory path="/var/www/icons" url="http://www.example.com/images/" /> <directory path="/var/www/docroot" url="http://www.example.com/" default_file="index.html" />

|path|required|States the initial path. Sitemap Generator will traverse this directory and all subdirectories.| |:---|:-------|:----------------------------------------------------------------------------------------------| |url |required|Specifies the URL equivalent of the path value. | |default_file|optional|Specifies the default file for a directory on the server. |

accesslog

Optional tag that you can use to specify the path and filename of IIS and Apache-style access logs so the Sitemap Generator can automatically pick up URLs from them. <accesslog path="/etc/httpd/logs/access-0.log" encoding="UTF-8"/>

|path|required|States the path to the file.| |:---|:-------|:---------------------------| |encoding|optional|Specifies encoding of the file, if not UTF-8.|

sitemap

Optional tag that you can use to specify the path and filename of existing Sitemaps that you have created with the Sitemap Generator. The Sitemap Generator will create a single Sitemap that includes the URLs contained in each Sitemap. <sitemap path="/var/www/docroot/subpath/sitemap.xml" />

|path|required|States the path to the Sitemap file.| |:---|:-------|:-----------------------------------|

filter

Optional tag that you can use to build rules that include or exclude specific files. Filters are obeyed in the order in which they appear in the config.xml file. However, intermixing filter entries and input entries (url, urllist, directory, or accesslog) has no additional effect - every URL the Sitemap Generator adds to the Sitemap is first compared against every filter. If no filter matches a URL, the default is to include the URL in the Sitemap. <filter action="drop" type="wildcard" pattern="*/internal/*" />

|action|optional|The action the filter should take. Valid values are: drop - excludes matching URLs. This is the default action, so if no action is specified, the generator assumes "drop". pass - includes matching URLs.| |:-----|:-------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |type |optional|The type of filtering. Valid values are: wildcard - standard path wildcards (? and *) are used to compare against URLs. This is the default type, so if no type is specified, the generator assumes "wildcard". regexp - regular expressions are used.| |pattern|required|Specifies the pattern to match against. |

Encodings

Files referenced by your configuration file, either URL lists or web server logs, can use encodings other than the default UTF-8. You can specify alternate encodings in config.xml to affect how the Sitemap Generator reads your files. Some common encodings are:

  • encoding="utf-8" is the assumed default
  • encoding="ascii" is a subset of UTF-8 so you don't have to specify it
  • encoding="iso-8859-1" is common for many west European languages

Additional information for creating a mobile configuration file

You create a configuration file for a Mobile Sitemap in the same way as for a non-mobile Sitemap. However, you must create a separate config file for each markup language and run the Sitemap Generator with each config file separately so that you create a separate Sitemap for each.

Each config file must:

  • Specify a different filename for the store_into value.
  • Use filters to specify the URLs to exclude and include for the markup language. Remember that each Sitemap should include URLs for only one markup language. This means that the same URL may be included in multiple Sitemaps, if those URLs serve multiple markup languages.

Examples of filtering

Below are some examples of how you can use extension-based filters to generate Mobile Sitemaps for different markup languages. The specific filtering you use should be based on the types of markup languages used in your site, and how you specify each type. If you have implemented the details of your site differently (for instance, you may organize URLs with different markup languages in separate folders), you should filter based on the specifics of your site implementation. Remember that filters are applied in the order you list them in the config file. So, the first filter you should list is a "pass" action that specifies the URLs you want to include in the Sitemap.

To create a Sitemap for WML (WAP 1.2) content: <filter action="pass" type="wildcard" pattern="*.wml" /> <filter action="drop" type="wildcard" pattern="*.*" /> To create a Sitemap for XHTML mobile profile (WAP 2.0) content: <filter action="pass" type="wildcard" pattern="*.xhtml" /> <filter action="drop" type="wildcard" pattern="*.*" />

[<< 1. Downloading the program files] | [3. Uploading the necessary files >>]