• Search Appliance

  • Overview
  • Documentation Home

• Constructing URL Patterns

• Support Resources

  • Frequently Asked Questions
  • Google Search Appliance/Google Mini Group
  • Contact Support

Constructing URL Patterns

Google Search Appliance software version 5.0
Google Mini software version 5.0
Posted October 2006

A URL pattern is a set of ordered characters to which the Google Search Appliance or Google Mini matches actual URLs that the crawler discovers. You can specify URL patterns for which your index should include matching URLs and URL patterns for which your index should exclude matching URLs. This document explains how to construct a URL pattern.

Contents

1. Introduction
2. Comments in URL Patterns
3. Case Sensitivity
4. Simple URL Patterns
   1. Matching domains
   2. Matching directories
   3. Matching files
   4. Matching protocols
   5. Matching ports
   6. Using the prefix option
   7. Using the suffix option
   8. Matching specific urls
   9. Matching specified strings
5. SMB URL Patterns
6. Exception Patterns
7. Google Regular Expressions

Introduction

A URL pattern is a set of ordered characters that is modeled after an actual URL. The URL pattern is used to match one or more specific URLs. An exception pattern starts with a hyphen (-).

URL patterns specified in the Crawl URLs page control the URLs that the search appliance includes in the index. To configure the crawl, use the Crawl and Index > Crawl URLs page in the Admin Console to enter URLs and URL patterns in the following boxes:

• Start Crawling from the Following URLs
• Follow and Crawl Only URLs with the Following Patterns
• Do Not Crawl URLs with the Following Patterns

The search appliance starts crawling from the URLs listed in the Start Crawling from the Following URLs text box. Each URL that the search appliance encounters is compared with URL patterns listed in the Follow and Crawl Only URLs with the Following Patterns and Do Not Crawl URLs with the Following Patterns text boxes.

A URL is included in the index when all of the following are true:

• The URL is reachable through the URLs specified in the Start Crawling from the Following URLs field.
• The URL matches at least one pattern in the Follow and Crawl Only URLs with the following Patterns field.
• The URL does not match an exception pattern in the Follow and Crawl Only URLs with the following Patterns field.
• The URL meets one of the following criteria:
  • The URL does not match a pattern in the Do Not Crawl URLs with the Following Patterns field.
  • The URL matches an exception in the Do Not Crawl URLs with the Following Patterns field.

Alternatively, URLs can be excluded from an index through the use of a robots.txt file or robots meta tags.

For complete information about the Crawl URLs page, in the Admin Console, click Help Center > Crawl and Index > Crawl URLs.

Back to top

Comments in URL Patterns

A line that starts with a # (pound) character is treated as a comment, as shown in the following example.

#This is a comment.

Back to top

Case Sensitivity

URL patterns are case sensitive. The following table uses www.example.com/ to illustrate an example that does not match the URL pattern, and another example that does match the pattern.

URL Pattern	www.example.com
Invalid URL	http://www.EXAMPLE.com/mypage.html
Matching URL	http://www.example.com/mypage.html

The Google Search Appliance and Google Mini treats URLs as case-sensitive, because URLs that differ only by case can legitimately be different pages. To capture URLs with variable case use a regular expression. More information about regular expressions, see the Google Regular Expressions section of this document.

Back to top

Simple URL Patterns

The following notation is used throughout this document:

• Brackets < > denote variable strings in the expression format.
• The slash (/) at the end of the site name is required.

Format	<site>/
Example	www.example.com/

Matching domains

To match URLs from all sites in the same domain, specify the domain name. The following example matches all sites in the domain example.com.

Format	<domain>/
Example	example.com/
Matching URLs	www.example.com support.example.com sales.example.com

Matching directories

To describe URLs that are in a specific directory or in one of its sub-directories, specify the directory and any sub-directory in the pattern.

The following example matches all URLs in the products directory and all sub-directories under products on the site sales.example.com.

Format	<site>/<directory>/
Example	sales.example.com/products/
Matching URLs	sales.example.com/products/about.html http://www.sales.example.com/products/cost/priceList.html

The following example matches all URLs in the products directory and all sub-directories under products on all sites in the example.com domain.

Format	<domain>/<directory>/
Example	example.com/products/
Matching URLs	accounting.example.com/products/prices.htm example.com/products/expensive.htm

The following example matches all URLs in an images directory or sub-directory, in any side.

Note: If one of the pages on a site links to another external site or domain, this example would also match the /image/ directories of those external sites.

Format	/<directory>/
Example	/images/
Matching URLs	www.example1423.com/images/myVacation/ www.EXAMPLE.com/images/tomato.jpg sales.example.com/images/

Matching files

To match a specific file, specify its name in the pattern and add the dollar ($) character to the end of the pattern. Each of the following examples will only match one page.

Format	<site>/<directory>/<file>$
Example	www.example.com/products/foo.html

Format	<domain>/<directory>/<file>$
Example	example.com/products/foo.html

Format	/<directory>/<file>$
Example	/products/foo.html

Format	/<file>$
Example	/mypage.html

Without the dollar ($) character at the end of the pattern, the URL pattern may match more than one page.

Format	/<directory>/<file>
Example	/products/mypage.html
Matching URLs	/products/mypage.html /product/mypage.html /products/mypage.htmlx

Matching protocols

To match URLs that are accessible by a specific protocol, specify the protocol in the pattern. The following example matches HTTPS URLs that contain the products directory.

Format	<protocol>://<site>/<path>/
Example	https://www.example.com/products/mydir/mydoc.txt/

Matching ports

To match URLs that are accessible by means of a specific port, specify the port number in the pattern. If you don't specify a port, the search appliance uses the default port, which is 80 for HTTP and 443 for HTTPS.

• This example matches host www.example.com on any port: www.example.com:*/
• This example matches host www.example.com on port 8888: www.example.com:8888/
• This example matches host www.example.com on the default port: www.example.com/

Note: If you explicitly include a port number, the pattern matches only URLs that explicitly include the port number, even if you use the default port. For example, a URL pattern that includes www.example.com:80/products/ does not match www.example.com/products/.

Using the prefix option

To match the beginning of a URL, add the caret (^) character to the start of the pattern. Do not match a prefix character followed by only a protocol because the result could resolve to most of the Internet.

Format	^<protocol>://<site>/<directory>/
Example	^http://www.example.com/products/

Format	^<protocol>://<site>/
Example	^http://www.example.com/

Format	^<protocol>
Example	^https

Format	^<protocol>://<partial_site>
Example	^http://www.example
Matching URLs	http://www.example.com/ http://www.example.de/ http://www.example.co.jp/

Using the suffix option

To match the end of a URL, add the dollar ($) character to the end of the pattern.

The following example matches http://www.example.com/mypage.jhtml, but not http://www.example.com/mypage.jhtml;jsessionid=HDUENB2947WSSJ23.

Format	<protocol>://<site>/<directory>/<file>$
Example	http://www.example.com/mypage.jhtml$

Format	<site>/<directory>/<file>$
Example	www.example.com/products/mypage.html$

Format	<domain>/<directory>/<file>$
Example	example.com/products/mypage.html$

Format	/<directory>/<file>$
Example	/products/mypage.html$

The following example matches mypage.htm, but does not match mypage.html.

Format	<file>$
Example	mypage.htm$

The following example is useful for specifying all files of a certain type, including .html, .doc, .ppt, and .gif.

Format	<partial_file_name>$
Example	.doc$

Matching specific urls

To exactly match a single URL, use both caret (^) and dollar ($). The following example matches only the URL: http://www.example.com/mypage.jhtml

Format	^<exact url>$
Example	^http://www.example.com/mypage.jhtml$

Matching specified strings

To match URLs with a specified string use the contains: prefix. The following example matches any URL containing the string "product."

Format	contains:<string>
Example	contains:product
Matching URLs	http://www.example.com/products/mypage.html https://sales.example.com/production_details/inventory.xls

Back to top

SMB URL Patterns

To match SMB (Server Message Block) URLs, the pattern must begin with the smb: protocol. SMB URLs refer to objects that are available on SMB-based file systems, including files, directories, shares, and hosts. SMB URLs use only forward slashes. Some environments, such as Microsoft Windows, use backslashes ("\") to separate file path components. However, for these URL patterns, you must use forward slashes.

The following example shows the correct structure of an SMB URL.

Format	smb://<host>/<share>/<directory>/<file>
Example	smb://fileserver/myshare/mydir/mydoc.txt

The following SMB URL patterns are not supported:

• Top-level SMB URLs, such as the following: smb://
• URLs that omit the hostname, such as the following: smb://myshare/mydir/
• URLs with workgroup identifiers in place of hostnames, such as the following: smb://workgroupID/myshare/

 

Back to top

Exception Patterns

To specify exception patterns, prefix the expression with a hyphen (-). The following example includes sites in the example.com domain, but excludes secret.example.com.

Format	-<expression>
Example	example.com/ -secret.example.com/

The following example excludes any URL that contains content_type=calendar.

Example	-contains:content_type=calendar

You can override the exception interpretation of the hyphen (-) character by preceding the hyphen (-) with a plus (+).

Example	+-products.xls$
Matching URLs	http://www.example.com/products/new-products.xls

Back to top

Google Regular Expressions

A Google regular expression describes a complex set of URLs. For more information on GNU regular expressions, see the Google Search for "gnu regular expression tutorial". Google regular expressions are similar to GNU regular expressions, with the exception of the following differences:

• A case insensitive expression starts with the following prefix: regexpIgnoreCase:
• A case sensitive expression does not require a prefix, but the regexpCase: and regexp: prefixes can be used to specify case sensitivity.
• Special characters are escaped with a double backslash (\\).

Metacharacters are either a special character or special character combination, which is used in a regular expression to match a specific portion of a pattern. Metacharacters are not used as literals in regular expressions. The following list describes available metacharacters and metacharacter combinations:

• The . character matches any character.
• The .* character combination matches any number of characters.
• The ^ character specifies the start of a string.
• The $ character specifies the end of a string.
• The [0-9a-zA-Z]+ character combination matches a sequence of alphanumeric characters.
• The following characters must be preceded with the double backslash (\\) escape sequence:   ^.[$()|*+?{\

The following example matches any URL that references an images directory on www.example.com using the HTTP protocol.

Example	regexp:http://www\\.example\\.com.*/images/
Matching URLs	http://www.example.com/images/logo.gif http://www.example.com/products/images/widget.jpg

The following example matches any URL in which the server name starts with auth and the URL contains .com.

Example	regexpCase:http://auth.*\\.com/
Matching URLs	http://auth.www.example.com/mypage.html http://auth.sales.example.com/about/corporate.htm

This example does not match http://AUTH.engineering.example.com/mypage.html because the expression is case sensitive.

The following pattern matches JHTML pages from site www.example.com. These pages have the jsessionid, type=content parameters, and id.

Example	regexp:^http://www\\.example\\.com/page\\.jhtml;jsessionid= [0-9a-zA-Z]+&type=content&id=[0-9a-zA-Z]+$
Matching URLs	http://www.example.com/page.jhtml;jsessionid= A93KF8M18M5XP&type=content&id=gpw9483

Note: Do not begin or end a URL pattern with period+asterisk (.*) if you are using the regexp: prefix, as this pattern is ineffective and may cause performance problems.

Back to top

Note: Invalid regular expression patterns entered on the Crawl and Index > Crawl URLs page in the Admin Console can cause search appliance crawling to fail.

Back to top

©2008 Google - Privacy Policy - Terms and Conditions - About Google