Search Protocol Reference

Google Search Appliance software version 6.0
Posted June 2009

Google has developed a simple HTTP-based protocol for serving search results that enables you to control how search results are requested and presented to an end user. This guide describes the technical details of search requests and results. This guide assumes that you have a basic understanding of the HTTP protocol and the HTML document format.

For terminology definitions, see the Google Enterprise Glossary.

Contents

1. Introduction
2. Request Format
   1. Request Overview
      1. Submitting a Search Request
      2. Search Request Examples
   2. Search Parameters
      1. Custom Parameters
   3. Query Terms
      1. Special Characters: Query Term Separators
      2. Special Query Terms
   4. Filtering
      1. Automatic Filtering
      2. Language Filters
   5. Internationalization
      1. Character Encoding Values
   6. Sorting
      1. Sort By Relevance (Default)
      2. Sort By Date
   7. Meta Tags
      1. Requesting Meta Tag Values
      2. Filtering by Meta Tags
      3. Using inmeta to Filter by Meta Tags
      4. Limits
3. Results Format
   1. Custom HTML
      1. Custom HTML Output
      2. Internationalization
   2. XML Output
      1. XML Output Overview
      2. Character Encoding Conventions
      3. Google XML Results DTD
      4. Google XML Tag Definitions
4. Dynamic Result Clustering Service /cluster Protocol
   1. Dynamic Result Clustering JSON Request and Response
   2. Dynamic Result Clustering XML Request and Response
5. Query Suggestion Service /suggest Protocol
   1. Query Suggestions Parameters
   2. Query Suggestions Requests and Responses
6. Appendices
   1. Appendix A: Estimated vs. Actual Number of Results
      1. Counting Results in Secure Search
      2. How Number of Results Returned is Determined
      3. Navigation
      4. Automatic Filtering
   2. Appendix B: URL Encoding
   3. Appendix C: Date Formatting
      1. Acceptable Date Formats
      2. Date Formatting Notes
      3. Examples of Rules

Introduction

The Google Search Appliance accepts search requests as input, and returns search results as output.

Search requests, the input, are simple HTTP requests to the Google search engine. Search users typically use HTML forms displayed in a web browser to make these requests, but other applications can also send search requests by making appropriate HTTP requests. For information on the search request format and options, see Request Format.

Search results, the output, are returned in either HTML or XML formats, as specified in the search request.

HTML-formatted results can be displayed directly in a web browser. The search appliance generates HTML results by applying an XSL stylesheet to the XML results. You can customize the appearance of the HTML results by modifying this stylesheet. For more information, see Custom HTML Output Overview.

XML-formatted output makes it possible to process the search results in web applications or other environments. For information on the XML results format, see XML Output.

Note: In this guide, long URLs may appear as multiple lines for better readability. In a browser, all URLs are continuous strings.

Request Format

The information in this section helps you create custom searches for your web site. By using search parameters, special query terms and filters in your search requests, you can refine and enhance searches to serve your needs.

This section contains:

• Request Overview
• Search Parameters
• Query Terms
• Filtering
• Internationalization
• Sorting
• Meta Tags
• Limits

Request Overview

Using the Google search protocol is as simple as requesting a page from a web server. The Google search request is a standard HTTP GET command, which returns results in either XML or HTML format, as specified in the search request.

The search request is a URL that combines the following:

• Your Google Search Appliance host name or IP address, which were assigned when the search appliance was set up
• Search interface port (usually 80)
• A path describing the search query. The path starts with "/search?", and is followed by one or more name-value pairs (input parameters) separated by the ampersand (&) character.

Submitting a Search Request

Typically, search users make search requests by entering search parameters in a HTML form rendered in a web browser (like the following):

<form method="GET" action="http://search.mycompany.com/search">
   <input type="text" name="q" size="32" maxlength="256" value="query string">
   <input type="submit" name="btnG" value="Google Search">
   <input type="hidden" name="site" value="default_collection">
   <input type="hidden" name="client" value="default_frontend">
   <input type="hidden" name="output" value="xml_no_dtd">
   <input type="hidden" name="proxystylesheet" value="default_frontend">
</form> 

Such forms are the most recognizable methods for generating GET requests, but there are numerous other ways. For example, a web page may include a direct link that brings users to a page of search results:

http://search.mycompany.com/search?q=query+string
                           &site=default_collection 
                           &client=default_frontend
                           &output=xml_no_dtd
                           &proxystylesheet=default_frontend

Alternatively, a web application may make a HTTP GET request directly:

GET /search?q=query+string&site=default_collection 
                           &client=default_frontend 
                           &output=xml_no_dtd 
                           &proxystylesheet=default_frontend HTTP/1.0

Each of these examples results in the same GET request. The HTTP response to this request contains the first page of search results for the query "query string", restricted to URLs in the collection named "default_collection." The results are rendered into HTML format using the XSL stylesheet associated with the front end named "default_frontend".

You can search multiple collections by separating collection names with the OR character ( | ) or the AND character (.), for example: &site=col1.col2 or &site=col1|col2.

The rest of the examples that follow use the raw HTTP GET format (as in the last example).

Back to top

Search Request Examples

Example 1. This request returns the first 10 results that match the search query terms "bill" and "material":
GET /search?q=bill+material&output=xml&client=test&site=operations

Explanation:

The search query is "bill material".
GET /search?q=bill+material&output=xml&client=test&site=operations

Search is limited to the documents in the "operations" collection.
GET /search?q=bill+material&output=xml&client=test&site=operations

Results are returned in the Google XML output format.
GET /search?q=bill+material&output=xml&client=test&site=operations

Example 2. This request returns results numbered 11-15 that match the same query terms and collection as example 1. As specified by the proxystylesheet parameter, the results are rendered in the custom HTML output format defined by the front end named "test."
GET /search?q=bill+material&start=10&num=5&output=xml_no_dtd&proxystylesheet=test&client=test&site=operations

Explanation:

This search request uses the same search query terms and collection as in Example 1.
GET /search?q=bill+material&start=10&num=5&output=xml_no_dtd&proxystylesheet=test&client=test&site=operations

Results numbered 11 - 15 are returned.
GET /search?q=bill+material&start=10&num=5&output=xml_no_dtd&proxystylesheet=test&client=test&site=operations

Results are returned in custom HTML output format, which is created by applying the XSL stylesheet associated with the "test" front end to the standard XML results. See proxystylesheet.
GET /search?q=bill+material&start=10&num=5&output=xml_no_dtd&proxystylesheet=test&client=test&site=operations

Example 3. This request returns the first 10 German results that match the search query "Star Wars Episode +I":
GET /search?q=Star+Wars+Episode+%2BI&output=xml_no_dtd&lr=lang_de&ie=latin1&oe=latin1&client=test&site=movies
&proxystylesheet=test

Explanation:

The search query term is "Star Wars Episode +I". Search is limited to documents in the "movies" collection.
GET /search?q=Star+Wars+Episode+%2BI&output=xml_no_dtd&lr=lang_de&ie=latin1&oe=latin1&client=test&site=movies
&proxystylesheet=test

Results show the first 10 German results.
GET /search?q=Star+Wars+Episode+%2BI&output=xml_no_dtd&lr=lang_de&ie=latin1&oe=latin1&client=test&site=movies
&proxystylesheet=test

Results are returned in Google custom HTML output format, which is created by applying the XSL stylesheet associated with the "test" front end to the standard XML results.
GET /search?q=Star+Wars+Episode+%2BI&output=xml_no_dtd&lr=lang_de&ie=latin1&oe=latin1&client=test&site=movies
&proxystylesheet=test

Back to top

Search Parameters

This section lists the valid name-value pairs that can be used in a search request and describes how these parameters modify the search results.

All search requests must include the parameters site, client, and output. All parameter values must be URL-encoded, except where otherwise noted.

Back to top

Custom Parameters

In addition to the Search Parameters, you can also define custom parameters in a search request. The search appliance returns custom parameters and their values in the search results.

For security reasons, all space characters in a custom parameter are replaced by an underscore (_). For example:

http://search.customer.com/search?q=customer+query
 &site=collection
 &client=collection
 &output=xml_no_dtd
 &myparam=test+this

This search request includes the custom parameter myparam with a value of test+this . The space character (represented as "+") in the custom parameter myparam is replaced by the underscore character (_) in the XML output.

The resulting XML output looks like this:

<param name="q" value="customer query" original_value="customer+query"/>
<param name="myparam" value="test_this" original_value="test+this" />

The unmodified value can be retrieved from the original_value attribute.

Query Terms

By default, Google returns only pages that include all of your search terms. You do not need to include "AND" between terms. The order of search terms affects the search results. To further restrict a search, just include more terms.

Google may ignore common words and characters such as where and how and other digits and letters that slow down a search without improving the results.

If a common word is essential to getting the results you want, you can include the word by putting a plus sign (+) in front of it. Make sure to include a space before the plus sign. For example, to ensure that Google includes the "I" in a search for "Star Wars Episode I", enter the search query as follows:

Star Wars Episode +I

Special Characters: Query Term Separators

By default, non-alphanumeric characters in a search query separate the query terms in the same way as space characters.
The following characters are exceptions:

If a document contains a number, with or without a decimal point, that has letters immediately before or after it, the letters are treated as a separate word or words. For example, the string 802.11a is indexed as two separate words, 802.11 and a.

Note: An underbar is not a query term separator. For example, if you search for taino_the_parrot, the only valid search result is a document that contains the exact phrase, taino_the_parrot. A search for taino or parrot will not return the taino_the_parrot result.

Back to top

Special Query Terms

Google search supports the following special query terms. A user or search administrator can use these terms to access additional search features.

Note: All query terms must be correctly URL-encoded in a search request.

Back to top

Filtering

Google search provides many ways for you to filter the results that are returned from your search query. In addition to the automatic filtering and language filtering described in this section, the search appliance provides filtering by query parameters, query terms and meta tags, which are documented in their respective sections.

Automatic Filtering

Google uses automatic filtering to ensure the highest quality search results.

Google search uses two types of automatic filters:

• Duplicate Snippet Filter - If multiple documents contain identical titles as well as the same information in their snippets in response to a query, only the most relevant document of that set is displayed in the results.
• Duplicate Directory Filter - If there are many results in a single web directory, then only the two most relevant results for that directory are displayed. An output flag indicates that more results are available from that directory.

By default, both of these filters are enabled. You can disable or enable the filters by using the filter parameter settings as shown in the table.

When a search filter is enabled and removes some results, the search results output indicates that results were filtered. See Estimated vs. Actual Number of Results for more information about how a filtered result set is identified and for recommendations for displaying the results.

Although the filter=0 option exists, Google recommends against setting filter=0 for typical search requests, because filtering significantly enhances the quality of most search results.

When the Google Search Appliance filters results, the top 1000 most relevant URLs are found before the filters are applied. A URL that is beyond the top 1000 most relevant results is not affected if you change the filter settings.

Back to top

Language Filters

This section covers:

• Automatic Language Filters
• Combining Language Filters

Automatic Language Filters

Language filters limit a search to pages in the specified languages. The algorithm for automatically determining the language of a web document is not customizable. The language determination algorithm is mainly based on the majority language used in the web document body text.

Note: Encoding schemes for input and output of search requests are important when providing international search. For more information, see Internationalization.

The automatic language filters are:

Combining Language Filters

Search requests that use the lr parameter support the Boolean operators identified in the following table in order of precedence.

Note: Spaces are not valid characters in the collection string.

Back to top

Internationalization

To support searching documents in multiple languages and character encodings, Google provides the ie and oe parameters. The ie parameter indicates how to interpret characters in the search request. The oe parameter indicates how to encode characters in the search results. To appropriately decode the search query and correctly encode the search results, supply the correct ie and oe parameters, respectively, in the search request.

Note: When you are providing search for multiple languages, Google recommends using utf8 encoding value for the ie and oe parameters.

Examples

Example 1. The following search request interprets the search query "gloves" using latin1 encoding , searches for English or French results, and returns results using latin1 encoding:

GET /search?q=gloves&client=test&site=test&lr=lang_en|lang_fr&ie=latin1&oe=latin1

Example 2. This request interprets the search query "gloves" using latin2 encoding, searches for results which are not in Hungarian or Czech, and returns results using latin2 encoding:

GET /search?q=gloves&client=test&site=test&lr=(-lang_hu).(-lang_cs)&ie=latin2&oe=latin2

Example 3. This request interprets the search query "gloves" using utf8 encoding, searches for results which are in Simplified or Traditional Chinese, and returns results using utf8 encoding:

GET /search?q=gloves&client=test&site=test&lr=lang_zh-CN|lang_zh-TW&ie=utf8&oe=utf8

Note: For information on language-specific searches that use the lr parameter, see Language Filters.

Character Encoding Values

Here is a list of encoding values that can be used with the parameters ie and oe:

Back to top

Sorting

Google search provides two sorting options for search results:

• Sort By Relevance
• Sort By Date

Sort By Relevance (Default)

By default, Google combines hypertext-matching analysis and PageRank technologies to provide users with highly relevant results. Hypertext-matching analysis uses the design of the page, examining over 100 factors to determine the best result for your query term. PageRank considers the link structure of the entire index to understand how each page links to the other pages in the index.

Sort By Date

Google search engine can order search results by date in ascending or descending order. The date of a web document is defined by parameters configured by the search administrator. When a search request uses the sort-by-date feature, the date associated with each result document is used to determine the order of the results.

When using the sort-by-date feature, the automatic quality filter will sometimes re-order results when performing result grouping. This can be disabled by adding the filter=0 parameter to the search request when performing search by date.

Example

The following request returns the first 10 top results that match the query "chicken teriyaki" in the "test" collection:

GET /search?q=chicken+teriyaki&output=xml&client=test&site=test&sort=date:D:S:d1

Results are sorted by date and relevancy.

Details

To sort the results by date, include the sort parameter in the search request, formatted as follows:

date:<direction>:<mode>:<format>

The following table shows the possible values for <direction>, <mode>, and <format>.

Back to top

Meta Tags

Google search engine provides search parameters and special query terms that enable you to leverage the meta tags that are available in your content. These make it possible to find matches specifically in meta data content, rather than content occurring anywhere in the document.

A results page can display matches for up to 64 meta tags.

This section describes the following methods of using meta data:

• Requesting meta tag values using the getfields parameter
• Filtering by meta tags using the requiredfields or partialfields parameters
• Using inmeta to filter by meta tags

Requesting Meta Tag Values

Use the getfields parameter in a search request to specify meta tag values to return with the search results. The search engine returns only meta tag information for results that actually contain the meta tags. The search for meta tags is case-insensitive. Use only whole words in the getfields parameter, not partial words or word "stems." There is a limit of 320 characters returned for each meta tag when using getfields. This character limit includes the meta tag name and content.

Usage

GET /search?q=[search term]&output=xml&client=test&site=test&getfields=[meta tag name]

Example

The following search request returns the first 10 results that match the query "books" in the "test" collection:

GET /search?q=books&output=xml&client=[test]&site=[test]&getfields=author.title.keywords

If any of the results contain the author, title or keywords meta tags, then the values of those meta tags are returned with the respective results. For example, the following tags could be returned with this search request:

<meta name="author" content="Jakob Nielsen">
<meta name="title" content="Usability Engineering">
<meta name="keywords" content="Usability, User Interface, User Feedback">

Details

To specify multiple meta tag values to be returned, list all meta tag names separated by a period (.) as in the first example. To request all available meta tags for each search result, specify an asterisk (*) as the value for the getfields parameter.

When meta tag values are requested, they are not displayed in results requested in the default HTML format. You can use the custom HTML or XML output options, or set the XSLT variable show_meta_tags to display meta tags in results. For more information, see Creating the Search Experience: Advanced Customization Topics.

All specified meta tag names and values must be double URL-encoded.

Filtering by Meta Tags

The search appliance can filter results by the values of the results' meta tags. This section describes how to use the requiredfields and partialfields input parameters to filter results using meta tag values. You can use these parameters to include only search results that contain specified meta tag values.

The term partialfields refers to part of the meta tag content, rather than part of a word. For information on other filtering techniques, see Filtering.

You can use the operators in the following table when filtering by meta tags.

Operator Note: A search can be performed to find all documents containing a set of words and/or metadata, such as A AND B AND C. These terms can also be negated, such as A AND B AND NOT C. The terms can also be converted into an OR condition, such as (A1 OR A2) AND B AND (C1 OR C2), but neither NOT nor OR can contain any other operator inside them. Searches for (A OR NOT B) and ((A AND B) OR C) and NOT(A AND B) are not supported and do not return results.

Usage

GET /search?q=[search term]&output=xml
                           &client=test
                           &site=test
                           &requiredfields=[meta tag name]:[meta tag content]       

The q= parameter is optional when using requiredfields or partialfields parameters, however, the whole query needs to have at least one positive term, be it part of the query or in the metadata restricts.

Examples

Example 1:

The following search request returns the first 10 results that match the query "checks" in the "test" collection and also contain either of the following meta tags (the %2520 operator in the GET statement shows double encoding where %20 (space) is double encoded so that the % character (hexadecimal 25) is appended to the hexadecimal 20):

<META NAME="department" CONTENT="Human Resources">
<META NAME="department" CONTENT="Finance">

GET /search?q=checks&output=xml&client=test
                               &site=test
                               &requiredfields=department:Human%2520Resources|department:Finance

Example 2:

The following search returns the first 10 results that match the query "checks" in the "test" collection that do NOT contain the following meta tag:

<META NAME="department" CONTENT="Engineering">

GET //search?q=checks&output=xml&client=test
                                &site=test
                                &requiredfields=-department:Engineering

Example 3.

The following search request returns the first 10 results that match the query "books" in the "test" collection, and also contain the word "Scott" somewhere in the "author" meta tag. Some example meta tags that satisfy this search request are:

<META NAME="author" CONTENT="Sir Walter Scott">
<META NAME="author" CONTENT="F. Scott Fitzgerald">

GET /search?q=books&output=xml
                   &client=test
                   &site=test
                   &partialfields=author:Scott

Details

Multiple meta tag constraints can be specified using the requiredfields and partialfields parameters. To filter for the presence of a meta tag, indicate the name of the meta tag to be found. To filter on a specific meta tag value, indicate the name of the meta tag followed by the colon ":" character and then the specific value. The partialfields parameter matches complete words, not parts of words.

To combine multiple name-value pairs, use the following Boolean operators.

• Boolean OR [ | ]

  Returns results that satisfy either meta tag constraint.
  Example: department:Sales|department:Finance

• Boolean AND [ . ]

  Returns results that satisfy both meta tag constraints.
  Example: author:William.author:Jones

• Combined OR and AND with [ ( ) ]

  Evaluates conditions in parentheses first: (department=Sales OR department=Finance) AND (author=Williams OR author=Jones).
  Example: (department:Sales|department:Finance).(author:William|author:Jones)

Boolean operators are left associative with equal precedence. You can use parentheses to change the order of precedence. For example, A . (B | C | D) evaluates the OR (|) operators in the parentheses before the AND (.) operator.

Note: Not all combinations are valid for searches. Only the AND operator can have other nested operators underneath. The OR operator can only be applied to one or more positive terms, and the NOT operator can only be applied to a single positive term. No other operators can be nested under the OR or NOT operators.

Example: None of the following expressions are supported by the search appliance: (A OR NOT B), ((A AND B) OR C) and NOT(A AND B).

Searches with unsupported expressions are not performed and do not return results.

By default, non-alphanumeric characters in a partialfields query separate the query terms in the same way as space characters. Generally use spaces as separators even when the original content used different content as a separator. For example if you were trying to do a partialfields query for the following meta tag:

<meta name="part" content="aaa-bbb+ccc*ddd-fff"> 

You should use queries like:

partialfields=part:aaa%20bbb
partialfields=part:bbb%20ccc

The following non-alphanumeric characters are exceptions:

<meta name="number" content="1.1222"> 

partialfields=number:1%252e1222

<meta name="pet" content="dancing.parrot"> 

partialfields=pet:dancing%20parrot

<meta name="serialnumber" content="A1.2" 

partialfields=serialnumber:A1%202  

<meta name="letters" content="a&b">

partialfields=letters:a%2526b 

<meta name="letters" content="a_b"> 

partialfields=letters:a_b

Using inmeta to Filter by Meta Tags

The special query term inmeta provides meta tag filtering directly from the search box. In combination with simple operators, inmeta filters by meta tags in the same way as the requiredfields or partialfields search parameters. You can further refine inmeta filtering using the double-period (..) separator and the daterange query term to search by number and date range. (For more information, see Query Terms.)

The special query term inmeta and relevant search parameters map to each other in this way:

Usage Notes:

1. The OR keyword separating query terms in which a date range appears returns inconsistent results.
   Examples:
   

   The following example returns one result when each portion of the query already returns one different result:

   inmeta:TainoParrot6:1..244227 OR inmeta:TainoParrot6=244228
   

   The following example returns two results and both are correct:

   inmeta:TainoParrot6=244227 OR inmeta:TainoParrot6=244228
   

   The following example returns 112 results when empty alone returns 112 results and the number range query returns 3 results:

   empty OR inmeta:TainoParrot6:244227..244229
   

   The following example returns 113 results and is correct:

   empty OR inmeta:TainoParrot6=244228
   

   The following example returns three results when yvette alone returns the same results and no results from the date range query appear:

   yvette OR inmeta:TainoParrot6:1..244228
   

   The AND operator works correctly:

   inmeta:TainoParrot6:1..244228 AND inmeta:TainoParrot6=244228
   inmeta:TainoParrot6:1..244228 AND -inmeta:TainoParrot6=244228
   

2. An OR of two inmeta range terms does not return results if the meta content uses fixed point notation.

   If a set of documents each contain a meta tag with numerical content declared in fixed point notation (with a period), two inmeta range searches that return results in isolation do not return results when combined with an OR. For example, if one document contains <META NAME="price" VALUE="20.00"> and another contains <META NAME="price" VALUE="40.00">. The search inmeta:price:15..25 returns the first document, while the search inmeta:price:35..45 returns the second document. However the search inmeta:price:15..25 OR inmeta:price:35..45 does not return results.

   If however the meta tags are defined in integer notation as <META NAME="price" VALUE="20">, the searches function correctly. This behavior is independent of whether the inmeta search term is itself declared in integer or fixed point notation. The search term inmeta:price:15.00..25.00 OR inmeta:price:35.00..45.00 does not return results when the meta tags are declared in fixed point notation, but does return results when they are declared in integer notation. In addition this is restricted only to the OR operator.

   If you have two inmeta range searches that in isolation return result sets that overlap, combining them with AND returns the intersection of those sets correctly, regardless of the notation used for the meta tag content or the range search itself.

3. An inmeta search for a number range only returns results only when a number contains six or fewer digits.

   For example, if a document contains a meta tag of <meta name="NumDateRange" content="20081230">, then a search query of inmeta:NumDateRange=20081230 works correctly, or a search where the six significant digits are respected, such as querying for inmeta:NumDateRange=1..20101230. You can use a six digit number for dates with two digits for the year, two digits for the month, and two digits for the day. If a search is made where the range includes more than six digits, then no results occur, such as with inmeta:NumDateRange=20081201..20101231.

4. An inmeta search is unable to search by multiple keywords or perform phrase searches.

   For example, consider the following meta tags:

   <meta name="department" content="Human Resources">
   <meta name="department" content="Finance">

   The following query does not work correctly:

   checks inmeta:department=Human+Resources+OR+checks inmeta:department=Finance 

   Instead, use multiple inmeta query terms, for example:

   inmeta:department=Human OR inmeta:department=Resources

5. An inmeta search of meta text with special characters, such as "." and using the operator "~" doesn't work, but using operator "=" with the full meta text will work.
6. When using daterange or inmeta queries, spelling suggestions are not returned.

   To view spelling suggestions, use the requiredfields parameter instead of inmeta.

7. An inmeta search with quotes must contain a string within the quotes. For example, the absence of a string (inmeta:name="") causes the following messages to appear in the browser:

   Message:        A server error has occurred.
   Description:    Check server response code in details.
   Details:        500
   

8. When the search appliance indexes a Microsoft Office 2007 Word document, the following metadata in meta tags becomes available for inmeta search queries:

   <meta name="Author" content="Polly Hedra"></meta>
   <meta name="Keywords" content="Resume"></meta>
   <meta name="last saved by" content="Ray Polanco"></meta>
   <meta name="revision number" content="1"></meta>
   <meta name="last print date" content="5/27/2009 14:03:00"></meta>
   <meta name="creation date" content="4/27/2009 13:15:00"></meta>
   <meta name="Last Saved Date" content="4/27/2009 13:44:00"></meta>
   <meta name="template" content="Taino Parrot Resume Template.dotx"></meta>
   <meta name="edit minutes" content="23"></meta>
   <meta name="page count" content="3"></meta>
   <meta name="word count" content="220"></meta>
   <meta name="character count" content="1512"></meta>
   <meta name="source" content="Microsoft Office Word"></meta>
   <meta name="security" content="0"></meta>
   <meta name="Count Lines" content="12"></meta>
   <meta name="Count Paragraphs" content="3"></meta>
   <meta name="Scale Crop" content="no"></meta>
   <meta name="company" content="Coqui Parrot Inc."></meta>
   <meta name="links up to date" content="no"></meta>
   <meta name="Count Characters with Space" content="1729"></meta>
   <meta name="shared doc" content="no"></meta>
   <meta name="Links Dirty" content="no"></meta>
   <meta name="Application Version" content="12.0000"></meta>
   

9. Metadata can have multiple attributes with the same name. For example:

   <metadata>
     <meta name="Name" content="Jenny Wong"/>
     <meta name="Phone" content="x12345"/>
     <meta name="Phone" content="x789"/>
     <meta name="Floor" content="3"/>
   

   If multiple values are available and if any of the attribute values match the search query, a link to the document appears in the search results.

Examples

Example 1. The following search request returns results that contain the word "Scott" somewhere in the "author" meta tag. Some example meta tags that satisfy this search request are:

<meta name="author" content="Sir Walter Scott">
<meta name="author" content="F. Scott Fitzgerald">

books inmeta:author~Scott 

Example 2. The following search request returns results that contain "size" meta tag values between 30 and 50 inches:

flat+panel+TV inmeta:size:30..50

Example 3. The following is an open-ended date range search request that returns results containing "date" meta tag values later than 2007-01-01:

Monica inmeta:date:daterange:2007-01-01..

Date meta tags must contain only the date information. If you want to filter by date meta tags, make sure the meta tag content fields do not contain any information other than a date.

Limits

Search Request Limits

The following table describes the size limits of a search request.

Meta Data Limits

The following is information on the size limits of meta data results.

Maximum number of meta tags that can be returned with getfields: 64.
Maximum number of bytes per meta tag returned, including the name of the meta tag and its contents: 320 bytes.
Maximum number of bytes of meta data returned per search result: 4K bytes.

Back to top

Results Format

This section covers the following topics:

• Custom HTML
• XML Output

Custom HTML

This section describes the custom HTML results.

• Custom HTML Output Overview
• Internationalization

Custom HTML Output Overview

Google search engine has a built-in XSLT (eXtensible Stylesheet Language Transformation) server, and can generate custom HTML using your XSL stylesheet. Search requests that include the output parameter set to xml_no_dtd and a valid proxystylesheet parameter value are automatically processed by the XSLT server as requests for custom HTML output.

Using the XSL stylesheet specified by the proxystylesheet parameter, the XSLT server applies the transformation rules found in the XSL stylesheet to the standard Google XML results. Although this document assumes that the output generated by applying the XSL stylesheet is HTML, almost any output format can be generated by using appropriate XSL stylesheet rules. For any front end, the default XSL stylesheet can be customized or replaced by the search administrator.

To customize the XSL stylesheet used to generate custom HTML output, see XML output format to determine the XML tags that may be transformed using a customized XSL stylesheet.

Additionally, you can leverage the proxycustom parameter to pass custom XML tags to the XSLT server. Because including custom XML does not generate search results, this feature is useful for implementing additional static search pages, such as an advanced search page.

Customizations to XSLT stylesheets may result in vulnerability to cross-site scripting (XSS) attacks. Google recommends that you run XSS test after customizing an XSLT stylesheet.

Notes:  

• XSL stylesheets used by the XSLT server are cached for 15 minutes. To force the XSLT server to use the latest version of an XSL stylesheet, set the proxyreload input parameter to a value of 1 in your search request.
• XSL stylesheets that include other files may not be used with the Google search engine. An XSL stylesheet that contains the following tags generates an error result:
  • <xsl:import>
  • <xsl:include>
  • xmlns:
  • document()
• When you request cached results in custom HTML output, the BLOB XML tag and associated value are automatically converted to the original text before the XSL stylesheet rules are applied. When using an XSL stylesheet that customizes cache results, simply use the values of the CACHE_LEGEND_TEXT, CACHE_LEGEND_NOTFOUND and CACHE_LEGEND_HTML XML tags directly instead of applying a rule on the BLOB subtag.
• If you use input or output encodings other than latin1, see Internationalization for more details.
• More information about XSL and XSLT can be found on the W3C web site.

Back to top

Internationalization

The Google search engine handles over 20 character encoding schemes. This section discusses special considerations for the custom HTML output format with encoding schemes other than latin1.

To support all the encoding schemes supported by Google, the XSLT server follows a process to ensure that the results are returned in the correct encoding scheme. When requesting search results through the XSLT server, the server translates the results to the UTF8 encoding scheme before applying the selected XSL stylesheet. After the XSL stylesheet rules are applied to generate the results, the results are converted to the encoding scheme that is specified by the output encoding parameter, oe. The one exception to this rule is cached result pages, which get converted to the encoding scheme of the cached document after XSLT processing.

Each front end for your search appliance is associated with an underlying stylesheet. All XSL stylesheets must be in latin1 or UTF8 formats.

XML Output

The description of the XML results format contains the following sections:

• XML Output Overview
• Character Encoding Conventions
• Google XML Results DTD
• Google XML Tag Definitions

XML Output Overview

For maximum flexibility, Google provides search results in XML format. Using the Google XML results, you can use your own XML parser to customize the display for your search users. If you are using an XSL stylesheet to transform the XML results instead of developing your own XML parser, proceed to Custom HTML.

Notes:  

• Element values are valid HTML and are suitable for display, unless otherwise noted in the XML tag definitions. Some values are URLs and must be HTML-encoded to be displayed.
• To remain forward-compatible, your XML parser that parses Google search results should ignore attributes or tags that are not documented. By ignoring unknown tags, your custom XML parser can continue working without modification when Google adds more features to the XML output in the future.
• For custom parameters that contain spaces, each space is replaced with "_". You can still retrieve the unmodified value from the original_value attribute. For example:

  <param name="temp" value="token_ring" original_value="token+ring" />

Character Encoding Conventions

The first line of the XML results indicates which character encoding is used. See XML Standard for information about character encoding.

Certain characters must be escaped when they are included as values in XML tags. These characters are documented in XML Standard, and are shown in the table that follows. All other characters in the XML results are presented without modification.

Back to top

Google XML Results DTD

Google XML results can be returned with or without a reference to the most recent DTD (Document Type Definition) describing Google's XML format. The DTD is a guide to help search administrators and XML parsers understand the XML results output. Because Google's XML grammar may change from time to time, do not configure your parser to use the DTD to validate the XML results.

XML parsers should not be configured to fetch the DTD every time a search request is performed. Because the DTD is updated infrequently, these fetches create unnecessary delay and bandwidth requirements.

To get results in XML output format, use one of the following parameters in the search request:

• output=xml_no_dtd (recommended), or
• output=xml
When you use the xml output format, the XML results include the line:

  <!DOCTYPE GSP SYSTEM "google.dtd">

The DTD is available on the Google Search Appliance at http://<appliance_hostname>/google.dtd.

Google XML Tag Definitions

This section contains an index of Google's XML tags.

Subtags Legend

? = zero or one instance of the subtag
* = zero or more instances of the subtag
+ = one or more instances of the subtag
| = Boolean OR

Index

The tables in Details list the XML tags in alphabetical order.

In the following summary table, click the first letter of an XML tag to jump to the correct section.

Details

Dynamic Result Clustering Service /cluster Protocol

Dynamic result clustering narrows searches by providing dynamically formed subcategories that appear at the top or right side of the search results.

The following illustration shows the dynamic result clustering at the top of the search results (enclosed in the red box):

The search appliance generates alternative search queries by analyzing indexed documents based on a user's current search query. The results appear as query suggestions to help the user modify the query.

You can enable dynamic result clustering for a front end in the Admin Console at Serving > Front Ends > Output Format > Search Results > Dynamic result clusters.

After enabling dynamic result clustering for a front end, the search appliance enables the XSLT spreadsheet variables to enable the feature and specify the position on the search results page for the dynamic result clustering:

<!-- *** dynamic result cluster options *** -->
<xsl:variable name="show_res_clusters">1</xsl:variable> 
<xsl:variable name="res_cluster_position">position</xsl:variable> 

Where position can be right or top.

When a user enters a query, the search appliance:

1. Uses the http://Search_Appliance/cluster.js JavaScript to provide the dynamic result clustering.
2. Fetches the /cluster content.
3. Triggers an AJAX call to the cluster service to populate the cluster position holders. The cluster position holders have the following DOM Ids depending on their position:

   <xsl:when test="$res_cluster_position = 'top'">
     <table>
       <tr>
       <td id='cluster_label0'></td>
       <td id='cluster_label2'></td>
       <td id='cluster_label4'></td>
       <td id='cluster_label6'></td>
       <td id='cluster_label8'></td>
       </tr>
       <tr>
       <td id='cluster_label1'></td>
       <td id='cluster_label3'></td>
       <td id='cluster_label5'></td>
       <td id='cluster_label7'></td>
       <td id='cluster_label9'></td>
       </tr>
     </table>
   </xsl:when>
   <xsl:when test="$res_cluster_position = 'right'">
     <ul>
       <li id='cluster_label0'></li>
       <li id='cluster_label1'></li>
       <li id='cluster_label2'></li>
       <li id='cluster_label3'></li>
       <li id='cluster_label4'></li>
       <li id='cluster_label5'></li>
       <li id='cluster_label6'></li>
       <li id='cluster_label7'></li>
       <li id='cluster_label8'></li>
       <li id='cluster_label9'></li>
     </ul>
   </xsl:when>
   

The default style sheet activates dynamic result clustering using onload attribute of the <body> tag on the search result page. The following is an example of the body opening tag:

<body onload="cs_loadClusters('{search query}', cs_drawClusters);"> 

Where {search_query} is the current search request, as shown in the following example (broken for readability):

q=culebra&btnG=Google+Search&access=p&client=default_frontend&output=xml_no_dtd&
proxystylesheet=default_frontend&sort=date%3AD%3AL%3Ad1&entqr=3&entsp=a&oe=UTF-8&
ie=UTF-8&ud=1&site=default_collection

The default XSLT stylesheet provides the clustering CSS id value for the page heading, cluster position, and loading message.

<div id='clustering'>
  <h3>Narrow your search</h3>
...

For more information, see Using Dynamic Result Clusters to Narrow Searches in Creating the Search Experience: Best Practices.

Note: The cluster.js file depends on additional JavaScript files listed in the application.

Dynamic Result Clustering Request

Administrators can test the /cluster feature by submitting a custom HTTP POST form.

The search appliance processes cluster requests:

1. The cluster request inherits all request parameters and the search appliance transports the parameters into an internal search query. If any of the /search parameters are present in the parameter list for the request to /cluster, they are passed to the internal search request.
2. If custom parameters exist, the search appliance submits the parameters without filtering.

   The POST request must have all the parameters encoded in the URI.
   The clustering service recognizes the following parameter (in addition to the /search parameters).

   Parameter	Description	Default Value
   coutput	Cluster output type: json or xml. Indicates the output you requested. Specify json for JSON output on /cluster POST requests. Specify xml for XML output as either a GET or POST. The xml value is generally used with /cluster as a RESTful service and the GET method. All request parameters must appear in the URI of a POST request.	json

   
   
3. The search appliance stylesheet adds all parameters to the request related to the current search query, as well as the custom parameters. Although the search appliance passes all parameters, not all are used.

Dynamic Result Clustering JSON Request and Response

The following example HTML provides a POST form that you can use to get JSON output (statements are wrapped for readability). The query is for the island of Culebra.

<html>
<head> <title> HTTP POST to view JSON for dynamic result clustering </title> </head>
<body>

<!-- Post parameters contiguous in a URL -->
<form method='post' action='http://Search_Appliance/cluster?q=culebra&
btnG=Google+Search&access=p&entqr=0&ud=1&sort=date%3AD%3AL%3Ad1&
output=xml_no_dtd&oe=UTF-8&ie=UTF-8&client=default_frontend&
proxystylesheet=default_frontend&site=default_collection'>  
<input type=submit value='Post'></form>
</body>
</html> 

Click the Post button to view the JSON response.

The search appliance returns the following JSON response:

{ "clusters": [
    { "algorithm": "Concepts", 
      "clusters": [
        { "label": "canada chile culebra", 
          "docs": [ 18,19,20,21,23,26,27,29,30,32] 
        }, 
        { "label": "dewey culebra", 
          "docs": [ 1,9,36] 
        } 
      ]
    }
  ],
  "documents": [ 
    { "url": "http://server.example.com/file42.pdf", 
      "title": "TLA Annual Report 2009--Acronyms in the Public Sector <b>...</b>", 
      "snippet": "<b>...</b> Soy Flz (<b>Culebra</b>) <b>Culebra</b> 
                 34,102 34,102 2.28 <b>...</b> Soy Flz (<b>Culebra</b>) 
                 was re-elected<br> Executive Director of <b>Culebra</b>, 
                 effective May 1, 2009. <b>...</b>"
    },

    ...,

    { "url": "http://server.example.com/turtle_island.html",
      "title": "Puerto Rico Travel",
      "snippet": "<b>...</b> rentals and useful information about <b>Culebra</b> <b>...</b>"
    }
  ]
}

The top-level entries are described in the following table.

Note: The dynamic result clustering service's default JavaScript client ignores the documents element and does not use the docs array.

Back to top

Dynamic Result Clustering XML Request and Response

The POST form returns XML output by adding the coutput=xml parameter to the action= URL:

<form method='post' action='http://Search_Appliance/cluster?q=culebra&coutput=xml&
btnG=Google+Search&access=p&entqr=0&ud=1&sort=date%3AD%3AL%3Ad1&output=xml_no_dtd&
oe=UTF-8&ie=UTF-8&client=default_frontend& proxystylesheet=default_frontend&
site=default_collection'>   <input type=submit value='Post'></form> 

The search appliance returns the following XML response:

<?xml version="1.0"?>
<toplevel>
  <Response>
    <algorithm data="Concepts"/>
    <t_cluster int="75"/>
    <cluster>
      <gcluster>
        <label data="canada chile culebra"/>
        <doc int="18"/>
        <doc int="19"/>
        <doc int="20"/>
        <doc int="21"/>
        <doc int="23"/>
        <doc int="26"/>
        <doc int="27"/>
        <doc int="29"/>
        <doc int="30"/>
        <doc int="32"/>
      </gcluster>
      <gcluster>
        <label data="dewey culebra"/>
        <doc int="1"/>
        <doc int="9"/>
        <doc int="36"/>
      </gcluster>
    </cluster>
  </Response>
  <t_fetch int="134"/>
  <document>
    <url data="http://server.example.com/file42.pdf"/>
    <title data="TLA Annual Report 2009--Acronyms in the Public Sector <b>...</b>"/>
    <snippet data="<b>...</b> Soy Flz (<b>Culebra</b>) <b>Culebra</b>
    34,102 34,102 2.28 <b>...</b> Soy Flz (<b>Culebra</b>)
    was re-elected<br> Executive Director of <b>Culebra</b>,
     effective May 1, 2009. <b>...</b>"/>
  </document>
  <!-- ... -->
  <document>
    <url data="http://server.example.com/turtle_island.html"/>
    <title data="Puerto Rico Travel"/>
    <snippet data="<b>...</b> rentals and useful information about <b>Culebra</b>
    <b>...</b>"/>
  </document>
</toplevel>

The top-level entries are described in the following table.

Note: The dynamic result clustering service's default JavaScript client ignores the <document> element and does not use the <doc> array. The XML response is very basic, and does not use any validations such as a DTD or XML.

The following DTD defines the XML rules, however the XML output is not validated against these rules:

<?xml version="1.0"?>

<!ELEMENT toplevel (Response, t_fetch, document+)>
<!ELEMENT Response (algorithm, t_cluster, cluster)>
<!ELEMENT cluster (gcluster+)>

<!-- each gcluster element is an alternate query and its location indexes from the top results -->
<!ELEMENT gcluster (label, doc+)>

<!-- each document element is search result, complete with url, title, and snippet -->
<!ELEMENT document (url, title, snippet)>

<!ELEMENT algorithm EMPTY>
<!ELEMENT t_fetch EMPTY>
<!ELEMENT label EMPTY>
<!ELEMENT doc EMPTY>
<!ELEMENT url EMPTY>
<!ELEMENT title EMPTY>
<!ELEMENT snippet EMPTY>

<!ATTLIST algorithm
   data (Concepts)>
<!ATTLIST t_cluster
   int CDATA #REQUIRED>
<!ATTLIST label
   data CDATA #REQUIRED>
<!ATTLIST doc
   int CDATA #REQUIRED>
<!ATTLIST url
   data CDATA #REQUIRED>
<!ATTLIST title
   data CDATA #REQUIRED>
<!ATTLIST snippet
   data CDATA #REQUIRED>

Back to top

Query Suggestion Service /suggest Protocol

This feature lists suggestions to automatically complete a user's search query. As a user enters a search query in the search box, a drop-down menu appears at the search box with suggestions to complete the search query. The search appliance uses the most popular search queries of all users to determine the top suggestions that list for a query.

You can use the /suggest feature to capture JSON response output from query suggestions and filter the information, which you then display to the user. Sites can include or exclude information in the suggestion that users view, such as to impose limits on what users can access.

Search implementations that consumes search results in XML form can use the /suggest responses to add information to their custom interface.

Enable the /suggest capability for a front end from Serving > Front Ends > Output Format > Search Box > Query suggestions. The search appliance then sets the XSLT stylesheet element show_suggest element:

<xsl:variable name="show_suggest">1</xsl:variable>

The search appliance provides access to the http://Search_Appliance/suggest_js.js JavaScript file. The JavaScript in the client makes calls to the /suggest URI and fetches the results, responding with JSON output. The AJAX response handler in the JavaScript client populates the list of suggestions.

The following JavaScript code in the XSLT stylesheet activates /suggest:

<script language='javascript'>
  sgst('q'); 
  sgst('as_q'); 
</script>

You can add this code after the search form <form/> element to activate /suggest for a search box.

To add query suggestions to the XSLT stylesheet, see Updating an Existing XSLT Stylesheet for Query Suggestions in the Guide to Software Release 6.0.

For more information on this feature, see Providing Query Suggestions in Creating the Search Experience: Best Practices.

Query Suggestions Parameters

Query suggestions provides these parameters.

Query Suggestion Requests and Responses

The following query suggestion example requests up to 10 suggestions to appear in the drop-down menu for a user:

/suggest?token=h&max_matches=10 

The following JSON response occurs (only two suggestions are currently configured for the token h):

["hay creek","hay creek2"]

Or if no suggestions are configured for the token, an empty JSON response displays:

[]

Back to top

Appendices

This section contains:

• Appendix A: Estimated vs. Actual Number of Results
• Appendix B: URL Encoding
• Appendix C: Date Formatting

Appendix A:  Estimated vs. Actual Number of Results

The Google search engine does not guarantee the ability to return a particular number of results for any given search query. The total count of results is an estimate of the actual number of results for the search request.  This section covers issues relating to this topic.

Counting Results in Secure Search

The total count of search results is not provided when a secure search is performed, regardless of which type of output format, XML or HTML, is used. A secure search request includes the parameters access=a or access=s.

How Number of Results Returned is Determined

When search results are returned, the number of results is determined by one of the following conditions:

• If Google has results to satisfy the search request, then the requested number of results are returned.
• If Google has fewer results than the number requested in the search request, the last page of results is returned. The last page is determined by dividing the total number of results into pages based on the number of results requested.
• If no results are found, then an empty result set is returned.

To determine if a results page is the last page of available results, check for any of the following conditions:

• The first result number returned does not match the first result number requested.
• The number of results returned is less than the number of results requested.
• The results returned do not contain a link to the next result set.

Navigation

When the total number of results returned is an estimate, the navigation structure for search results is based on this estimate. Google recommends two approaches for generating a navigation scheme for your search results:

1. Only provide the search user with the ability to navigate to the previous results page and the next results page. The output format can be configured to provide links to the previous and next result set when appropriate.
2. Provide the search user with the ability to jump to any search page within the estimated number of results. If the user requests a results page beyond which results are actually available, the last results page is returned. The navigation structure is updated when the last page is displayed. This is the behavior you see in the default output of the Google Search Appliance.

Automatic Filtering

When the automatic filtering feature is active, the number of results returned is significantly reduced. Automatic filtering reduces undesirable results such as duplicate entries. You can disable this feature using the instructions in Automatic Filtering.  

Filtered search results are identified in the returned results. For example, the <FI/> XML tag is present in XML search results where automatic document filtering occurs.   

Google recommends that the search results page displays a message on the last page similar to the following, when automatic filtering occurs:

In order to show you the most relevant results, we have omitted some entries very similar to the search results already displayed. If you like, you can repeat the search with the omitted results included.

This is the behavior you see in the default output format of the Google Search Appliance.

The underlined text in the message should be a hypertext link to submit the same search again with the parameter filter=0. Google finds that this method of informing users about automatic document filtering is effective. This method is used on the Google Internet search site.

If you are using OneBox modules to provide additional query results to your users, note that the results served through a OneBox module are reported separately. The number of OneBox results are not added to the number of standard results.

Back to top

Appendix B:  URL Encoding

Some characters are not safe to use in a URL without first being encoded. Because a Google search request is made by using an HTTP URL, the search request must follow URL conventions, including character encoding, where necessary.

The HTTP URL syntax defines that only alphanumeric characters, the special characters $-_.+!*'(), and the reserved characters ;/?:@=& can be used as values within an HTTP URL request. Since reserved characters are used by the search engine to decode the URL, and some special characters are used to request search features, then all non-alphanumeric characters used as a value to an input parameter must be URL encoded.

To URL-encode a string:

• Replace space characters with a "+" character
• Replace each non-alphanumeric character by its hexadecimal ASCII value, in the format of a "%" character followed by two hexadecimal digits. (Such an ASCII value may be referred to as an escape code.)

Some input parameters require that the values passed to Google search are double-URL-encoded. This requirement means that you must apply the URL encoding to the string twice in succession to generate the final value. See the input parameter descriptions for more information.

Note: For more information about URL encoding, see W3C and IETF web sites.

Examples

Appendix C:  Date Formatting

The search appliance recognizes dates in most reasonable formats. However, dates that only mention the year (YY or YYYY), such as 2008, are not used. For dates in the format month year, the date is assumed to be the first of the month. The search appliance currently recognizes most Latin1 month names, but not Chinese, Japanese, or Korean month names.

Acceptable Date Formats

The following table lists date formats that you can use with the Google Search Appliance.

Date Formatting Notes

1. The YYYYMMDDHHmm pattern for specifying dates is supported, however, the search appliance has no notion of sorting search results based on the difference of time in document dates. For example, if a document has a meta tag with a value of 200910212150 and a second document with a value of 200910210900 then the search appliance discards both dates and sets document dates to their modification time (because the YYYYMMDDHHmm format does not get parsed).
2. Use meta tags with dates in the ISO-8601 format (YYYY-MM-DD) to avoid the confusion caused by multiple dates and multiple formats in the title or text of the documents.
3. The date of each file is returned in the date field of the results. This cannot be turned off, but you can choose not to display it on the front end to your users. To learn more about sorting by date, see Sorting.
4. If no date is found for a file, it is indexed without date data. Results that do not contain date data are displayed at the end of the results with dates, sorted by relevance.
5. If you have documents that contain exceptions to the default dates rule, enter the specific URL or pattern for the file and place these rules at the top of your list. The rules are handled in the order in which they are specified in the rule list. The first rule containing a valid date for the document determines the date of the document.

To specify rules for dates of documents:

1. Click Crawl and Index > Document Dates.
2. In the Host or URL Pattern column, enter the host or pattern to which the rule will apply.
3. Use the drop-down list in the Locate Date In column to select the location of the date for the documents in the specified URL pattern.
4. If you select Meta Tag, specify the name of the meta tag in the Meta Tag Name column.
5. To add more rules, click the Add More Lines button.
6. After all the rules are specified, click the Save Changes button.

Examples of Rules

Because the document http://www.foo.com/example/foo.html matches the URL pattern in rule 1, we first check for the date in the title of the document. The URL doesn't match rule 2, so we check against rule 3. If we are unable to find a valid date in the title or the URL, we look for the date in the meta tag named publication_date according to rule 3. If we are unable to find a valid date in the meta tag, we default to the last modified date of the HTTP server, according to rule 5.

The date from the URL http://www.foo2.com/archives/20040605/abc.html will be extracted.

Since the document http://www.foo.com/foo.html does not match the URL pattern in rule 1, we look for the date in the meta tag, according to rule 3 and default to rule 5 if we cannot find a valid date in rule 3.

For the document http://www.foo2.com/foo.html, we look for the date in the body and default to the last-modified date.

For the document http://www.foo3.com/foo.html, we look for the date only on the last-modified header as it only matches the URL pattern of rule 5.