English | Site Directory

Google Apps APIs

Google Apps Email Migration API Developer's Guide: Protocol

The Google Apps Email Migration API allows administrators and users of Google Apps to migrate mail from legacy email systems into their domain's hosted Gmail accounts. Your client application can upload email messages into these accounts using standard Google data API feeds.

Note: This API is only available to Google Apps Premier, Education, and Partner Edition domains, and cannot be used for migration into Google Apps Standard Edition email or Gmail accounts.

Contents

  1. Audience
  2. Getting Started
  3. Authenticating to the Email Migration service
    1. ClientLogin username/password authentication
  4. Migrating email messages
    1. Sample RFC 822 email message
  5. Usage notes and troubleshooting

Audience

This document is intended for programmers who want to write client applications to allow domain administrators or end users to migrate email into Google Apps mailboxes. It provides some background on the capabilities of the Email Migration API, as well as examples of basic data API interactions using raw XML and HTTP, with explanations. After reading this document, you may wish to learn more about interacting with the API using our client libraries by reading the language-specific sections of this developer's guide.

This document assumes that you understand the general ideas behind the Google data APIs.

For Apps Email Migration API reference information, see the reference guide.

Getting Started

You may want to create a Google Apps account for testing purposes. Email Migration uses Google Apps Accounts, so if you already have a test Google Apps account, you're all set. You can create new test accounts with the Google Apps control panel.

Authenticating to the Email Migration service

To migrate email messages, you must first authenticate using the ClientLogin authentication system. To use this system, you submit the email address and password of the domain user or administrator making the request. The authentication service returns an authentication token that your client can then send to the Email Migration API service along with every subsequent request on behalf of that user.

ClientLogin username/password authentication

To obtain an authentication token, submit an HTTP POST request to the following URL:

https://www.google.com/accounts/ClientLogin

The POST body should contain a set of query parameters, as described in the following table. They should look like parameters passed by an HTML form, using the application/x-www-form-urlencoded content type.

Parameter Description
Email The email address of the domain administrator or user.
Passwd The password for the domain administrator or user.
accountType The string HOSTED, which indicates that this is an authorization for a hosted Google Account.
service The string apps, which is the service name for Google Apps.

If the authentication request fails, the server returns an HTTP 403 Forbidden status code.

If the request succeeds, then the server returns an HTTP 200 OK status code, plus three long alphanumeric codes in the body of the response: SID, LSID, and Auth. The Auth value is the authentication token that you'll send to the Email Migration API with your request, so keep a copy of that value. You can ignore the SID and LSID values.

Since all requests to Email Migration API feeds require authentication, you have to set the Authorization header in the request, using the following format:

Authorization: GoogleLogin auth=yourAuthToken

Where yourAuthToken is the Auth string returned by the authentication request.

Note: Authentication tokens expire after 24 hours. If an authenticated request fails, it may mean you need to acquire another token by posting the credentials to the ClientLogin URL again.

We recommend that you keep the token in memory rather than writing it to a file.

For information on how to handle a CAPTCHA challenge while obtaining an authentication token, see the knowledge base.

Migrating email messages

To migrate mail messages into a hosted Gmail account, start by creating an XML entry for each message you're migrating, as demonstrated in the following example:

<atom:entry xmlns:atom='http://www.w3.org/2005/Atom'>
  <atom:category scheme='http://schemas.google.com/g/2005#kind'
    term='http://schemas.google.com/apps/2006#mailItem'/>
  <apps:rfc822Msg xmlns:apps='http://schemas.google.com/apps/2006'>
    RFC 822 text of the message
  </apps:rfc822Msg>
  <apps:mailItemProperty xmlns:apps='http://schemas.google.com/apps/2006'
    value='IS_STARRED'/>
  <apps:mailItemProperty xmlns:apps='http://schemas.google.com/apps/2006'
    value='IS_UNREAD'/>
  <apps:label xmlns:apps='http://schemas.google.com/apps/2006'
    labelName='Event Invitations'/>
  <apps:label xmlns:apps='http://schemas.google.com/apps/2006'
    labelName='Friends'/>
</atom:entry>

Include the following elements in the Atom entry:

  • An <apps:rfc822Msg> element, containing the XML-escaped RFC 822 content of the email message. (See below for an example.)
  • Optionally, one or more <apps:mailItemProperty> elements, corresponding to properties that the server should apply to the message when the message is inserted into Gmail. The value attribute for each element must be one of the following:
    IS_DRAFT
    The message should be marked as a draft when inserted.
    IS_INBOX
    The message should appear in the Inbox, regardless of its labels. (By default, a migrated mail message will appear in the Inbox only if it has no labels.)
    IS_SENT
    The message should be marked as "Sent Mail" when inserted.
    IS_STARRED
    The message should be starred when inserted.
    IS_TRASH
    The message should be marked as "Trash" when inserted.
    IS_UNREAD
    The message should be marked as unread when inserted. Without this property, a migrated mail message is marked as read.
  • Optionally, one or more <apps:label> elements, corresponding to Gmail labels that should be applied to the message when it is inserted. In the example above, we're specifying that the message should have the "Friends" and "Event Invitations" labels in Gmail.

It's possible to post a single mail item entry at a time, but in most cases when you're migrating mail you'll be migrating multiple entries at once. We therefore recommend that you take advantage of the batch insertion feature of the Email Migration API, which greatly reduces the number of HTTP requests you have to make and thereby lowers your client's bandwidth. For details about creating batch requests, see the reference documentation on batch processing with Google data APIs.

To migrate your mail items in a batch, construct an <atom:feed> containing one or more entries like the one above. To each entry, add a batch ID. For example:

<batch:id>1</batch:id>

Here's an example of a two-item feed:

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
    xmlns:batch="http://schemas.google.com/gdata/batch"
    xmlns:gd="http://schemas.google.com/g/2005">
  <entry>
    <category term="http://schemas.google.com/apps/2006#mailItem"
      scheme="http://schemas.google.com/g/2005#kind" />
    <apps:rfc822Msg xmlns:apps="http://schemas.google.com/apps/2006">
      RFC 822 text of message 0
    </apps:rfc822Msg>
    <apps:mailItemProperty value="IS_STARRED"
      xmlns:apps="http://schemas.google.com/apps/2006" />
    <apps:mailItemProperty value="IS_UNREAD"
      xmlns:apps="http://schemas.google.com/apps/2006" />
    <apps:label labelName="Event Invitations"
      xmlns:apps="http://schemas.google.com/apps/2006" />
    <apps:label labelName="Friends"
      xmlns:apps="http://schemas.google.com/apps/2006" />
    <batch:id>0</batch:id>
  </entry>
  <entry>
    <category scheme="http://schemas.google.com/g/2005#kind"
      term="http://schemas.google.com/apps/2006#mailItem" />
    <apps:rfc822Msg xmlns:apps="http://schemas.google.com/apps/2006">
      RFC 822 text of message 1
    </apps:rfc822Msg>
    <apps:mailItemProperty value="IS_INBOX"
      xmlns:apps="http://schemas.google.com/apps/2006" />
    <apps:label labelName="Important"
      xmlns:apps="http://schemas.google.com/apps/2006" />
    <apps:label labelName="Business"
      xmlns:apps="http://schemas.google.com/apps/2006" />
    <batch:id>1</batch:id>
  </entry>
</feed>

Next, POST the feed to the batch mail item feed URL for your domain:

POST https://apps-apis.google.com/a/feeds/migration/2.0/yourDomain.com/username/mail/batch

where yourDomain.com is your Google Apps domain name, and username is the username that will own the message after the migration. The username is only a username, not a full email address; for example, if someone is migrating messages to be owned by liz@example.com, the username to use is liz. The Content-Type of the POST request must be application/atom+xml or the server will reply with a 415 Unsupported Media Type status code.

There are two different groups who can migrate mail:

  • A domain administrator can migrate mail to any mailbox, by specifying a username to be used in the above URL.
  • An end user can migrate mail only to their own mailbox, and only if an administrator has enabled end-user access to migration. The username in the above URL must be the same as the currently authenticated username.

If your request is successful, the server returns a feed containing the inserted mail item entries. Each entry also contains a <batch:status> element indicating whether the insertion was successful.

The following feed shows what the server might return in response to the two-item feed shown above. We've used placeholders in place of the entry IDs to improve readability, and we've left out the <atom:link> elements that normally appear in each entry.

<?xml version="1.0" encoding="UTF-8"?>
<atom:feed xmlns:atom="http://www.w3.org/2005/Atom"
    xmlns:openSearch="http://a9.com/-/spec/opensearchrss/1.0/"
    xmlns:batch=" http://schemas.google.com/gdata/batch"
    xmlns:apps=" http://schemas.google.com/apps/2006">
  <atom:id>http://apps-apis.google.com/a/feeds/migration/2.0/example.com/liz/mail</atom:id>
  <atom:updated>2007-11-09T01:01:29.238Z</atom:updated>
  <atom:title>Batch Feed</atom:title>
  <atom:link rel="http://schemas.google.com/g/2005#feed"
    type="application/atom+xml"
    href="http://apps-apis.google.com/a/feeds/migration/2.0/example.com/liz/mail"/>
  <atom:link rel="http://schemas.google.com/g/2005#post"
    type="application/atom+xml"
    href="http://apps-apis.google.com/a/feeds/migration/2.0/example.com/liz/mail"/>
  <atom:link rel="http://schemas.google.com/g/2005#batch"
    type="application/atom+xml"
    href="http://apps-apis.google.com/a/feeds/migration/2.0/example.com/liz/mail/batch"/>
  <atom:entry>
    <atom:id>entry0URL</atom:id>
    <atom:category scheme="http://schemas.google.com/g/2005#kind"
      term="http://schemas.google.com/apps/2006#mailItem"/>
    [<atom:link> elements appear here.]
    <batch:id>0</batch:id>
    <batch:status code="201" reason="Created"/>
    <batch:operation type="insert"/>
    <apps:mailItemProperty value="IS_STARRED"/>
    <apps:mailItemProperty value="IS_UNREAD"/>
    <apps:label labelName="Event Invitations"/>
    <apps:label labelName="Friends"/>
  </atom:entry>
  <atom:entry>
    <atom:id>entry1URL</atom:id>
    <atom:category scheme="http://schemas.google.com/g/2005#kind"
      term="http://schemas.google.com/apps/2006#mailItem"/>
    [<atom:link> elements appear here.]
    <batch:id>1</batch:id>
    <batch:status code="201" reason="Created"/>
    <batch:operation type="insert"/>
    <apps:mailItemProperty value="IS_INBOX"/>
    <apps:label labelName="Important"/>
    <apps:label labelName="Business"/>
  </atom:entry>
</atom:feed>

Sample RFC 822 email message

Below is a sample RFC 822 email message that might appear in an <apps:rfc822Msg> element. We recommend that you use a public library such as JavaMail to construct these messages. For more information about this format, including an explanation of how each field is interpreted, refer to the official RFC 822 standard definition.

Received: by 140.23.6.190 with HTTP; Mon, 16 Jul 2007 10:12:26 -0700 (PDT)
Message-ID: &lt;c8acb6980707161012i5d395392p5a6d8d14a8582613@mail.gmail.com&gt;
Date: Mon, 16 Jul 2007 10:12:26 -0700
From: &quot;Elizabeth Bennet&quot; &lt;bennet@example.com&gt;
To: &quot;Fitzwilliam Darcy&quot; &lt;darcy@example.com&gt;
Subject: Lunch on Monday
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Delivered-To: darcy@example.com

This is the body of the email message.  Would you like to have lunch this week?

Note that you need to ensure that the message won't be interpreted as XML. You can do this in several ways:

  • Escape the following characters, using the corresponding entity references: less-than (<), greater-than (>), ampersand (&), apostrophe ('), and double quotation mark ("). The example above uses this approach.
  • Enclose the whole message in a CDATA section.
  • Add the encoding="base64" attribute to the <apps:rfc822Msg> element, and encode the entire message in Base64. For example, the message above would be sent as: 
    <apps:rfc822Msg encoding="base64">
UmVjZWl2ZWQ6IGJ5IDE0MC4yMy42LjE5MCB3aXRoIEhUVFA7IE1vbiwgMTYgSnVsIDIwMDcgMTA6
...
</apps:rfc822Msg>

    This approach is particularly helpful if your message contains text in an encoding other than UTF-8, such as ISO-2022-JP.

Note: Each line in an RFC 822 message should be terminated by a CR+LF (that is, "\r\n") style newline.

Usage notes and troubleshooting

Here are some notes about limits and error situations:

  • You may send a maximum of 32MB of data in each HTTP POST request.
  • The Gmail server may reject malformed messages. In particular, make sure that each message has at least the From:, To:, and Date: header fields required by the RFC822 specification. If a message is rejected as malformed, you'll receive a 400 Bad Request batch status code.
  • If your client exceeds the maximum allowed rate of message uploads per second, the entries that failed contain the batch status code 503 Service Unavailable. If you receive that status code, then record which entries failed (using their batch IDs) and retry the upload. We recommend using an exponential backoff strategy for this process. For example, you might wait thirty seconds and retry the upload; then if your request still returns 503s, wait 60 seconds before trying again, and so on.