Google Apps Email Migration API Developer's Guide: Protocol

Last updated April 13, 2011 (see recent changes)

The Google Apps Email Migration API allows administrators and users of Google Apps to migrate mail from legacy email systems into their domain's hosted Gmail accounts. Your client application can upload email messages into these accounts using standard Google Data API feeds.

Note: This API is available to Google Apps for free, Business, Education, and ISPs accounts, and cannot be used for migration into consumer Gmail accounts.

Contents

1. Audience
2. Getting Started
3. API Limits and Quotas
4. Authenticating to the Email Migration service
   1. OAuth for web applications
   2. ClientLogin username/password authentication
   3. Authentication using client libraries
5. Migrating email messages
   1. Sample RFC 822 email message
   2. Migrating messages using client libraries
6. Usage notes and troubleshooting

Audience

This document is intended for programmers who want to write client applications to allow domain administrators or end users to migrate email into Google Apps mailboxes. It provides some background on the capabilities of the Email Migration API. It also provides examples of basic data API interactions using raw XML and HTTP, as well as by using the client libraries.

This document assumes that you understand the general ideas behind the Google Data APIs.

For Apps Email Migration API reference information, see the reference guide.

Getting Started

You may want to create a Google Apps account for testing purposes. Email Migration uses Google Apps accounts, so if you already have a test account, you're all set. You can create new accounts from the Google Apps control panel or using the Provisioning API.

Fully functional samples demonstrating the Email Migration API in action are available in the client libraries. Pick a flavor you're most comfortable using. Follow the instructions below to test them out:

import com.google.gdata.client.appsforyourdomain.migration.*;
import com.google.gdata.data.appsforyourdomain.migration.*;
import com.google.gdata.util.*;

python migration_example.py -d <domain> -e <email> -p <password> -u <username> -f <filename>

python migration_example.py -d 'example.com' -e 'admin@example.com' -p 'p@$$w0rd' -u 'john.smith' -f '/home/administrator/archives/jsmith/messages'

from gdata.apps.migration import service

migrationsample <domain> <userEmail> <password> [destinationEmail]

migrationsample example.com admin@example.com your_password test@example.com

using Google.GData.Apps.Migration;
using Google.GData.Client;
using Google.GData.Extensions.Apps;

Email Migration API Limits and Quotas

Limits and quotas protect the Google infrastructure from an automated process that uses the Email Migration API in an inappropriate way. Excessive requests from an API might result from a harmless typo, or may result from an inefficiently designed system that makes needless API calls. Regardless of the cause, blocking traffic from a specific source once it reaches a certain level is necessary for the overall health of the Google Apps system. It ensures that one developer's actions cannot negatively impact the larger community.

In the unlikely event that your API request fails, you'll receive an HTTP status code response. A status code of 403 has error information about incorrect input and an HTTP status code of 503 has error information indicating which API quotas have been exceeded. These responses allow your custom application to detect these errors and take appropriate action.

If your requests need to be completed in a fixed period of time, send your requests in parallel or use multiple threads in your Java or C# application. An example of parallel requests is requesting small batches of emails from different users rather than adding or removing lots of emails from one user simultaneously. In the case of threads, try starting with 10 threads, one thread per user email. Note, the thread recommendation has trade-offs and is not useful for all API situations. If the number of requests gets too high, quota errors will occur. Another trade-off example is the Email Migration API's quota for the maximum overall message upload rate. The upload rate is one API request - per second - per user, no matter how many threads are making upload requests.

For all errors that are time based (maximum of N things for N seconds per thread), especially the 503 status code errors, we recommend your code catch the exception and, using an exponential backoff algorithm, wait for a small delay before retrying the failed call. An Email Migration API example for one thread is to wait 5 seconds and retry the failed call. If the request is successful, repeat this pattern for the other threads. If the second request is not successful, your application should scale back on the frequency of the request until a call is successful. For example, increase the initial 5 second delay to 10 seconds and retry your failed call again. Also, decide on a retry limit. For example retry a request 5 to 7 times with different delay times before your application returns an error to the user.

Another email usage tip includes you may not be able to send larger attachments to contacts who use other email services with smaller attachment limits.

Authenticating to the Email Migration service

To migrate email messages, you must first authenticate your client. It can authenticate using either of two approaches: ClientLogin username/password or username/access code authentication, or OAuth for web applications.

If your client is a standalone single-user "installed" client (such as a desktop application), then you should use the ClientLogin Interface system; if your client is a multi-user web application client, then you should use 3-legged OAuth. Both of these methods involve interacting with an authentication service. The authentication service returns an authentication token that your client can then send to the Email Migration API service along with every subsequent request on behalf of that user.

For more information about authentication with Google Data APIs in general, see the authentication documentation.

The samples in this document assume you are providing the appropriate authentication.

Note: If you're using a client library, authentication will be managed for you by the library itself. You may skip directly to the section below to know how to do this.

OAuth for web applications

Email Migration API supports 3-legged OAuth scheme for allowing web applications to migrate messages. The OAuth guide provides information on how to obtain access tokens. For more information, refer to the complete documentation on using OAuth with the Google Data APIs. If you're working with client libraries, please follow these instructions.

OAuth scope parameter

https://apps-apis.google.com/a/feeds/migration/

ClientLogin username/password authentication

To use this system, you submit the email address, a password or the two-step verification access code of the domain user or administrator making the request. The authentication service returns an authentication token that your client can then send to the Email Migration API service along with every subsequent request on behalf of that user.

To obtain an authentication token, submit an HTTP POST request to the following URL:

https://www.google.com/accounts/ClientLogin

The POST body should contain a set of query parameters, as described in the following table. They should look like parameters passed by an HTML form, using the application/x-www-form-urlencoded content type.

If the authentication request fails, the server returns an HTTP 403 Forbidden status code.

If the request succeeds, then the server returns an HTTP 200 OK status code, plus three long alphanumeric codes in the body of the response: SID, LSID, and Auth. The Auth value is the authentication token that you'll send to the Email Migration API with your request, so keep a copy of that value. You can ignore the SID and LSID values.

Since all requests to Email Migration API feeds require authentication, you have to set the Authorization header in the request, using the following format:

Authorization: GoogleLogin auth=yourAuthToken

Where yourAuthToken is the Auth string returned by the authentication request.

Note: Authentication tokens expire after 24 hours. If an authenticated request fails, it may mean you need to acquire another token by posting the credentials to the ClientLogin URL again.

We recommend that you keep the token in memory rather than writing it to a file.

For information on how to handle a CAPTCHA challenge while obtaining an authentication token, see the following FAQ.

Authentication using client libraries

MailItemService mailItemService = new MailItemService("exampleCo-exampleApp-1");
mailItemService.setUserCredentials(userEmail, password);

migration_service = service.MigrationService(email=email, password=password,
                                             domain=domain, source=app_name)
migration_service.ProgrammaticLogin()

MailItemService mailItemService = new MailItemService(domain, "Sample Migration Application");
mailItemService.setUserCredentials(userEmail, password or two-step verification access code);

Migrating email messages

To migrate mail messages into a hosted Gmail account, issue multipart POST requests, where the first part contains an Atom entry specifying the message properties and labels, and the second part contains the complete RFC 822 message as UTF-8 text. Alternatively, you can use the client libraries to simplify the email migration.

Note: Base64 encoded RFC 822 message capability is now deprecated. The RFC 822 message should be UTF-8 text.

HTTP Header:

Content-Type: multipart/related;boundary="----=_Part_0_25934938.1266495790627"

HTTP Body:

------=_Part_0_25934938.1266495790627
Content-Type: application/atom+xml

<?xml version='1.0' encoding='UTF-8'?><entry xmlns='http://www.w3.org/2005/Atom'
 xmlns:apps='http://schemas.google.com/apps/2006'>
<category scheme='http://schemas.google.com/g/2005#kind' term='http://schemas.google.com/apps/2006#mailItem'/>
<atom:content xmlns:atom='http://www.w3.org/2005/Atom' type='message/rfc822'/>
<apps:mailItemProperty value='IS_INBOX'/>
</entry>

------=_Part_0_25934938.1266495790627
Content-Type: message/rfc822

MIME-Version: 1.0
Received: by 10.216.52.85 with HTTP; Thu, 18 Feb 2010 04:16:09 -0800 (PST)
Date: Thu, 18 Feb 2010 17:46:09 +0530
Delivered-To: audit1@yourdomain.com
Message-ID: <5db9e26e1002180416n9436618p2e828b22753764f9@mail.gmail.com>
Subject: Hello Multipart HTTP
From: anirudh dewani <audit1@yourdomain.com>
To: audit user <audit2@yourdomain.com>
Content-Type: multipart/alternative; boundary=0016e64c1c48a28a49047fdeee47

--0016e64c1c48a28a49047fdeee47
Content-Type: text/plain; charset=ISO-8859-1

This is a test mail using multipart HTTP requests for Email Migration API.

Thanks.

--0016e64c1c48a28a49047fdeee47
Content-Type: text/html; charset=ISO-8859-1

This is a test mail using multipart HTTP requests for Email Migration API.<div><br></div><div>Thanks.</div>

--0016e64c1c48a28a49047fdeee47--
------=_Part_0_25934938.1266495790627--

Include the following elements in the Atom entry:

• Optionally, one or more <apps:mailItemProperty> elements, corresponding to properties that the server should apply to the message when the message is inserted into Gmail. The value attribute for each element must be one of the following:

  IS_DRAFT

  The message should be marked as a draft when inserted.

  IS_INBOX

  The message should appear in the Inbox, regardless of its labels. (By default, a migrated mail message will appear in the Inbox only if it has no labels.)

  IS_SENT

  The message should be marked as "Sent Mail" when inserted.

  IS_STARRED

  The message should be starred when inserted.

  IS_TRASH

  The message should be marked as "Trash" when inserted.

  IS_UNREAD

  The message should be marked as unread when inserted. Without this property, a migrated mail message is marked as read.

• Optionally, one or more <apps:label> elements, corresponding to Gmail labels that should be applied to the message when it is inserted. In the example above, we're specifying that the message should have the "Friends" and "Event Invitations" labels in Gmail.

Note: Batch insertion mode has been deprecated. We recommend that you use multi-threaded clients (one thread per user) with multipart HTTP requests to improve the speed of the process.

There are two different groups who can migrate mail:

• A domain administrator can migrate mail to any mailbox, by specifying a username to be used in the above URL.
• An end user can migrate mail only to their own mailbox, and only if an administrator has enabled end-user access to migration. The username in the above URL must be the same as the currently authenticated username.

If you are an end user migrating mail to your own mailbox, you can also use the "default feed" for convenience:

POST https://apps-apis.google.com/a/feeds/migration/2.0/default/mail

If your request is successful, the server returns a feed containing the inserted mail item entry.

Sample RFC 822 email message

Below is a sample RFC 822 email message. We recommend that you use a public library such as JavaMail to construct these messages. For more information about this format, including an explanation of how each field is interpreted, refer to the official RFC 822 standard definition.

MIME-Version: 1.0
Received: by 10.216.170.147 with HTTP; Fri, 26 Mar 2010 12:54:28 -0700 (PDT)
Date: Sat, 27 Mar 2010 03:54:28 +0800
Delivered-To: darcy@example.com
Message-ID: <203b3a4f1043461254v14855b71w9ce59e3abcf45962@mail.gmail.com>
Subject: hello multipart
From: Elizabeth Bennet <bennet@example.com>
To: Fitzwilliam Darcy <darcy@example.com>
Content-Type: multipart/alternative; boundary=000e0cdf7740f8017e0482b9877a

--000e0cdf7740f8017e0482b9877a
Content-Type: text/plain; charset=ISO-8859-1

howdy multipart!

--000e0cdf7740f8017e0482b9877a
Content-Type: text/html; charset=ISO-8859-1

howdy multipart!

--000e0cdf7740f8017e0482b9877a--

Note: Each line in an RFC 822 message should be terminated by a CR+LF (that is, "\r\n") style newline.

Migrating messages using client libraries

MailItemEntry entry = new MailItemEntry();
Rfc822MediaSource mediaSource = new Rfc822MediaSource(rfc822text);
entry.setMediaSource(mediaSource);

entry.addMailProperty(MailItemProperty.STARRED);
entry.addMailProperty(MailItemProperty.UNREAD);

entry.addLabel(new Label("Friends"));
entry.addLabel(new Label("Event Invitations"));

mail_item_properties = ['IS_INBOX', 'IS_STARRED']
mail_labels = ['Imported']
migration_service.ImportMail(user_name=username,
                             mail_message=content,
                             mail_item_properties=mail_item_properties,
                             mail_labels=mail_labels)

for message in messages:
  migration_service.AddMailEntry(mail_message=message,
                                 mail_item_properties=mail_item_properties,
                                 mail_labels=mail_labels)
migration_service.ImportMultipleMails(user_name=username)

MailItemEntry entry = new MailItemEntry();

entry.Rfc822Msg = new Rfc822MsgElement(rfcTextOfMessage);

entry.MailItemProperties.Add(MailItemPropertyElement.STARRED);
entry.MailItemProperties.Add(MailItemPropertyElement.UNREAD);

entry.Labels.Add(new LabelElement("Friends"));
entry.Labels.Add(new LabelElement("Event Invitations"));

for (int i = 0; i < entries.Length; i++)
{
  entries[i].BatchData = new GDataBatchEntryData();
  entries[i].BatchData.Id = i.ToString();
}

MailItemFeed feed = mailItemService.Batch(domain, username, entries);

foreach (AtomEntry entry in batchResult.Entries)
{
    GDataBatchEntryData batchData = entry.BatchData;

    Console.WriteLine("Mail message {0}: {1} {2}",
        batchData.Id, batchData.Status.Code, batchData.Status.Reason);
}

new Rfc822MsgElement(base64EncodedMessage, Rfc822MsgElement.EncodingMethod.BASE64)

Usage notes and troubleshooting

Here are some notes about limits and error situations:

• You may send a maximum of 32MB of data in each HTTP POST request.
• The Gmail server may reject malformed messages. In particular, make sure that each message has at least the From:, To:, and Date: header fields required by the RFC822 specification. If a message is rejected as malformed, you'll receive a 400 Bad Request HTTP return code.
• If your client exceeds the maximum allowed rate of message uploads per second, you will recieve 503 Service Unavailable. If you receive that status code, then retry uploading the failed entry. We recommend using an exponential backoff strategy for this process. For example, you might wait thirty seconds and retry the upload; then if your request still returns 503s, wait 60 seconds before trying again, and so on.