Debugging Google Data API Clients: Exploring Traffic from Within your Program

Jeffrey Scudder, Google Data APIs Team
June 2007

Introduction

Sometimes there's no substitute for seeing what goes over the wire. This is especially true when writing software which uses web services like the Google Data APIs, where lots of operations involve making HTTP requests. When all else fails, you can verify that your program is doing what you'd expect by seeing the actual transmitted and received bytes. Many of the client libraries for the Google Data APIs have a debugging mode which displays the HTTP traffic. This is especially useful when you you don't have access to a packet sniffer like WireShark or Fiddler.

I can't count the number of times that I could have sworn my program was correct, only to find upon inspecting a packet trace that there was an extra newline character, or a misnamed HTTP header. Programming against a web service without looking at the HTTP traffic can be like trying to thread a needle with your eyes glued shut.

However, you may find yourself in a situation where a packet sniffer is unavailable or is inadequate to deal with encrypted packets. Never fear-you can get around this limitation by leveraging some in-program logging mechanisms. By utilizing these logging facilities, you can see some, if not all, of the exchanged data, even for encrypted HTTPS data or remote running code.

For this article, I've written sample diagnostic code in 3 languages using the Google Data API client libraries for Java, .NET, and Python. In each example, I turn on logging or debugging, authenticate using client login, and then get a list of my Google Spreadsheets and print out their titles.

Java

You can use the java.util.logging classes to set the logging levels (and consequently expose traffic data) for a couple of key objects in the client library. In the example below, I chose to look at the HTTP headers and the activities of the XML parser to get a complete view of what is traveling over the wire.

The Google Data Java client library has separate classes to handle HTTP requests and XML parsing; thus, I need to create two Logger objects, one for each class: com.google.gdata.client.http.HttpGDataRequest handles the HTTP traffic while com.google.gdata.util.XmlParser is responsible for XML parsing.

The logger instances will record activities for HttpGDataRequest and XmlParser, and you can control the level of detail of each logger's output. For this demonstration, I've chosen to view all of the events produced by the HttpGDataRequest and XmlParser objects.

Once I've created and configured my Loggers, I need to tell them what to do when they receive an event from their classes. For now, I want to write all logging information out to the console, so I create a ConsoleHandler and add it to both of my Loggers.

Here's my sample code:

import com.google.gdata.client.spreadsheet.*;
import com.google.gdata.data.spreadsheet.*;
import com.google.gdata.util.*;
import java.io.*;
import java.net.URL;
import java.util.*;
import java.util.logging.*;

public class PrintSpreadsheetsWithLogging {
   
   
public static void main(String [] args) throws AuthenticationException,
                                                   
ServiceException, IOException {
       
// Configure the logging mechanisms.
       
Logger httpLogger = Logger.getLogger("com.google.gdata.client.http.HttpGDataRequest");
        httpLogger
.setLevel(Level.ALL);
       
Logger xmlLogger = Logger.getLogger("com.google.gdata.util.XmlParser");
        xmlLogger
.setLevel(Level.ALL);
       
// Create a log handler which prints all log events to the console.
       
ConsoleHandler logHandler = new ConsoleHandler();
        logHandler
.setLevel(Level.ALL);
        httpLogger
.addHandler(logHandler);
        xmlLogger
.addHandler (logHandler);
       
       
SpreadsheetService service = new SpreadsheetService("testing-loggingExampleApp-1");
        service
.setUserCredentials(email, password);
     
       
// Get a list of your spreadsheets.
        URL metafeedUrl
= new URL("http://spreadsheets.google.com/feeds/spreadsheets/private/full ");
       
SpreadsheetFeed feed = service.getFeed(metafeedUrl, SpreadsheetFeed.class);
     
       
// Print the title of each spreadsheet.
       
List spreadsheets = feed.getEntries();
       
for (int i = 0; i < spreadsheets.size(); i++) {
         
SpreadsheetEntry entry = (SpreadsheetEntry)spreadsheets.get(i);
         
System.out.println("\t" + entry.getTitle().getPlainText());
       
}
   
}
}

When you run this program, you'll see something like this on the console (I cut out some of the less interesting parts):

Jun 7, 2007 10:24:50 AM ...HttpGDataRequest setPrivateHeader
FINER: Authorization: <Not Logged>
Jun 7, 2007 10:24:50 AM ...HttpGDataRequest setHeader
FINER: User-Agent: ...
...
Jun 7, 2007 10:25:20 AM ...HttpGDataRequest execute
FINE: 200 OK
Jun 7, 2007 10:25:20 AM ...HttpGDataRequest execute
FINER: Date: Thu, 07 Jun 2007 17:25:24 GMT
Jun 7, 2007 10:25:20 AM ...HttpGDataRequest execute
FINER: null: HTTP/1.1 200 OK
Jun 7, 2007 10:25:20 AM ...HttpGDataRequest execute
FINER: Content-Type: application/atom+xml; charset=UTF-8
Jun 7, 2007 10:25:20 AM ...HttpGDataRequest execute
FINER: Last-Modified: Thu, 07 Jun 2007 17:25:22 GMT
...
Jun 7, 2007 10:25:20 AM ...XmlParser startElement
FINE: Start element id
Jun 7, 2007 10:25:20 AM ...XmlParser endElement
FINE: End element id
...
Jun 7, 2007 10:25:20 AM ...XmlParser startElement
FINE: Start element title
Jun 7, 2007 10:25:20 AM ...XmlParser startElement
FINER: Attribute type='text'
Jun 7, 2007 10:25:20 AM ...XmlParser endElement
FINE: End element title
...
Jun 7, 2007 10:25:20 AM ...XmlParser endElement
FINE: End element entry
...
Jun 7, 2007 10:25:20 AM ...XmlParser endElement
FINE: End element feed

These logs can get quite large, so you might want to be more selective in setting the Loggers' levels. You could also create a FileHandler instead of a ConsoleHandler to allow you to store the log data for later use.

Of course, if Java isn't your bag, you could try .NET.

.NET

To capture the HTTP traffic in the .NET client library, you can replace the default request factory in the client with a GDataLoggingRequestFactory.

The HTTP requests in the .NET library are created by the GDataRequestFactory which is inside each Service object. The normal request factories don't perform any logging but the GDataLoggingRequestFactory, which is a subclass of the GDataRequestFactory, has logging built in. You can specify the full path of the log file by setting CombinedFileName.

After setting up your request factory, you need to replace the request factory in your Service object by setting the RequestFactory of the service object. Your code might look something like this:

using System;
using Google.GData.Client;
using Google.GData.Extensions;
using Google.GData.Spreadsheets;

namespace LogginTest
{
   
class Program
   
{
       
static void Main(string[] args)
       
{
           
SpreadsheetsService service = new SpreadsheetsService("-exampleApp-1");
            service
.setUserCredentials(email, password);

           
Google.GData.Client.GDataLoggingRequestFactory factory = new GDataLoggingRequestFactory("wise", "SpreadsheetsLoggingTest");
            factory
.MethodOverride = true;
            factory
.CombinedLogFileName = "c:\\temp\\xmllog.log";
           
Console.WriteLine("Log file name:" + factory.CombinedLogFileName);
           
            service
.RequestFactory = factory;

           
SpreadsheetQuery query = new SpreadsheetQuery();
           
SpreadsheetFeed feed = service.Query(query);

           
Console.WriteLine("Your spreadsheets:");
           
foreach (SpreadsheetEntry entry in feed.Entries)
           
{
               
Console.WriteLine(entry.Title.Text);
           
}

           
Console.ReadKey();
       
}
   
}
}

The resulting log file contains the XML requests and responses. Here's an abbreviated example which I've formatted using tidy.

<?xml version='1.0' encoding='utf-8'?>

<feed xmlns='http://www.w3.org/2005/Atom'
xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/'>
  <id>
  http://spreadsheets.google.com/feeds/spreadsheets/private/full</id>
  <updated>2007-06-07T22:05: 02.674Z</updated>
  <link rel='self' type='application/atom+xml'
  href='http://spreadsheets.google.com/feeds/spreadsheets/private/full'>

  </link>
  ...
  <entry>
    <updated>2007-03-28T17:28:57.250Z</updated>
    <category scheme=' http://schemas.google.com/spreadsheets/2006'
    term='http://schemas.google.com/spreadsheets/2006#spreadsheet'>
    <title type='text'>events</title>

    <content type='text'>events</content>
    ...
  </entry>
  <entry>
    <updated>2007-05-25T22:11:08.200Z</updated>

    <category scheme=' http://schemas.google.com/spreadsheets/2006'
    term='http://schemas.google.com/spreadsheets/2006#spreadsheet'>
    </category>
    <title type='text'>UnitTest</title>
    <content type='text'>UnitTest</content>
    ...
  </entry>

  ...
</feed>

But perhaps you are really into scripting languages, and you prefer using Python.

Python

To capture the HTTP traffic in the Python client library, you can echo the HTTP header traffic to the console by turning on debug mode in the HTTP client. The service object has a debug member which you can set to True.

Setting debug to true will set the debug flag in the underlying HTTPRequest object which is contained in the service object.

Here's an example which will echo the HTTP headers sent from the spreadsheets server when you ask for a list of your spreadsheets.

#!/usr/bin/python

import gdata.spreadsheet.service

client
= gdata.spreadsheet.service.SpreadsheetsService()
client
.debug = True

client
.ClientLogin(email, password)

feed
= client.GetSpreadsheetsFeed()

for entry in feed.entry:
 
print entry.title.text

And you will see something like this on your console:

reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/atom+xml; charset=UTF-8
header: Last-Modified: Thu, 07 Jun 2007 18:22:35 GMT
header: Cache-Control: max-age=0, must-revalidate, private
header: Transfer-Encoding: chunked
...
header: Date: Thu, 07 Jun 2007 18:22:35 GMT
header: Server: GFE/1.3

As you perform additional operations, such as an insert or update, you'll see corresponding request data echoed to your console.

Conclusion

This brief tutorial has illustrated how you can add basic logging functionality into a Java, .NET, or Python program which uses the Google Data API client libraries. These techniques can be useful if you need to debug HTTP exchanges, but don't have access to a packet sniffer. I've only scratched the surface with these examples. Many of the logging mechanisms present in these languages are much more powerful than what is shown here. If you'd like more information on logging or the Google Data APIs, check out the list of resources below.

Client libraries covered in this article can be found on these pages:

Related knowledge base items:

Discussion groups: We have quite a few, with more coming as more Google Data APIs are rolled out. We actively monitor the groups.

If you have questions or suggestions, I'd enjoy hearing from you. Hop on the discussion group and start posting.