• Quick Start Guide
• Introduction
• Running the test gadgets against your container
• Getting an overview of test results
• Diving in on single test failures
• Running specific tests in your container
• Generating an XML report of test results
• Saving a baseline of results for later comparison
• Current Test Suites
• Current Test Cases

Quick Start Guide

The compliance tests can be used to quickly determine if an OpenSocial container is compliant with the spec. To run the compliance tests simply install the appropriate gadget (JavaScript or REST) and load the canvas view.

To publish the results, click the "Generate XML Report" link and save the output to your local machine. Then visit the Compliance matrix and upload the results. You should also add a link to your compliance test results to your container page on the OpenSocial OpenSocial wiki (howto).

Following is more detail on what is tested, how to interpret the results, and most importantly, how you can help make the tests better!

Introduction

The OpenSocial Compliance tests are gadgets that you can install in any OpenSocial container. These gadgets run a series of tests to exercise the container's support for the JavaScript and REST APIs. The gadgets will output whether the container passes or fails compliance for each of over 100 tests. There are three levels of tests, p0, p1, and p2. The p0 tests must pass in order for a container to be spec compliant. The p1 and p2 tests should pass but are more about developer usability issues and are not required for spec compliance. Lower numbered issues are more urgent, for example, a p1 test failure indicates a higher priority issue than p2 test failure. Sets of related tests are grouped into test suites. A list of all current test suites and tests is at the end of this document.

These tests currently cover the OpenSocial and gadgets specs, and are a work in progress. If you wish to contribute to the tests by refactoring, fixing problems with the tests themselves, or adding your own test cases, all patches are very welcome! The tests currently live in the svn tests directory as http://opensocial-resources.googlecode.com/svn/tests/trunk The structure of the gadget and test framework, as well as how to get the source code and add tests, is described more fully in OpenSocialTestFramework. Please go through the !JavaScript style guide to make sure your patches adhere to it.

Running the test gadgets against your container

You can quickly try the latest submitted version of the tests by installing the app with the public SVN hosted URL for the gadget XML file:

Simply add the gadget to your container with this URL. You should disable caching when working with this gadget as it may be changing rapidly. For example, in the orkut sandbox, append &bpc=1 to the page URL when viewing this app.

Getting an overview of test results

The gadgets will take some time to render, as they run over a hundred tests covering many aspects of the OpenSocial spec. Please be patient when first loading the page.

After the gadget first renders, you'll see a message something like the following - the number of passed, failed, warn, and unverified tests will update dynamically as more tests are completed:

Executing...	Compliance Tests Directory	

Passed	Failed	Warnings	Unverified	Total
33	1	2       	9         	45

What do these states mean?

• Pass indicates the test performed exactly as designed.
• Fail indicates the test failed in verification.
• Warning indicates the test behaved as designed, but didn't get the exact message or value for an undefined area. e.g. Unexpected error message on failure
• Unverified indicates the test didn't perform any check. e.g. The test exited because the feature is not supported, or the test is manual.
• Total is the number of tests that should have run.

When the test is complete, you'll see a summary box at top, followed by detail sections for each test suite. In the summary box, you will see output like the following:

Finished	Compliance Tests Directory	
Passed	Failed	Warnings	Unverified	Total
136	13	4       	13        	166
Generate XML report
Priority	Description	Failures
P0 	You are not doing something stated in the spec. You are not spec compliant	4
Activity Test Suite :: ACT100
Activity Test Suite :: ACT102
Activity Test Suite :: ACT300
Opensocial spec 0.7 Definition :: DEFINE
P1 	Important bugs not exactly falling in the P0s. But have negative impact	3
Activity Test Suite :: ACT200
Activity Test Suite :: ACT203
Activity Test Suite :: ACT202
P2 	Functional nice to have breaks. i.e Exceptions for undefined behavior	6
People/Person Requests Test Suite :: PPL801
People/Person Requests Test Suite :: PPL506
MakeRequest Test Suite :: MKRT013
gadgets.io.* TestSuite :: GIO102
gadgets.io.* TestSuite :: GIO103
gadgets.io.* TestSuite :: GIO104

You'll see failures broken down into the three categories of test priority, p0, p1, and p2, with a count and list of failed test names. We'll explain how to get more detail on a particular failed test below. If you see no failures, your container passed all tests. Congratulations!

Diving in on single test failures

We'll use an example to illustrate how to debug a single test failure. In the output above, we saw a P0 test failure: "Activity Test Suite :: ACT100"

This text indicates a failure of a test in the Activity Test Suite. Below the summary box, there are many boxes, one per test suite. One of them is labeled "Activity Test Suite". If there were p0 failures in this suite, the text will be red to indicate failures. It looks something like this:

(Expand)	Activity Test Suite	
Passed	Failed	Warnings	Unverified	Total
11	6	0       	0         	17

To find the failure above, click the Expand button. This will expand the Activity Test Suite box to display all tests in the suite and their status. In our example, we're looking for test ACT100. Scroll down in the list of expanded tests, and you'll find one test box labeled ACT100. Inside this box, click the Expand button to get more detail on the particular test. The expanded box will look something like this:

[ACT100] [P0 ]:: opensocial.requestCreateActivity using title: FAILED [1109 ms]
Stable Test
(Expand)(Run again)
Description> Create activity using title, verify if fetching is supported.
[ACT100.0] dataResponse.hadError(): FAILED: (got 'true'), expected 'undefined'

In this example, the test is creating an activity using a title, then verifying that a fetch of activities returns an activity with that title. The test is failing because calling hadError() on the response returns True unexpectedly.

Running specific tests in your container

Running the entire set of tests suites each time you make a change to your container and deploy a fix can be time consuming. After running the entire set of test suites once, you can select an individual test suite or test to run again to speed this cycle up.

To run one suite of tests, first run all tests once. After the run completes, scroll down to the suite you'd like to run. Click the Expand All button to view details about this suite. Click the Re run suite button to run this suite of tests again, without running the other suites. If there is just one test in the suite, you can run that test as described below.

To run just one test, first run all tests once. After the run completes, expand the test suite containing the test you'd like to run, then scroll down to the test and click the (Run again) button to rerun just this one test.

To run one suite of tests as individual gadget you can select .xml file for that suite and load it separately. i.e. To run people tests you can pick .../suites/0.7/people/people.xml

Generating an XML report of test results

The gadget can generate a report of test results. This can be uploaded to the Compliance Matrix at http://opensocial-compliance.appspot.com, and used for tracking feature coverage. To generate this report, first install the app, then allow all tests to run to completion. Once the run is complete, click on the (Generate XML report) link at the top of the gadget UI. This will pop up a new window with an XML representation of the test suites, tests, and the result of running the tests in this container. The text from this popup window can be saved to a text file and easily translated into other formats. (Currently you must cut-and-paste the XML text, as it is dynamically generated)

Saving a baseline of results for later comparison

The gadget is capable of storing a baseline result from running the tests in a given container, then compare those results with a later run. To save a baseline, first install the app, then allow all tests to run to completion. Once the run is complete, scroll to the bottom of the gadget. You'll find a text area labeled "Results:". This should contain a JSON data representation of the results of the text. Click on the (Save these results as baseline) button. This will use the OpenSocial data store mechanism to save the output of this test to the container, for comparison with later runs. Next time the gadget is viewed, the results should appear in the Baseline Results box for comparison with the current run.

Current Test Suites

Here is the list of test suites that are currently covered by the 0.7 test gadget:

• OpenSocial spec 0.7 Definition

This suite contains a single test to verify the existence of expected datastructures, constants, and APIs in the gadgets, and opensocial APIs. It does not call methods to verify they work, or validate the values of constants. See tests for this suite at compliance/reference.

• Activity Test Suite

This suite tests opensocial APIs to create and request activities. See tests for this suite at activities.

• AppData Test Suite

This suite tests single and batched update and fetch of opensocial PersonAppData. See tests for this suite at appdata.

• Environment Test Suite

This suite tests opensocial.Environment calls. See tests for this suite at environment.

• gadgets.io Test Suite

This suite tests gadgets.io.encodeValues and gadgets.io.getProxyUrl with various inputs. See tests for this suite at gadgets/core.

• gadget.Prefs Test Suite

This suite tests setting and retrieving individual gadget prefs, and container-level prefs. See tests for this suite at gadgets/core.

• gadgets.util Test Suite

This suite tests escape and unescape of strings, getting features, parameters, url params, etc. See tests for this suite at gadgets/core.

• MakeRequest Test Suite

This suite tests making request to fetch content of various mimetypes, both GET and POST, unsigned, and signed. See tests for this suite at makerequest.

• Miscellaneous

This suite tests creating messages, requesting user permission, and collections. See tests for this suite at misc.

• People/Person Requests Test Suite

This suite tests opensocial Person fields and methods. See tests for this suite at people.

Current Test Cases

http://code.google.com/p/opensocial-resources/wiki/ComplianceTestsDirectory