Using Opensocial test framework...

Introduction

This document describes the inner workings of the Compliance gadget described at ComplianceTests in technical detail. It gives an overview of the class structure and files that implement the testing framework used by the gadget.

The test framework is implemented entirely in xml and javascript. If you'd like to modify a test, or add additional tests or test suites, check the code out from the svn server, and modify the xml and javascript files. There's a list of test suites and their description, as well as individual tests at ComplianceTests. If you have made changes that you think might be valuable to others, please submit them back to svn - we'll all benefit!

Getting the Source Code

You can check out the source code from SVN here:

svn co http://opensocial-resources.googlecode.com/svn/tests/trunk

Test hierarchy

There are three levels in the hierarchy of tests: test, test suite, and test suite directory. Following are descriptions of the three levels of hierarchy, and required attributes for each level.

Test - this is a basic building block of a test suite. Tests have the following attributes:

id: A short, symbolic id for the test
name: The human-readable name of the test
run: The main function of this test
description: A longer, human-readable description of the test
timeout: The number of milliseconds that the run function will have to finish before it has a timeout error. by default is 60000 milliseconds
queue: The testing queue this test belong to, by default queue is 6
tags: List of tags for this test
bugs: List of bugs associated with this test
manual: A flag indicating the test is manual and should not be executed
priority: Specifies a priority for the test
data: Additional user data should be stored here

Test suite - a logical collection of tests. A test suite has the following attributes:

id: A short, symbolic id for the test
name: The human-readable name of the suite
suiteSetup: A method that will run only once before the test suite is run
suiteTearDown: A method that will run only once after the entire suite has run
tests: List of tests that belong to this suite

Test suite directory: Collection of suites to be executed in single gadget

id: A short, symbolic id for the suite directory
name: The human-readable name of the suite directory
suites: List of suites that belong to this directory

Typical test suite

Here's a sample test suite implementation:

function testSuiteName() {
  this.name = 'Test Suite Name';
  this.id = 'TestSuite ID';
  this.tests = [
    { name: 'test name',
      id: 'test id',
      priority: Test.PRIORITY.P0, // {P0 to P4}
      description: 'description of test',
      run: function(context, callback, result) {
        ....
        ....
        result.addSubResult(this.name, Assert.assertEquals,
            Helper.getObjectOptions(this.name), this.name);
        ....
        callback(result);
      }
    },
    // ... more tests here
  ]
};

Each test case has a run method which will be called by the validator that executes the test cases. It takes three parameters. They are propagated here by the testrunner in the common directory at http://opensocial-resources.googlecode.com/svn/tests/trunk/common/

Context: Context in which the gadget is being run. i.e. OWNER and VIEWER information of gadget.
Callback: Callback function called upon completion of test. If you don't call the callback, your result will not be registered with the framework, so make sure to call this upon completion.
Result: Object that stores the outcome of the test. It provides a few methods to save information. Each result object is closely tied with the individual test.

How to add a test suite in the framework

Assuming that you have written a new suite with a couple of tests, you'll need link them with existing framework. This is as easy as editing two text files:

Add references to your suite .js file to three locations in the gadget .xml file

For the main gadget, for example, you'd add the .js file to your local version of the file at: http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/compliance/reference/reference.xml - add
<script src="...">
</script>
tags to the CDATA sections for canvas, profile, and home view as with the existing suites.

Include your suite in controller.js

Edit your local copy of: http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/compliance/reference/controller.js to add an instantiation of your test suite inside the Master.suites array.

Framework

Here's a broad overview of the components of the test framework.

TestRunner

This is the entry point of the tests. Configure the test runner with the test suite directory or test suite (either is acceptable). TestRunner will first setup the tests, then start execution with the help of the validator.

setupTests: This method will see if we are running tests in the right context, by verifying viewer and owner information is provided. It also retrieves the baseline results for these tests from a previous baseline run, if any. Finally, it asks logger to prepare the report format and show it to the user. We show the skeleton of the report at this point, and will fill in the details as each test reports it's result. This makes sure that end user see progress on the fly rather than waiting for all tests to finish. It's particularly helpful if we have over a hundred tests to run. After setup is complte, it's time to run the tests!

executeTests: This method will call validator with the location of baseline results, how to run manual tests, what to do on completion of each test, and when all tests are complete. It will then ask the validator to execute the tests.

Validator

This is where the call to the test you wrote will be made. Calls are asynchronous in nature - we call each test in sequence and ask the test to report on completion by giving a callback. We don't wait for the test to complete synchronously.

runTests: This method will prepare queues of tests to run merging all suites. This is done get parallel execution.

BEFORE 

suites to run
|
|-----------------------------------------------------------------\
|                                   |                             |
suite-1                             suite-2          ...          suite-n
|                                   |                             |
|                                   |                             |
q1---q2-----qn                   q1---q2-----qn                   q1---q2-----qn
|    |      |                    |    |      |                    |     |     |
|    |      t6                   |    |      t12                  |     |     t18
|    t4, t5                      |    t10, t11                    |     t16, t17
t1, t2, t3                       t7, t8, t9                       t13, t14, t15


AFTER

suites to run
|
|
q1---t1, t2, t3 , t7, t8, t9
|
q2---t4, t5, t10, t11, t16, 17 
|
|
qn---t6, t12, t18

Once segregated, validator calls runQueues.

runQueue: This method takes the current queue sequence and picks one test at a time from it. It creates a result object for each test and initializes it with appropriate status, 'running'. It calls the setUp method for the test, if any. After setup, it checks if the test is manual, and if so skip it. It also runs any pre-condition of the test before calling the run method of the test. It passes in a newly created result object, callback and context information to run object. The callback passed in is the captureResult method of the validator.

captureResult: All tests must call captureResult upon completion. It changes the status of the result to 'finished'. Next, it calls the test-finish function in testrunner to display the result of the test. If we have finished all suites, it calls the all-finished function in testrunner. Else, if just the suite is completed, it calls runQueues to pick another suite. Finally, if no suites are completed, it calls runQueue to proceed.

HtmlLogger:

This is a reporting module, which displays test results in html format. HtmlLogger:

Displays html grid in the beginning of test execution
Updates test result, suite summary table, and final summary table for each test upon completion of test execution
Updates test suite headers upon completion of a test suite
Provides JSON output for the results and displays baseline results in JSON format

Assert: This method of HtmlLogger provides a different assert methods as in junit and jsunit

Helper: This provides helper methods used by the framework. Most important are bind and function runner.

► Sign in to add a comment