My favorites | Sign in
Project Home Wiki Issues Source
Search
for
PostCommitWebHooks  
How to use Post-Commit Web Hooks for your project.
Updated Mar 14, 2011 by apa...@google.com

Post-Commit Web Hooks


Overview

Post-Commit Web Hooks allow projects to setup web services that receive project commit notifications from Google Code. Such services could be used to integrate external tools including continuous build systems, bug trackers, project metrics, and social networks.

Details

Project owners may enable this feature by specifying a target URL in the Administer/Source tab. If the URL contains the special patterns "%p" and "%r", those will be automatically replaced for each commit with the project name and comma-separated list of revisions, respectively.

The POST request payload describes the commit using the Web Hooks model, and consists of a UTF8-encoded JSON dictionary in the following format:

 {
   "project_name": "atlas-build-tool",
   "repository_path": "http://atlas-build-tool.googlecode.com/svn/",
   "revision_count": 1,
   "revisions": [
     { "revision": 33,
       "url": "http://atlas-build-tool.googlecode.com/svn-history/r33/",
       "author": "mparent61",
       "timestamp":   1229470699,
       "message": "working on easy_install",
       "path_count": 4,
       "added": ["/trunk/atlas_main.py"],
       "modified": ["/trunk/Makefile", "/trunk/constants.py"],
       "removed": ["/trunk/atlas.py"]
     }
   ]
 }
 

While we will make a best effort to promptly deliver all Post-Commit Web Hook notifications, messages are not guaranteed to be delivered, may arrive multiple times, and may not arrive in order of commit. All requests have a 15 second timeout. If we fail to reach the specified URL, we will retry several times over a 24 hour period. This allows your services to be down for short maintenance windows and still receive all messages.

Web services should respond to the POST request with a 2XX response code to indicate successful delivery. Redirects (3XX response codes) are not followed, and no further delivery attempts will be made. All other response codes, as well as request timeouts, are treated as failures and will be retried.

Note: Notifications for commits via the 'svnsync' command are not yet supported.

Notification Format

The payload's JSON dictionary contains the following items:

Field Type Description
project_name String The name of the project.
repository_path String The project's repository URL.
revision_count Number Number of revisions contained in the 'revisions' list.
revisions List A list of dictionaries describing 1 or more repository commits (see table below).

Each revision contained in the 'revisions' list is a dictionary with the following items:

Field Type Description
revision Number(SVN) / String(Hg) Repository identifier for this commit.
urlStringURL to browse repository history for this revision.
author String Username responsible for commit.
timestamp Number Repository commit timestamp.
messageStringCommit log message.
path_countNumberTotal number of paths modified in this revision. Only a fixed number of paths will be included per revision, so this number is used to determine whether a partial list was sent.
addedListA list of String paths added by this revision.
modifiedListA list of String paths modified by this revision.
removedListA list of String paths removed by this revision.

It is important to note that a revision's list of changed paths will be truncated for large commits in order to limit message sizes. Full commit information can be obtained using standard repository tools.

Example: Processing a notification using a Python AppEngine service

 import logging
 from django.utils import simplejson
 from google.appengine import webapp
 class Listener(webapp.!RequestHandler):
   def post(self):
     payload = simplejson.loads(self.request.body)
     for revision in payload["revisions"]:
       logging.info("Project %s, revision %s contains %s paths",
                    payload["project_name"],
                    revision["revision"],
                    revision["path_count"])
 

Authentication

Post-Commit Web Hooks use HMAC-MD5 to authenticate requests. Every project has a unique post-commit 'secret key', visible to project owners in the Administer/Source tab. This key is used to seed the HMAC-MD5 algorithm. Each POST request header contains a HMAC used to authenticate the payload. This value is a 32-character hexadecimal string contained in the 'Google-Code-Project-Hosting-Hook-Hmac' header. By combining your project's secret key and the POST request's HMAC value, you can authenticate the request.

Example: Authentication using a Python AppEngine service

 import hmac
 import logging
 from google.appengine import webapp
 class Listener(webapp.!RequestHandler):
   def post(self):
     project_secret_key = "0123456789abcdef"  # From Administer/Source tab
     m = hmac.new(project_secret_key)
     m.update(self.request.body)
     digest = m.hexdigest()
     if digest == self.request.headers["Google-Code-Project-Hosting-Hook-Hmac"]:
       print "Authenticated"
     else:
       print "Authentication failed!"
 

Continuous Integration with Hudson

Hudson, a continuous integration system, can be used to automate the build for Java projects (among others such as .NET projects). Projects built with Hudson can be triggered using Google Code's Post-Commit Web Hooks.

After Hudson has been setup, builds can be triggered on every commit by using the following url on your Hudson host as the Post-Commit URL:

 http://YOURHOST/hudson/job/PROJECTNAME/build 

For more information, refer to the Hudson documentation.

Comment by derek%ba...@gtempaccount.com, Mar 2, 2009

Can a java authentication example be provided? I'm trying to authenticate hooks received in a servlet.

Comment by derek%ba...@gtempaccount.com, Mar 2, 2009

After a timeout, I managed to roll my own auth implementation for GC's web hook messages. An example and the jar file are available at:

http://webhooks.googlecode.com

Comment by josh.reisner, Mar 4, 2009

In case you're wondering how to catch these web hooks with PHP, I posted my solution here. I just cover catching the POST data, which wasn't readily apparent to me. I haven't covered the authentication yet.

Comment by josh.reisner, Mar 4, 2009

Here's a pretty full script for catching and authenticating these hooks using PHP. I use my own library a little, but it would be easy to extrapolate the key details you'll need.

http://code.google.com/p/joshlib/wiki/CatchingWebHooks

Comment by aleritty, Mar 6, 2009

I realized a php script that reads the web-hook "modified", "added", and "removed" fields and syncronizes your "non-working copy" on the webserver (not running svn!). You can find the script at: http://www.aleritty.net/progetti/post-commitphp/

Comment by Unbearab...@gmail.com, Mar 10, 2009

Thanks for coming up with this! I've implemented an endpoint of this for the CIA.vc commit tracker, so people with google code can usefully have their commits reported. Latency looks pretty good so far, I'm getting around 5 seconds, most of which might be due to our side. Two things: It may be useful to allow people to set several addresses, "chaining" hook endpoints or setting up one that bounces on to all targets is clumsy. Also, it seems the current implementation treats only HTTP 200 as success: My original code returned a 202 (Accepted) since that seemed like the appropriate code, but this seems to have caused the hook system to retry a few times. You may want to change that.

Comment by challgren, Jul 18, 2009

For those that use Brightkite and Google code. I wrote a script that posts the commit to Brightkite http://pastebin.com/f7ac7cc00

Comment by hufeng1...@gmail.com, Jul 23, 2009

You can use php to proccess the PostCommitWebHooks

<?php
//google code secret key
$google_code_secret_key = 'RD5234999kc51l';
//get revision commit data
$revision_data=file_get_contents('php://input');
//build secret verify info;
$secret_verify=hash_hmac("md5",$revision_data,$google_code_secret_key);
//get google secret info
$google_secret_info=$_SERVER['HTTP_GOOGLE_CODE_PROJECT_HOSTING_HOOK_HMAC'];

//prase revision commit data
$revision_info=json_decode($revision_data);
//var_export $revision info
$revision_var_export=var_export($revision_info,true);

$fc="\n google code secret key : ". $google_code_secret_key;
$fc.="\n secret verify info : ". $secret_verify;
$fc.="\n google secret info : ". $google_secret_info;
$fc.="\n project name : ". $revision_info->project_name;
$fc.="\n revision count : ". $revision_info->revision_count;
foreach($revision_info->revisions as $revision){
$fc.="\n author:" .$revision->author;
$fc.="\n time: " .date('Y/m/d H:i:s',$revision->timestamp);
$fc.="\n url: ".$revision->url;
$fc.="\n message: ".$revision->message;
$fc.="\n revision: ".$revision->revision;
}


file_put_contents("log.txt",$fc);
Comment by nicolas....@gmail.com, Sep 3, 2009

For those doing continuous integration with Buildbot, I'm working on a ChangeSource? for Google Code web hooks.

Comment by nicolas....@gmail.com, Jan 21, 2010

It's a good thing that svnsync doesn't trigger web hooks. I just reset my repository and uploaded it back (with some newline corruption from old revisions fixed), and I certainly wouldn't have wanted that to trigger my buildbot for all 1300 revisions.

Comment by alaoui.rda@gmail.com, Jan 31, 2010

Here is a sample php code to send commit notifications via gmail.

You will need to install PEAR for PHP at first:

<?php
require_once "Mail.php";

$data = file_get_contents("php://input"); //raw POST data
$data = json_decode($data, true);


// Commit information
$project_name = $data["project_name"];

foreach($data["revisions"] as $d) {
	$revision = $d["revision"];
	$author = $d["author"];
	$message = $d["message"];
}

$from = '**********';
$to = '*********';
$subject = '['. $project_name .'] r'. $revision .' committed - '. $message;
$body = 'Author: '. $author ."\n";
$body .= 'Comments: '. $message ."\n";
$body .= 'URL: http://code.google.com/p/[project-name]/source/detail?r=' . $revision;

// Stick your GMAIL SMTP info here! ------------------------------
$host = "smtp.gmail.com";
$port = "587";
$username = "**********";
$password = "**********";
// --------------------------------------------------------------

$headers = array ('From' => $from,
  'To' => $to,
  'Subject' => $subject);
$smtp = Mail::factory('smtp',
  array ('host' => $host,
    'port' => $port,
    'auth' => true,
    'username' => $username,
    'password' => $password));

$mail = $smtp->send($to, $headers, $body);
$log = fopen("log.txt","a");
fputs($log, $subject);
fputs($log, "\n");
if (PEAR::isError($mail)) {
  fputs($log, $mail->getMessage());
  fputs($log, "\n");
  fputs($log, "\n");
} 
else {
  fputs($log, "Message successfully sent!");
  fputs($log, "\n");
  fputs($log, "\n");
}
?>
Comment by marc31boss, Feb 18, 2010

@nicolas.alvarez: Do you have any progress for the Buildbot ChangeSource? ?

Comment by conceicao.fernando@gmail.com, Mar 7, 2010

I build a joomla module for this, if you use jommla and want to show the changes of a project, see http://code.google.com/p/svnupdates

Comment by pstav...@gmail.com, May 18, 2010

Buildbot Users:

A buildbot changesource (polling on the Atom feed) is available at http://code.google.com/p/pitaara/

Works with both Hg and SVN repositories. Doesn't understand branches.

Comment by chris.eldredge@gmail.com, Dec 29, 2010

The latest release of Vulcan (a continuous integration java web app) supports these web hooks. See http://code.google.com/p/vulcan/wiki/GoogleCodeCommitHook.

Also, for anyone validating these requests in Java, here's some sample code to do it: SignedRequestAuthorizationFilter.

Comment by als...@gmail.com, Feb 10, 2011

why can't we use HTTPS protocol in our commit hook GET request?

Comment by jafelds@gmail.com, Mar 16, 2011

I would like to know when branches will be recognized in the Post-Commit Web Hook. The branch data is put in our source list, but not the commit hook.

Right now our project at http://code.google.com/p/sm-ssc/ has 7 active branches (and a few more than that that are closed, but that's another issue for another time), and it would be helpful if we didn't have to manually enter branch names on the commit.

Comment by jeremybu...@gmail.com, May 11, 2011
Comment by a...@anse.de, Aug 19, 2011

Can somebody please shed some light on this: I have a post commit which compiles something and takes more than the above timeout of 15 seconds allows. Normally, Google treats this as a non-successful hit, and would retry the request in a specific interval. To overcome these retries, I had the idea to manually send a status 200 quickly, before compiling. Well, the url is still loading in that case, and Google may treat the timeout again as an error. Is that the case?

Comment by kalessin...@gmail.com, Dec 19, 2011

FYI, support for the Google Code Web Hook has just been merged in buildbot: https://github.com/buildbot/buildbot/pull/291#issuecomment-3199354

Comment by ste...@stefansundin.com, Mar 7, 2013

To use PHP to verify the authentication hash, you can do this. It supports multiple projects and can also run svnsync:

<?php
$auth_keys = array(
    "proj1" => "0123456789abcdef",
    "proj2" => "0123456789abcdef",
);

if (!isset($_SERVER["HTTP_GOOGLE_CODE_PROJECT_HOSTING_HOOK_HMAC"])) {
    die("No authentication hash.");
}

$hmac = $_SERVER["HTTP_GOOGLE_CODE_PROJECT_HOSTING_HOOK_HMAC"];
$data = file_get_contents("php://input");
$json = json_decode($data, true);
if (!isset($json["project name"]) || !in_array($json["project name"],array_keys($auth_keys))) {
    die("Invalid project name.");
}
$p = $json["project name"]; // This can also be retrieved from $_GET["p"] if you set up your url as ?p=%p
$digest = hash_hmac("md5", $data, $auth_keys[$p]);
if ($digest != $hmac) {
    die("Invalid authentication hash.");
}

# Run svnsync with the correct user using sudo (remember to edit sudoers!)
system("sudo -u userwithaccesstosyncrep svnsync sync --non-interactive --no-auth-cache --username svnsync file:///home/path/to/svnsync/repo/for/$p");

However, this requires you to make your web server user (often 'www-data') to be able to launch svnsync without sudo having to ask for a password:

$ sudo visudo

Then add this to the bottom (it's important that it is in the bottom!):

www-data ALL=(ALL) NOPASSWD:/usr/bin/svnsync

However, this is really much work for just running svnsync. I suggest that, if that's your only goal, set up a cronjob to run svnsync @daily (or @hourly). It's easier and more resistant to errors. Run with user that has access to your svnsync repo:

$ crontab -e

And add:

@daily svnsync sync --non-interactive --no-auth-cache --username svnsync file:///home/path/to/svnsync/for/proj1
Comment by washie...@gmail.com, Aug 1, 2013

<?php //google code secret key $google_code_secret_key = 'RD5234999kc51l'; //get revision commit data $revision_data=file_get_contents('php://input'); //build secret verify info; $secret_verify=hash_hmac("md5",$revision_data,$google_code_secret_key); //get google secret info $google_secret_info=$SERVER['HTTP_GOOGLE_CODE_PROJECT_HOSTING_HOOK_HMAC'];

//prase revision commit data $revision_info=json_decode($revision_data); //var_export $revision info $revision_var_export=var_export($revision_info,true);

$fc="\n google code secret key : ". $google_code_secret_key; $fc.="\n secret verify info : ". $secret_verify; $fc.="\n google secret info : ". $google_secret_info; $fc.="\n project name : ". $revision_info->project_name; $fc.="\n revision count : ". $revision_info->revision_count; foreach($revision_info->revisions as $revision){ $fc.="\n author:" .$revision->author; $fc.="\n time: " .date('Y/m/d H:i:s',$revision->timestamp); $fc.="\n url: ".$revision->url; $fc.="\n message: ".$revision->message; $fc.="\n revision: ".$revision->revision; }

file_put_contents("log.txt",$fc);

Powered by Google Project Hosting