1.0 Introduction

A particularly useful abstraction in Hackystat is "Daily Project Data" (DPD), which aggregates together the raw data from all project members over the course of one day to create a useful abstraction. For example, it might be useful to know the total "DevTime" associated with each Project member for a particular Project for a particular day, or the total number of UnitTest invocations for a particular Project for a particular day. Example applications of DPD abstractions include:

• A "snapshot" of the Project status for a single day;
• A "To Date" summary of the Project (by aggregating DPD data over the Project lifetime)
• Software Telemetry, which shows trends at the daily, weekly, or monthly level using DPD data.

This DailyProjectData service makes this abstraction available to such higher level applications. The next section provides a summary table of the REST API, followed by details on each resource.

The design of this service is influenced by the DailyProjectData abstractions we created in Hackystat Version 7. It may be helpful to review the Hackystat Version 7 Telemetry Reduction Function Reference Guide Chapter, which shows one form of higher level usage of this information.

Each DailyProjectData service generates its abstractions by utilizing sensor data from exactly one SensorBase. DPDs cannot represent an abstraction of data from multiple SensorBases.

Although each DPD service must be connected to a single SensorBase, a Hackystat system can support multiple DPD service instances. The DPD service described in this document provides abstractions related to Build, Commit, Coupling, DevEvent, and other kinds of process and product data. It does not, however, provide abstractions related to Test Driven Design, for example. While this service could be extended to provide such DPD abstractions, an acceptable alternative is to create a second Hackystat service for TDD DPDs which would have its own REST API providing daily project data abstractions geared toward TDD. In this case, a higher level service, such as Telemetry, would access the DPD service described here for Build data and a different one for TDD data.

Finally, DPDs are intended to be abstractions of sensor data, and thus do not attempt to provide detail sufficient to support every conceivable high level analysis. Certain kinds of specialized analyses may need to retrieve the raw data directly from the SensorBase. The goal is that DPDs provide a useful layer of abstraction that supports common queries about Project process and product values and trends.

2.0 DPD REST API Summary

The following table summarizes the API calls used to retrieve DailyProjectData resource representations. For readability, the resource type associated with the operation is boldfaced.

In the following table:

• {host} typically resolves to something like "http://foo.com/hackystat/dailyprojectdata".
• {user} is the owner of the {project},
• {timestamp} indicates the start of the 24 hours representing the day.

"Abstractions" summarizes of the kinds of information the XML payload returned by the request will provide.

Except for the ping resource, which does not require authentication, all other requests require HTTP Basic authentication, using credentials stored in the underlying SensorBase.

Let's now turn to the details of each of these calls.

3.0 DPD REST API Details

This section provides details on the invocation of each Daily Project Data abstraction and the resulting XML resource representation.

3.1 Build

Each Build DPD provides, for each member of the project and for the requested build type, the number of passing and failing build invocations, and the total elapsed time spent building.

"Type" is a parameter that enables analyses to differentiate between "local" builds invoked by individual developers vs. "daily" or "continuous" builds. While developer builds might fail frequently, it is often a goal to have very few daily or continuous integration build failures. Thus, it is very useful to be able to differentiate between these different types of build events. ("Type" was called "configuration" in Version 7 DPD.) Build sensors are responsible for providing a property called "Type" with an associated string value in order to allow this form of build event filtering to occur during the creation of this abstraction.

If the "Type" parameter is not supplied, then all build data is returned.

Here is an example invocation:

An example return value is:

<?xml version="1.0"?>
<BuildDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" Type="daily">
 <MemberData MemberUri="http://dasha.ics.hawaii.edu:9877/sensorbase/users/johnson@hawaii.edu" success="5" failure="3"/>
</BuildDailyProjectData>

3.2 Code Issue

The CodeIssue DPD provides the total number of issues of a given type generated by a given tool.

The Tool parameter generally specifies a static analysis tool such as Lint, Checkstyle, FindBugs, PMD, etc.

Type values are tool-specific. For example, a sensor for Checkstyle might generate "Type" properties corresponding to the Checkstyle check types, such as "JavaDoc", "Naming", "Headers", "Imports", etc.

FindBugs has types such as "Bad practice", "Correctness", "Internationalization", "Malicious code vulnerability", etc. PMD has ruleset categories such as "Basic", "Braces", "Code Size", etc.

Both the "Tool" and "Type" parameters are optional, and either or both may be specified.

If CodeIssue data for a Project is generated multiple times during the day by a given Tool, then the runtime stamp is used to determine the latest "snapshot" of Sensor Data to use for a given tool.

Here is an example invocation:

An example return value might be:

<?xml version="1.0"?>
<CodeIssueDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" >
  <CodeIssueData Tool="Checkstyle" IssueType="JavaDoc" NumIssues="4"/>
  <CodeIssueData Tool="Checkstyle" IssueType="Indentation" NumIssues="6"/>
</CodeIssueDailyProjectData>

If you don't specify either the "Tool" or "Type" parameters, then you will get back a separate CodeIssueData element for each tool and each type associated with that tool. For example:

might return:

<?xml version="1.0"?>
<CodeIssueDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" >
  <CodeIssueData Tool="Checkstyle" IssueType="JavaDoc" NumIssues="4"/>
  <CodeIssueData Tool="Checkstyle" IssueType="Indentation" NumIssues="6"/>
  <CodeIssueData Tool="FindBugs" IssueType="NullPointerDereference" NumIssues="8"/>
</CodeIssueDailyProjectData>

Finally, if there is CodeIssue data for a day for a given tool, but no issues were found, then you will get back a representation indicating that zero issues were present in the sensor data:

<CodeIssueDailyProjectData Owner="johnson@hawaii.edu" Project="Default" StartTime="2008-02-08T00:00:00.000-10:00" >
  <CodeIssueData NumIssues="0" Tool="PMD"/>
  <CodeIssueData NumIssues="0" Tool="FindBugs"/>
  <CodeIssueData NumIssues="0" Tool="Checkstyle"/>
</CodeIssueDailyProjectData>

3.3 Commit

Each Commit DPD provides, for each member of the project, the total number of commits as well as the total number of lines of code added, deleted, and changed.

Here is an example invocation:

An example return value is:

<?xml version="1.0"?>
<CommitDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" >
 <MemberData MemberUri="http://dasha.ics.hawaii.edu:9877/sensorbase/users/johnson@hawaii.edu" Commits="5" LinesAdded="3" LinesDeleted="0" LinesChanged="5"/>
</CommitDailyProjectData>

Note that the "LinesChanged" attribute is optional and may not be present in all data since not all version control systems provide this value.

3.4 Complexity

Each Complexity DPD provides a measure indicating the complexity (or complexities) of each file associated with a Project for a given day. Currently, cyclomatic complexity, as generated by tools such as JavaNCSS is supported.

The Complexity DPD uses the "snapshot" approach to determining relevant sensor data. The snapshot approach avoids the problem of double counting and/or deleted files when a sensor is invoked more than once a day. Thus, tools that generate Complexity data must use the Runtime field to indicate that a given set of sensor data was all generated at a single point in time and should be associated together.

Unlike the FileMetric DPD, which assumes that a single value can represent the "size" of each file, the Complexity DPD, while based upon FileMetric sensor data, does not assume that complexity for a single file can be adequately represented as a single number. For example, cyclomatic complexity for a file is represented as a list of integers, where each integer represents the cyclomatic complexity associated with a single method.

Here is an example invocation:

An example return value is:

<?xml version="1.0"?>
<ComplexityDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" Type="Cyclometric" Tool="JavaNCSS" >
 <FileData FileUri="file://users/johnson/Foo.java" ComplexityValues="1,3"/>
 <FileData FileUri="file://users/johnson/Bar.java" ComplexityValues="1"/>
 <FileData FileUri="file://users/johnson/Baz.java" ComplexityValues="1,4,6"/>
</ComplexityDailyProjectData>

The Complexity URI also supports the following optional parameters:

3.5 Coupling

Each Coupling DPD provides the number of afferent (incoming) and efferent (outgoing) couplings. The "Type" specifies the granularity of system objects used for the dependency analysis, which is typically a value like "Method", "Class", or "Package".

Type is required and is thus part of the URI.

An optional Tool parameter allows you to specify the Tool (JDepend, DependencyFinder, etc.) whose coupling data you wish to retrieve. If you do not specify the Tool parameter, then the latest snapshot of coupling data will be returned.

Here is an example invocation:

An example return value is:

<?xml version="1.0"?>
<CouplingDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" Type="Class" Tool="DependencyFinder">
 <ConstructData Name="Foo" Afferent="2" Efferent="3"/>
 <ConstructData Name="Bar" Afferent="3" Efferent="6"/>
 <ConstructData Name="Baz" Afferent="2" Efferent="0"/>
</CouplingDailyProjectData>

The return value also indicates the tool (in this case, DependencyFinder) used to generate the data.

3.6 Coverage

Each Coverage DPD provides the number of covered and uncovered constructs given the coverage "granularity" for the Project.

Coverage "granularity" is generally a value like "Class", "Method", "Line", etc. Different coverage tools might generate different kinds of Coverage granularities.

The Coverage granularity must be specified in order to provide a meaningful DPD abstraction, which is why it is required as part of the URI.

If Coverage data for a Project is generated multiple times during the day, only the last instance of the Coverage data is used in the generation of this abstraction.

It is possible to use multiple Tools to generate different types of Coverage data. However, if two tools both generate Coverage data with the same granularity, then the data generated by the tool that ran last during the day is the one that is used in this abstraction.

Here is an example invocation:

An example return value is:

<?xml version="1.0"?>
<CoverageDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" Granularity="Line" Tool="Emma">
 <ConstructData Name="Foo" NumCovered="47" NumUncovered="63"/>
 <ConstructData Name="Bar" NumCovered="67" NumUncovered="22"/>
 <ConstructData Name="Baz" NumCovered="69" NumUncovered="0"/>
</CoverageDailyProjectData>

The return value also indicates the tool (in this case, Emma) used to generate the data.

3.7 Dev Time

Each DevTime DPD provides, for each member of the Project, the amount of DevTime (in minutes) they accumulated for the given project on the given day.

"DevTime" is computed by processing the the DevEvent sensor data entries associated with this Project and Interval. To compute DevTime from DevEvents, each day is partitioned into 288 five minute intervals. If there is at least one DevEvent generated by a user with a timestamp corresponding to the five minute interval, then those five minutes represent "DevTime".

Here is an example invocation:

An example return value is:

<?xml version="1.0"?>
<DevTimeDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" >
 <MemberData MemberUri="http://dasha.ics.hawaii.edu:9877/sensorbase/users/johnson@hawaii.edu" DevTime="65"/>
</DevTimeDailyProjectData>

3.8 File Metric

Each FileMetric DPD provides a measure indicating "size" of each file associated with a Project for a given day. The size metric is not limited to LOC, it could also be number of methods, number of classes, function points, or any other metric that a tool could associate with each file in the Project.

The FileMetric DPD uses the "snapshot" approach to determining relevant sensor data. The snapshot approach avoids the problem of double counting and/or deleted files when a sensor is invoked more than once a day. Thus, tools that generate FileMetric data must use the Runtime field to indicate that a given set of sensor data was all generated at a single point in time and should be associated together.

Here is an example invocation:

An example return value is:

<?xml version="1.0"?>
<FileMetricDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" SizeMetric="TotalLines" Tool="SCLC" Total="227">
 <FileData FileUri="file://users/johnson/Foo.java" SizeMetricValue="67"/>
 <FileData FileUri="file://users/johnson/Bar.java" SizeMetricValue="73"/>
 <FileData FileUri="file://users/johnson/Baz.java" SizeMetricValue="87"/>
</FileMetricDailyProjectData>

The return value also indicates the tool (in this case, SCLC) used to generate the data.

The FileMetric URI also supports the following optional parameters:

3.9 Issue

The Issue DPD provides a snapshot of the state of an Issue tracking system (such as Bugzilla, Jira, etc.) with respect to the given Project on a given day. Issue data will generally have four kinds of attributes associated with it:

You can supply an optional parameter "Version" in order to filter the Issue sensor data for this Project.

Here is an example invocation:

An example return value is:

<?xml version="1.0"?>
<IssueDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" Version="8.0.612">
 <IssueData Type="Defect" Status="Open" Priority="Major" Number="3"/>
 <IssueData Type="Defect" Status="Closed" Priority="Major" Number="1"/>
 <IssueData Type="Defect" Status="Closed" Priority="Minor" Number="1"/>
 <IssueData Type="Enhancement" Status="Closed" Priority="Minor" Number="1"/>
</IssueDailyProjectData>

Each IssueData entry represents a set of non-overlapping Issues. In other words, you can determine the total number of Issues for this Project on this day by summing up the number of Issues across all of the IssueData instances.

3.10 Unit Test

The UnitTest DPD provides, for each member, the total number of successful and unsuccessful unit test invocations for that Project on that particular day.

Here is an example invocation:

An example return value is:

<?xml version="1.0"?>
<UnitTestDailyProjectData Project="Default" Owner="johnson@hawaii.edu" StartTime="2006-01-31T00:00:00.000" >
 <MemberData MemberUri="http://dasha.ics.hawaii.edu:9877/sensorbase/users/johnson@hawaii.edu" success="5" failure="3"/>
</UnitTestDailyProjectData>

3.11 Ping

In addition to the DPD abstractions, this service also supports the "ping" URI, which is a low cost way of testing whether a host URI is responding to requests, and what kind of Hackystat service it is. No authentication is required. The ping request returns the string "DailyProjectData".

If user and password parameters are supplied, then the ping request returns "DailyProjectData authenticated" if they are valid. This involves checking with the underling SensorBase.

3.12 Cache

The DailyProjectData service supports caching of DailyProjectData instances using a single global cache called the "Front Side Cache". (There is a "back side cache" for Sensor Data instances, but that is not part of this API.)

DPD caching can greatly improve performance by reducing the amount of calls to the SensorBase service and resulting computation over those instances. However, when the definition of a project changes (such as by adding or deleting members, or by changing the UriPatterns), the current cached instances may no longer be valid. In this situation, the user needs to clear the cache for the associated project and user so that the next retrieval of a DPD instance for that project will reconstruct the analysis from scratch.

There are two commands to support cache deletion. The following command deletes all cache entries for the authorized user:

The following command deletes the cached instances associated with the authorized user that related to the specified project and its owner:

If the authorized user is not a member of the specified project, then an error is returned.

If caching is not enabled in this service, then this call has no effect.