• Introduction
• Automated Events
• Commit often
• Commit regularly
• Don't wait until the last minute
• Keep the repository clean
• Keep the repository really clean
• Perform QA locally before committing
• Keep the coverage high
• Keep the coupling low
• Keep the complexity low
• Collective ownership
• Provide Issue ID in commit messages (Issue Driven Design)
• Every member has an open issue (Issue Driven Design)
• Don't overload developers
• Self-reported Events
• Pair programming
• Code review
• Team meeting
• Toolsmithing
• The Wow Event
• Event Ideas
• Commit "right size" chunks
• Meeting the deadline
• Code reuse
• Documentation
• Development Ratio
• New Skills
• Recognition
• Coaching
• Open and Close Issues Timely
• Green Programming

Introduction

The heart of Devcathlon is the "event", which is an observable and repeatable behavior/characteristic of the software process or product along with an algorithm for how the behavior/characteristic will be identified and how points will be awarded based upon the observed behavior.

For example, one event might be summarized as "Maintain high coverage". One possible implementation of this event is a timer-based process that wakes up at the end of each day and checks to see if coverage data has been received. If so, and if the line-level coverage is above 90%, then the team will be awarded 10 points.

This same "Maintain high coverage" event might also be implemented by monitoring the sensorbase for commit events from a project, and then checking to see if coverage data is available within a short time frame after the commits (such as 1 hour). If so, and if the method-level coverage is above 85%, then the team will be awarded 5 points.

As this example illustrates, it is possible for the same high-level behavior or characteristic of software development to be implemented in multiple, equally reasonable ways. For a given match in Devcathlon, the participating teams should be able to decide both the high level behaviors they want observed (such as "maintain high coverage"), as well as the concrete implementation associated with it ("monitor once a day, provide 10 points if over 90% line-level coverage").

The following specifications of events thus begin by summarizing the behavior or characteristic of interest at a high level. This is followed by one or more concrete implementations, which are known in Devcathlon as "Point Award Rules" (PARs). PARs are implemented as Java classes that take a Match instance (from which it can determine the teams or individuals involved), a wakeup interval (how frequently to check for the behavior), and zero or more additional parameters. From this information, it will wake up periodically from the start of the match to the end, assess the behavior and invoke methods on the Match instance to award or deduct points.

Note that each event includes a "Decay/Incentive" section. This refers to the desire to design awards such that participants are incentivized to not "specialize" on a single kind of point getting, but to rather attempt to get points in as many different events as possible. To do this, some events support "decay", in which the amount of points awarded decreases if the participant has already been awarded points for that event previously. Conversely, an "incentive" is a bonus point award for the first time a team or individual gets a point award for that event.

Automated Events

These events can be automatically monitored via Hackystat sensor data. Values in italics should be configurable.

Commit often

Summary: Penalize developers for working on new code too long without committing it and thus making it available for others to see and integrate.

To compute this event, you must maintain a WakeupReferences table.

The WakeupReferences table contains a map from file paths (found in the Resource field of DevEvent data) to the number of times that file path has been found during a wakeup. For example, if you found file Foo.java during the last three wakeups in one or more DevEvent instances, then the table would contain {Foo.java, 3}. Note that you are not counting the total number of DevEvents for that file, just the number of times you found at least one DevEvent for that file during a given wakeup.

Simple algorithm: Get all DevEvents for the interval, create a HashSet containing all of the files that were referenced. Then, increment the WakeupReferences table by 1 for each file.

What we want to do is figure out each time we wake up if a file has been worked on "too much" since the last time it was committed.

PAR: Wakeup once every 60 minutes.

Update the WakeupReferences table. Then, get all Commit data during that interval. For each file referenced in the Commit data, zero their corresponding entry in the WakeupReferences table.

Next, check to see if any file in the WakeupReferences table has been referenced more than 10 times. This indicates that the file has been the subject of work for at least 10 intervals without any commits having been done.

Points are assigned as follows: 0 points if a file is committed before 10 WakeupReferences have occurred. At 10 WakeupReferences, deduct -10 points. Then, wait until double that number of WakeupReferences, and at that point deduct another -10 points.

Decay/Incentives: None needed. Commit early is a "one time" point award (per file).

Rationale: Penalize developers for negative behaviors, but don't reward spurious commits. Note that this event works best in conjunction with events that penalize developers for breaking the build or having low coverage.

Commit regularly

Summary: Reward teams who work regularly.

PAR: Wake up once per day during the match. Check to see that 100% of the team members have worked at least 60 DevTime minutes during the preceding interval, and that 50% of the team members have committed at least once during the interval. If so, then award the team +5 points.

Decay/Incentives: We could consider "bonus" points for the first two days of the match in order to incentivize getting started immediately.

Rationale: Working regularly on the project is a good thing.

Don't wait until the last minute

Summary: Penalize teams that work significantly more near the end of a match.

PAR: During the last 3 days of the match, wakeup once per day and compute the average DevTime, number of builds, and number of commits per day over the entire match thus far. If any two of DevTime, number of builds, and number of commits exceeds the average by 50%, then deduct -5 points.

Decay/Incentives: None needed.

Rationale: Schedule your work so that it doesn't pile up at the end.

Keep the repository clean

Summary: Penalize (or award) developers for committing code that fails (or passes) a continuous integration build.

This event requires that Build sensor data is generated from the continuous integration server, and that CI build sensor data has a special property associated with it (such as Type=continuous.integration). This is to distinguish continuous integration Build sensor data from local builds (for which frequent failure is to be expected.) Note that the specific tag to be checked for could be a configurable property for each match.

PAR-1: Wakeup every 60 minutes. If there is any CI Build sensor data that does not have a successful Result type, then assign -5 points. Note that you get a deduction of a maximum of -5 points no matter how many times you fail the build during the interval.

Rationale: If you commit code that fails the build, you shouldn't be penalized further for any additional commits you make trying to fix the failing build as long as you do them right away (within an hour of the original failing commit). However, if you wait more than an hour to try to fix it, then you should incur additional penalties.

PAR-2: Wake up every 1 day. If there are successful CI Build events during that day, then assign +5 points.

Rationale: Reward developers for passing continuous integration builds. Note that this is relatively easy to "fake", as there is no verification in the design of this PAR that any actual work was done before kicking off the build. We could look for threshold churn values to ensure that there was nontrivial editing, at least.

Keep the repository really clean

Summary: Award developers for having a CI build that includes extra quality assurance steps beyond compilation. For example, JUnit, Checkstyle, PMD, or FindBugs.

PAR: Wakeup every 60 minutes. If there is CI Build sensor data whose results are all passing, then award +5 points.

Rationale: We want to incentivize teams to implement rigorous CI builds with lots of quality assurance. The PAR as designed does not validate that extra QA steps are being taken; teams would need to be on the honor system regarding this.

Perform QA locally before committing

Summary: Reward developers for running quality assurance tests before committing.

PAR: Wakeup every 60 minutes. If there is Commit sensor data, then check to see if there is JUnit, Checkstyle, PMD, or FindBugs sensor data within five minutes of the commit by that same owner. If so, award +5 points.

Rationale: We do not penalize failure to run QA tests before commits because sometimes a small change does not warrant the checks. Furthermore, if you commit and the build fails, you'll get a deduction at that point.

Keep the coverage high

Summary: Reward (or penalize) developers who commit code that improves (or degrades) the quality of the test suite.

PAR: Wakeup every 60 minutes, and see if there has been a CI build event. If so, check to see if there is coverage data available since that CI build event. If so, compute the line-level coverage and check it against the last available coverage value (if any). If the coverage is within 1% of the previous coverage (or if no previous coverage data is available), do nothing. If the coverage is over 90%, do nothing. Otherwise, award +1 (or -1) points, up to a maximum of 10 points, for each increase (or decrease) in coverage. Always store the coverage data found on each wakeup for comparison to the next wakeup.

Decay/Incentives: Award a one time bonus of 10 points once the upper threshold of 90% is achieved.

Rationale: The limit of a 10 point award keeps this in line with other point awards. Having an upper threshold (like 90%) means that once you get over that threshold, falling coverage will not count against you. The incentive award effectively gives you automatic credit for going from 90% to 100% as soon as you cross that theshold.

Keep the coupling low

Summary: Penalize teams whose code coupling exceeds a threshold.

PAR: Wakeup every 1 day. Compute the average coupling (afferent + efferent) per class. If the average exceeds 5.0, make a one time point deduction of -5 points. Thereafter, make a one time point deduction of -5 points each time the average coupling increases by an additional full point. (In other words, deduct another 5 points when the average coupling reaches 6.0, and another if it reaches 7.0, etc. )

Decay/Incentives: Decay is already built in via the "one time" nature of the deductions.

Rationale: The PAR builds an incentive in to keep the coupling low and to not let it increase significantly.

Keep the complexity low

Summary: Penalize developers whose complexity exceeds a threshold.

PAR: Same as "Keep the coupling low".

Decay/Incentives: Same as "Keep the coupling low".

Rationale: Same as "Keep the coupling low".

Collective ownership

Summary: Reward developers for ensuring that most files are worked on by multiple developers.

PAR: Wake up once a day. Gather DevEvent data to construct/update a map of all files that are being worked on, which developers have worked on which files, and how much DevTime they have so far contributed to each file. If a developer has contributed at least 1 hour of DevTime to a particular file, then their contribution is recognized, otherwise it is not. Award the team +10 points if 50% of all files have been worked on by at least 2 developers.

Decay/Incentives: Award 5 bonus points if 90% of all files have been worked on by at least 2 developers. Do not award any points for a given day if less than 2 developers have worked on that day and contributed less than 1 DevTime hour.

Rationale: Want to encourage developers to work together on files. However, we do not want to award points when little development effort is being performed on the system.

Provide Issue ID in commit messages (Issue Driven Design)

Summary: A basic practice of Issue Driven Design is that every commit message should reference the Issue that the associated code changes are related to. Penalize developers when they don't have a log message with their commits. Reward them when their commit messages include an Issue ID.

PAR: Wake up once per day. Look at all commit sensor data for the given team. If there is commit sensor data for the given interval, and if all log messages have a message, and all those messages reference an Issue ID, then award 5 points. If there is commit sensor data with no log message, deduct 1 point.

Decay/Incentives: We could have some kind of decay rule that takes effect if a team always references the same Issue ID, indicating that they really aren't practicing IDD, they're just trying to get the points.

Rationale: Nothing further.

Every member has an open issue (Issue Driven Design)

Summary: Reward development teams where each project member has at least one open issue, indicating their current focus of work.

PAR: Wake up once per day. Check the current open issues. If each project member has at least 1 open issue, then award 5 points. If not, deduct -1 points.

Don't overload developers

Summary: Penalize teams where one or more members have too many open issues, indicating there is no clear focus for what they should work on next.

PAR: Wake up once per day. Check the current open issues. If any project member has more than 5 open issues, then deduct -5 points.

Rationale: It should be possible by looking at the issue list to see what each person should be working on. There should not be an excessive number of open issues associated with a single developer.

Self-reported Events

The following are events that Hackystat cannot currently monitor. Devcathlon must provide some kind of alternative interface for allowing developers to report on these events, as well as some mechanism for verification.

Pair programming

Summary: Reward developers who engage in pair programming.

PAR: After a pair programming session, one of the participating developer can login to a special section associated with the match they are participating in, and record the date, time, and duration of the pair programming session. The system can then check that there is DevTime for one of the two developers during that interval. This event instance then gets put on a queue of items to be validated by a member of the opposing team. If validated, the pair programming session is worth +10 points.

Decay/Incentives: No more than 2 pair programming sessions can be awarded in a single day to a single team.

Code review

Summary: Reward teams doing code review.

PAR: After a code review, one of the participating developers logs in to a special section to indicate their participation. They must indicate (a) how much time each member spent on the review, and (b) how many significant issues each member posted. They must also indicate the site (such as Google Project Hosting) where the review took place. This event instance then gets put on a queue of items to be validated by a member of the opposing team. If validated, the code review is worth +5 points for each participating member who found more than 5 significant issues.

Decay/Incentives: No more than 1 code review can be awarded in a single day to a single team.

Team meeting

Summary: Reward teams for productive face-to-face team meetings.

PAR: After a meeting, one of the participating developers logs in to a special section to indicate their participation. They must indicate (a) how long the meeting was; (b) how many members attended; and (c) the URL to the Minutes of the meeting. This event instance then gets put on a queue of items to be validated by a member of the opposing team. If validated, the meeting is worth +5 points.

Decay/Incentives: No more than 1 team meeting can be awarded in a single day to a single team.

Toolsmithing

Summary: Reward developers for creating a generally useful tool or finding a library that saves significant time gets points.

PAR: The developer logs in to a special section to indicate their toolsmithing. They must (a) describe the tool they created or the library they found, and (b) provide a URL to the tool. This event instance then gets put on a queue of items to be validated by a member of the opposing team. If validated, the toolsmithing is worth +5 points.

Decay/Incentives: No more than 2 toolsmithing awards can be provided in a single match.

The Wow Event

Summary: There are times when other developer does a task that causes you sit back and say "Wow". Either "Wow! that was amazing" or "Wow! what the bleep was that developer thinking". Example of a positive "Wow" is finding a "thinking outside the box" solution to a problem. Example of negative "Wow" is doing work that just was not needed. Reward developers for a positive "Wow" and punish for negative "Wow".

PAR: A grading developer finds a work done by other developers which is either a positive or negative "Wow". The grading developer who has only a few "Wow" gold badges can give those badges to other deserving developers. The grading developers can give infinite amount negative "Wow" badges. But each negative badge can be overturned by other developers which then causes the grading developer to lose all there gold "Wow" badges.

Decay/Incentives: Reward for all positive with gold "Wow" badge and penalize once for a negative "Wow" per day by subtracting points.

Event Ideas

Ideas for events that aren't fully worked out yet.

Commit "right size" chunks

Developers should strive for "Goldilocks" commit behavior: not too much code, not too little code.

Example point scheme: not sure

Rationale: Reward developers when they commit a "reasonable" amount of changes.

Meeting the deadline

Getting the project done on time or meeting a particular milestone deserves points.

Code reuse

Writing generic code that can be provided as a library.

Documentation

Summary: Writing documentation is good.

Development Ratio

Summary: The developer should have at least 1 commit per 1 hour of coding.

New Skills

Summary: Points or level advancement for the developer that applies something new i.e., new language or technology.

Recognition

Summary: The project or code gets noticed by the professional computing community.

Coaching

Summary: Reward to the individual that solves a problem and shares it with the other programmer, who cannot effectively solve problem alone. This other programmer has to understand the solution and how it was implemented so he can become a better developer.

PAR: After reaching a solution and shares with the other programmer, the developer that come up with the solution logs in to a special section to indicate their participation. They must indicate (a) what was the problem; (b) what was their solution; and (c) the other developer they were working with. This event instance then gets put on a queue of items to be validated by a member of the opposing team. If validated, the individual came up with the solution gets +15 points. While the other programmer gets +10 points.

Decay/Incentives: No more than 1 solution can be awarded in a single day to a single programmer. (phillip)

Open and Close Issues Timely

Summary: Points rewarded to players who accept issues and closes it within a certain amount of days. The number of days issues should be opened depends on the game administrator.

PAR: After changing an issue in the Google Project Hosting site to ACCEPTED, a player will receive 5 points for every issue that is closed within a certain amount of days (example: an issue must be closed after 1-2 days).

Decay/Incentives: Only a maximum of 2 issues can be closed a day to receive points. This is because if you are closing more than 2 issues a day, that means you are behind schedule and are rushing/cramming the project. Issues closed too early or too late will not be rewarded points. (scheller)

Green Programming

Summary: Points are rewarded to teams that reach a green stat in the Hackystat.

PAR: Admin picks out a stat or several stats where several teams have low scores in. They can gain points by returning those stats to the green. The amount of points can be based on how quickly they return to the green.

Decay/Incentives: In cases where stats are lower than they should be this gives programmers an incentive to raise those stats. (zhou)