Update: This needs to be fixed to cover the install process for our updated tools.

Introduction

The Socorro project welcomes new hackers. This page is an attempt to explain the general structure of the Socorro codebase, as well as introduce the various technologies being used. If you are getting started, it is highly recommended that you join the socorro-commits group, which will keep you informed of new SVN commits as well as issue updates.

General Structure

Socorro is divided into three distinct functions: the collector (SocorroCollector), the processor (SocorroMonitor & SocorroProcessor), and the reporter (SocorroReporter).

The Collector

The collector is a simple web-facing applications that accepts minidumps via HTTP POST. It saves the minidumps and relevant metadata to disk. The collector logic is mostly contained in collect.py. It is deployed from the dist/cgi-collector or dist/modpython-collector directories.

To ensure scalability and uptime of crash report collection, the collector does not connect to the database. Each crash report is given a random UUID at collection, and saved on disk in a directory monitored by the processor.

The Processor

The processor is an OS process that monitors a directory for incoming minidumps. It then runs the breakpad minidump_stackwalk tool, and inserts the resulting data into the database. The monitor and processor logic are deployed from the dist/processor directory.

The Reporter

The reporter is a web-facing application that allows developers and QA to see the information from crash reports, do aggregate queries on reports, and generate reports. The webapp directory contains the reporter code.

Optionally, the reporter webapp can also be configured to allow for direct minidump upload and processing. This is mainly intended for development, as it is not scalable to the same degree as the separate collector and pocessor.

The Database

The database model is primarily described in models/__init__.py. The main reports table is joined with secondary frames, modules, extensions, and dumps tables.

Because the crash report dataset may become very large, the schema was designed to allow postgres partitioning. See DatabasePartitioning for more details.

Technologies

Socorro makes use of the following technologies:

• Python (Scripting Language) Note that for reasons of stability, version 2.4 specified. Syntax, methods, modules from later python versions are not available.
• PostgreSQL Database - Documentation, SQL Reference
• Pylons webapp development framework
• Paste Webapp deployment
• SQLAlchemy database object-relational-mapping - Documentation, Reference
• Genshi XML templating language - Documentation