Description of the Open rtVision system

The machine vision applications help us to understand the dynamic environment where we live. There are a lot of target applications such as robotics, industrial inspection, monitoring and security systems, medicine … or, in our case, in driving assisted systems. In fact, in the introduction we mentioned we are working on the EU project DRIVSCO which stands for “Learning to emulate perception-action cycles in a driving school scenario”. In this project, our work consists on the development of an embedded system integrating the algorithms for the extraction of several low-level visual primitives: energy, orientation, phase and also, disparity and optical flow estimation. All this computation has to be performed in real-time.

The software platform provides the possibility of configuring the system and displaying the results connecting a standard computer with a co-processing board (in our case, we use XircaV4 and RC2000). As we mentioned in the introduction, the work involved the development of the interface libraries to establish the communication with the co-processing board as well (in this description we also call it reconfigurable device). It makes the platform flexible and generic therefore developing a new specific interface library we could use any other co-processing board.

System Architecture

The complete system includes two cameras (high performance scientific or webcams), connected with a computer (via the USB or the Camera Link interface). In the case of the scientific cameras, we need a specific image acquisition device, a frame grabber. The application runs in the computer, acquiring the images from the cameras and sending this processed information to the co-processing platform using the PCIe (XircaV4) or the PCI (RC2000) interfaces. Once, the hardware computation is completed, the application reads the results from the board memory and post-processes the results for an appropriate visualization or storage.

Fig. 1 Scheme of the complete system, including the input through the cameras, a generic co-processing board, the application including the communication libraries via the PCIe interface and the acquisition, pre- and post- processing modules.

Hardware Level

The communication between the computer and the co-processing board is provided by the software and hardware interface libraries. This duality is called co-design. In hardware, this communication is implemented using two different handshaking protocols: the first one is needed for the parameter pass and the communication channel establishment, the second one performs the transmission using a double buffer or ping-pong scheme. In software, we developed an interface library for each one of our co-processing platforms. This represents one of the advantages of our work, developing a new specific interface library we could use any other co-processing board. The headings of the different functions of the library are shown above.

# Initialization of the device loading a bitstream file
InitFPGA(fileName, deviceHandler);


# Opening the communication with the device
OpenFPGA(deviceHandler);


# Parameter pass
InitCommunicationProtocol(deviceHandler, buffer, bufferSize);
# Writing data to the device memory using the double buffer scheme
Write(deviceHandler, buffer, bufferSize, Offset);
# Reading data from the device memory using the double buffer scheme
Read(deviceHandler, buffer, bufferSize, Offset);
# Commuting the bank for the double buffer scheme
CommuteBank(deviceHandler);
# Closing the communication with the device
CloseFPGA(deviceHandler);

The load of the hardware using the platform can be done using a JTAG programmer or just loading a bitstream file using the PCI/PCIe connection, if this posibility is available.

Software Level

This work is mainly focused on the software layout. It's the result of joining all the different tools used for the development, configuration and display of the image processing results. It also involved the tools used for the validation of the computations and the packages needed for the use of all this tools in real-time.

System specification

The implemented systems has been developed using the Visual Studio .Net 2003 and 2005 environments. Furthermore, we used different libraries and software packages for the optimization of the data management. The first step is the optimization of the code, minimizing the read and write operations to disk and the cache faults, unrolling loops and using threads for the recording computation.

OpenCV® (Open Source Computer Vision) is a software library for vision applications and it is optimized for the data processing in this field. We used it for the display of the results and the optimization of the filter post- and pre-processing. It also improves the throughput in the results storage on the hard disk such as image or video files.

The IPP® library package (Intel ® Integrated Performance Primitives) provides low-level routines for high performance applications. It improves the operations on well-aligned data arrays. The library provides also routines for matrix arithmetic operations, convolutions … All the routines are optimized for their use under Intel architectures.

OpenMP® (Open Multi-Processing) provides support for multiplatform parallel programming with shared memory. We use it for complex loops taking advantage of the multicore architectures. In this case, the scheduling (static or dynamic) is an important key for maximizing the performances.

The optimization is performed using the different packages but, the main contribution is done by the OpenMP use. An example of the performances obtained by the implementation with the different packages is shown above.

Fig. 2 The graphic shows the frame rate obtained for the computation of a 320x240 image, extracting all the visual primitives (energy, orientation, phase, disparity and optical flow). The best performances are achieved by the OpenMP use. The table shows the frame rate for the complete system using a 512x512 image.

Functionality

We present a list of functions for the Open rtVision environment:

1. Image difference, useful for calibration and disparity debugging
2. Rectification using LUTs. It also can be done in hardware
3. Debayerization
4. Snapshots
5. Input recording and storage
6. Results post-processing and storage
7. Configuration of the camera parameters
8. Configuration for the different processing parameters of the platform (such as input/output resolution, recording file formats, frame rate, colormaps …)
9. Display of the different results

Fig. 3 The figure shows the different dialogs of the application.

Fig. 4 The figures show two examples of use of the software. The first image shows the results for the local features (energy, orientation and phase) and the optical flow estimation. The second image shows the results for all the computations: local features, disparity and optical flow in a lab scenario.

Future work

As future tasks we will develop libraries for other co-processing boards and for new camera interfaces. Another improvement is the implementation of the system for Linux. Finally, we are working also with sockets to improve the communication between the different modules.

Users

The following video shows the use of the platform, it could help new users of Open rtVision.