Real-Time Embedded Systems Lab
School of Computing, Informatics, and Decision Systems Engineering
Ira A. Fulton School of Engineering, Arizona State University
Home Contact us Site Map
Faculty People


Replay Debugger for Real-Time Embedded Systems

1. Introduction

The Replay Debugger based on Lamport’s happened-before relations [1] is designed for multi-threaded embedded software. The debugger considers a multithreaded execution as a partially ordered sequence of interactions and reproduces the sequence in cyclic debugging process. Hence, the developers can inspect the execution steps of each thread as well as the interactions among threads and external world. They will not be distracted by context switches or the physical time instants at which events and interactions occur. As a consequence, the debugging process can be focused in any single thread while the execution of other threads is coordinated precisely to reproduce identical software behavior.

2. Structure of Replay Debugger

The Replay Debugger is implemented as a record/replay framework layered between multithreaded applications and operating system kernel. Figure 1 shows the structure of the current implementation of Replay Debugger.

2. 1 Wrapper Functions

The Replay Debugger library consists of a replay scheduler and a set of wrapper functions for POSIX APIs. The current implementation supports the following APIs,

      • IO: read() and fread()
      • Message Queue: mq_open(), mq_send(), and mq_receive()
      • Pthread: pthread_create(), pthread_mutex_init(), pthread_mutex_lock(),
         and pthread_mutex_unlock()
      • Semaphore: sem_init(), sem_wait(), and sem_post()

2. 2 Record

During recording, IPC and IO function calls are replaced by corresponding wrapper functions. Then, the happened-before relations among the execution of multiple threads and IO events are constructed and saved into the log files. In addition, external input data is saved into the log files for replay.

2. 3 Replay

During replay, as in the record phase, IPC and IO function calls are replaced by corresponding wrapper functions. Using the log files from the record phase, the deterministic execution of threads can be repeated and thread execution can be controlled individually following the partial order of thread interactions. External input data is replayed using the saved data from the record. The threads in the replay are scheduled by the replay scheduler thread. The scheduler thread is created as a user thread in the application, but it is transparent to the user. At the entry of each wrapper function, the thread is suspended waiting for permission to execute the event from the scheduler thread. Let Ea be the event suspended and Eb be the event happened before Ea. If Eb is executed or does not exist, then the scheduler thread signals the suspended thread so that Ea can be executed. Figure 2 shows an example of the scheduling.

2. 4 Replay Mode

Given a partially ordered happened-before graph, there are many different execution orders. In addition to repeatable execution ordering through happened-before relations, Replay debugger provides two replay scheduling modes: Replay Minima and Replay Maxima. A user sets Replay Minima or Replay Maxima for a specific thread. In Replay Minima mode, only minimal amount of execution is carried out for the designated thread. On the other hand, for Replay Maxima mode, maximum amount of execution is carried out for the designated thread. An example of using the two replay modes is shown in Figure 3.

2. 5 Integration with GDB

To make the Replay Debugger more practical, the replay mechanism has been incorporated into GDB [2] with an extended debugging interface for thread control and observation. Since the deterministic replay is already guaranteed by the replay library, users can issue any GDB command without worrying about scheduling threads and feeding inputs to reproduce the same execution sequence as when bugs found. GDB communicates with the replay through log and control files. GDB can read the current execution status of the application and change the Replay mode. Aside from the normal GDB breakpoint, Replay Debugger provides LC breakpoint. Each execution of an event is defined by happened-before relation and recorded with Lamport Clock (LC) [1]. When a user sets a LC breakpoint at a LC, every thread stops at an event for the designated LC. Figure 4 shows an example of LC breakpoint.

The current implementation of Replay Debugger has been incorporated with GDB 6.8.

2. 6 Eclipse Support

We have also implemented a graphical environment for our Replay Debugger in the form of Eclipse [3] plug-ins. It supports build options for the Record/Replay and graphical display of threads and events. Figure 5 shows a snap shot of Eclipse support in the Replay Debugger. For more details see the User Manual.

3. Software Release

 The Replay Debugger 1.0.0 is now available for download --
              -- Compiled binaries
              -- Source Code
              -- Applications

4. Document

Replay Debugger 1.0.0
      - Replay Debugger with Eclipse   here
      - Eclipse Compile/Installation Guide   here
      - Replay Debugger in text mode using GDB   here

Publication --
Y. Lee, Y. Song, R. Girme, S. Zaveri, Y. Chen, “Replay Debugging for Multi-threaded Embedded Software”, EUC 2010, December 2010.
5. References

[1] L. Lamport, “Time, clocks, and the ordering of events in a distributed system,” Communications of the ACM, Vol.21(7), pp: 558-565, 1978.
[2] R. Stallman, R. Pesch, S. Shebs, et al., “Debugging with GDB (ninth edition, for GDB version 7.1),” Free Software Foundation, 2010.
[3] Eclipse,


Our Research
Our Goal
We do research on  Real-time Java, Embedded Software and Systems, Smart homes, and so on...
To build reliable real-time embedded system, contribute computer engineering community and make our future better.

 Copyright [2011] [RTES, School of Computing, Informatics, and Decision Systems Engineering. Arizona State University]. All rights reserved