Towards Automated Defect Analysis Using Execution Traces of Scenario-Based Models

Abstract

Modern software systems are so complex that at times engineers find it difficult to understand why a system behaves as it does under certain conditions, and, in particular, which conditions trigger specific behavior. This adds a significant burden to tasks like debugging or maintenance. Scenario-based specifications can mitigate some of the problems engineers face thanks to the scenarios’ intuitiveness, executability and amenability to formal methods such as verification and synthesis. However, as a specification grows it becomes increasingly difficult to fully comprehend the interplay of all scenarios, thus again making it difficult to understand the final system’s behavior. Therefore we propose a (semi-)automatic trace analysis method. It incorporates automatic techniques for identifying interesting traces, or subsequences within traces, from large sets of long execution traces. Developers are interested in knowing whether certain specification properties hold: if a property holds, what are possible executions which are evidence of this? If a property does not hold, what are examples that violate it? Scenario-based specifications are well-suited to generating meaningful execution traces due to being based on events which are meaningful in the domain of the system under design. A key observation we made was that interesting properties of a trace are often encoded in just one or very few scenarios. These concise scenarios, describing desired or forbidden behavior, are often already part of the specification or should be added to it as they encode implicitly made assumptions.

This paper incorporates and substantially extends the material of the paper published in MODELSWARD 2018 “Towards Systematic and Automatic Handling of Execution Traces Associated with Scenario-based Models” [1].