Comparing Two Methods for Checking Runtime Properties

Abstract

A number of tools for runtime verification have been developed over the years, and just as many different formalisms for specifying the types of properties that they can check over the runs of a system. Though there have been some attempts to develop them, there are as yet no general benchmark suites available with event logs of a realistic size, together with the relevant properties to check. This makes it hard to compare the expressiveness and ease of use of different specification formalisms, or even the relative performance of the available tools. In this paper we try to address this to some extent by considering a formal logic and its supporting tool, and compare both ease of formalization and tool performance with a tool that was designed for an entirely different domain of application, namely interactive source code analysis. The static analysis tool needs just a few small adjustments to support runtime verification of event streams. The results of the comparison are surprising in that they unexpectedly show the static analysis tool outperforming the tool that was designed for runtime verification, both in accuracy and in performance.