Impact-Minimizing Runtime Adaptation in Cloud-Based Data Stream Processing

Abstract

Recently, cloud-based data stream processing has emerged to process huge amounts of data. During such processing, the actual characteristics of data streams may vary, e.g., in terms of volume or velocity. For example, in the financial domain hectic markets can cause bursty streams of events leading to changes of the stream characteristics by several orders of magnitude. To handle such situations, adaptation of the data processing at runtime is desirable. While several techniques for changing data stream processing at runtime do exist, one specific challenge is to minimize the impact of runtime adaptation on the data processing, in particular for real-time data analytics.

In this research work, we aim at performing runtime adaptation in cloud-based data stream processing, namely, dynamically switching alternative distributed algorithms, which have similar functionality, but operate at different characteristics (tradeoffs). The goal of this work is to provide a generic approach which can automatically determine the algorithm switch with minimized impact on the data processing. To achieve this goal, we introduce the concept of a “safe” (transparent, gap-free) switch, which takes the characteristics of alternative algorithms into account. For the actual switch, we combine stream re-routing with buffering and stream synchronization along with a support of dynamic deployment of alternative stream processing algorithms into the cloud.

Supervisors: Klaus Schmid, Holger Eichelberger.