Split Dictionaries for In-memory Column Stores in Mixed Workload Environments

Abstract

Columnar in-memory databases use dictionary encoding as a compression technique, replacing long and frequently occurring values with short integers. Sorted dictionaries allow for more efficient query processing as comparisons can be performed directly on the compressed data whereas unsorted dictionaries are faster when inserting new values.

In this work, we propose a new type of dictionary compression called Split Dictionaries. These organize their values in fixed-sized splits, enabling fast inserts and comparable query performance while significantly reducing maintenance costs. We present a detailed performance analysis regarding inserts, range queries, and the merge process as well as a memory usage model. We argue that adjusting the dictionary size allows for a more balanced trade-off especially in mixed workload environments.