Towards a Definitive Measure of Repetitiveness

Abstract

Unlike in statistical compression, where Shannon’s entropy is a definitive lower bound, no such clear measure exists for the compressibility of repetitive sequences. Since statistical entropy does not capture repetitiveness, ad-hoc measures like the size z of the Lempel–Ziv parse are frequently used to estimate repetitiveness. Recently, a more principled measure, the size $\gamma$ of the smallest string attractor, was introduced. The measure $\gamma$ lower bounds all the previous relevant ones (including z), yet length-n strings can be represented and efficiently indexed within space $O(\gamma \log \frac{n}{\gamma })$, which also upper bounds most measures (including z). While $\gamma$ is certainly a better measure of repetitiveness than z, it is NP-complete to compute, and no $o(\gamma \log n)$-space representation of strings is known. In this paper, we study a smaller measure, $\delta \le \gamma$, which can be computed in linear time. We show that $\delta$ better captures the compressibility of repetitive strings. For every length n and every value $\delta \ge 2$, we construct a string such that $\gamma = \varOmega (\delta \log \frac{n}{\delta })$. Still, we show a representation of any string S in $O(\delta \log \frac{n}{\delta })$ space that supports direct access to any character S[i] in time $O(\log \frac{n}{\delta })$ and finds the occ occurrences of any pattern $P[1{.\,.}m]$ in time $O(m\log n + occ\log ^\varepsilon n)$ for any constant $\varepsilon >0$. Further, we prove that no $o(\delta \log n)$-space representation exists: for every length n and every value $2\le \delta \le n^{1-\varepsilon }$, we exhibit a string family whose elements can only be encoded in $\varOmega (\delta \log \frac{n}{\delta })$ space. We complete our characterization of $\delta$ by showing that, although $\gamma$, z, and other repetitiveness measures are always $O(\delta \log \frac{n}{\delta })$, for strings of any length n, the smallest context-free grammar can be of size $\varOmega (\delta \log ^2 n/\log \log n)$. No such separation is known for $\gamma$.

Part of this work was carried out during the Dagstuhl Seminar 19241, “25 Years of the Burrows–Wheeler Transform”.

T. Kociumaka—Supported by ISF grants no. 1278/16, 824/17, and 1926/19, a BSF grant no. 2018364, and an ERC grant MPM (no. 683064) under the EU’s Horizon 2020 Research and Innovation Programme.

G. Navarro—Supported in part by Fondecyt grant 1-170048, Chile; Millennium Institute for Foundational Research on Data (IMFD), Chile.