Architecting a Network Query Engine for Producing Partial Results

Abstract

The growth of the Internet has made it possible to query data in all corners of the globe. This trend is being abetted by the emergence of standards for data representation, such as XML. In face of this exciting opportunity, however, existing query engines need to be changed in order to use them to effectively query the Internet. One of the challenges is providing partial results of query computation, based on the initial portion of the input, because it may be undesirable to wait for all of the input. This situation is due to (a) limited data transfer bandwidth (b) temporary unavailability of sites and (c) intrinsically long-running queries (e.g., continual queries or triggers). A major issue in providing partial results is dealing with non-monotonic operators, such as sort, average, negation and nest, because these operators need to see all of their input before they can produce the correct output. While previous work on producing partial results has looked at a limited set of non-monotonic operators, emerging hierarchical standards such as XML, which are heavily nested, and sophisticated queries require more general solutions to the problem. In this paper, we define the semantics of partial results and outline mechanisms for ensuring these semantics for queries with arbitrary non-monotonic operators. Re-architecting a query engine to produce partial results requires modifications to the implementations of operators. We explore implementation alternatives and quantitatively compare their effectiveness using the Niagara prototype system.