Testing Whether a Prediction Scheme is Better than Guess

Abstract

To be able to predict a future event is the most convincing side of science. Usually in the beginning of an investigation, the prediction is not perfect, i.e., an event may be missing or a prediction turns out to be a false alarm. When past prediction records are available, can we determine whether the prediction scheme is promising? In this paper, we use the naive optimal prediction as a yard stick, i.e., testing whether the new prediction scheme is better than the naive optimal scheme through statistical hypothesis testing. Here the naive scheme is defined as we know only the distribution of the inter-arrival times without any other auxiliary information. We use the trade off curve between false alarm rate and sensitivity to measure the prediction performance and the bootstrap method to compute the p-value. Real data and simulation examples are presented.