Robust Ranking Model via Bias-Variance Optimization

Abstract

Improving average effectiveness is an objective of paramount importance of ranking model for the learning to rank task. Another equally important objective is the robustness—a ranking model should minimize the variance of effectiveness across all queries when the ranking model is disturbed. However, most of the existing learning to rank methods are optimizing the average effectiveness over all the queries, and leaving robustness unnoticed. An ideal ranking model is expected to balance the trade-off between effectiveness and robustness by achieving high average effectiveness and low variance of effectiveness. This paper investigates the effectiveness-robustness trade-off in learning to rank from a novel perspective, i.e., the bias-variance trade-off, and presents a unified objective function which captures the trade-off between these two competing measures for jointly optimizing the effectiveness and robustness of ranking model. We modify the gradient based on the unified objective function using LambdaMART which is a state-of-the-art learning to rank algorithm, and demonstrate the strategy of jointly optimizing the combination of bias and variance in a principled learning objective. Experimental results demonstrate that the gradient-modified LambdaMART improves the robustness and normalized effectiveness of ranking model by combining bias and variance.