Dynamic pricing and learning with competition: insights from the dynamic pricing challenge at the 2017 INFORMS RM & pricing conference

Abstract

This paper presents the results of the Dynamic Pricing Challenge, held on the occasion of the 17th INFORMS Revenue Management and Pricing Section Conference on June 29–30, 2017 in Amsterdam, The Netherlands. For this challenge, participants submitted algorithms for pricing and demand learning of which the numerical performance was analyzed in simulated market environments. This allows consideration of market dynamics that are not analytically tractable or can not be empirically analyzed due to practical complications. Our findings implicate that the relative performance of algorithms varies substantially across different market dynamics, which confirms the intrinsic complexity of pricing and learning in the presence of competition.

Appendix: Competitor algorithms

Competitor logit

This competitor models the demand according to a finite mixture logit model, where the mixture is taken over the number of possible customer arrivals. Thus, a probability distribution over the number of arrivals in a single period is estimated and for each possible number of arrivals, a different multinomial logit model is estimated as well. Each multinomial logit model here, induces a probability distribution over the competitors, i.e., it specifies with which probability an arriving customer purchases from each competitor (including a no-purchase option). In doing so, it is assumed that the utility of buying from competitor i is of the form $a-bp_i$, where $p_i$ is the price posted by competitor i and a and b are assumed to differ across the mixture components.

In practice, this competitor uses the first 100 time periods to estimate the maximum number of arriving customers in a single time period. This is done by setting a price of 0 for the first period and for each of the following 99 periods of this exploration phase, the price is set as the minimum of the prices observed in the previous period. After these 100 periods, an upper bound on the number of arrivals in a single period is taken as the maximum realized demand in a single period multiplied by $(m+1)$, i.e., the number of competitors plus one. Subsequently, an Expectation-Maximization algorithm is used to estimate a probability distribution over the number of arrivals, as well as the parameters of the multinomial logit models. All these parameters are updated every 20 time periods.

To optimize prices, in every period the competitors’ prices for the period to come are predicted. For this purpose, it is assumed that the sorted prices of the competitors follow a multivariate normal distribution, where the sorted prices are used to mitigate the effect of price symmetries. Subsequently, 1000 competitor prices are sampled from the multivariate normal distribution and the revenue function is approximated by averaging over these realization. To optimize the price, a crude line search over a discretization of the assumed price space (0, 100) is executed and the price with the highest revenue is chosen.

Competitor ols

The approach of this competitor to pricing is to favor simplicity. The view is taken that competitors’ actions cannot be controlled and that for all intents and purposes, they are random. Thus, they are modeled as an aggregate source of random “noise” and the focus is on how the competitor’s own price influences demand in this environment. The algorithm is split into an exploration segment and a “running” segment. The exploration segment lasts for the first 40 periods and the running segment lasts for the rest of the 960 periods.

In the exploration segment, the algorithm explores the field to ensure sufficient variation in data. In each period, a price is sampled uniformly from the interval (0, 100). After the exploration period, the algorithm enters the running segment. In the running segment, the majority of the time consists of estimating a demand curve based only on the competitor’s own historical prices and optimizing accordingly. To do so, four linear regression models are fit, taking all combinations of log-transformation of both independent (price) and dependent (demand) variables, and the model with the highest $R^2$ value is chosen (ols is an acronym for ordinary least squares). Using this model, the price is optimized using a crude line search and, subsequently, a small perturbation is added to the price for further exploration.

Finally, in each period in the running segment there is a 5% chance of further exploration and a 1% chance of “competitive disruption”. Here, “competitive disruption” is an action designed to intentionally confuse competitors who attempt predict competitor prices or who use competitor prices in their model. When this action is initiated the model sets the price to zero in an attempt to confuse competitors via extreme actions.

Competitor b-grid

b-grid is adapted from the $\varepsilon$-greedy multi-armed bandit algorithm (Sutton and Barto 1998). It assumes a bandit framework with ten arms, where the arms pertain to the prices $10, 20, \ldots , 100$ (b-grid is an acronym for bandit on a grid). Thus selecting the first arm means posting a price of 10. This algorithm neglects competition and simply keeps track of the average revenue under each arm. With probability $\varepsilon$, an arm is selected randomly, whereas with probability $1-\varepsilon$, the arm that has the highest observed average revenue is selected. The exploration parameter $\varepsilon$ is set to 0.2, so that on average 200 time periods are used for exploration and 800 for exploitation.

Competitor b-bucket

This competitor considers the problem of learning and pricing in a multi-armed bandit framework similar to that of b-grid. In doing so, the optimal price is assumed to be contained in the interval (0, 100], which is split into ten intervals of even length, i.e., it is split into price buckets $(0, 10], (10, 20], \ldots , (90, 100]$. Each of these price buckets pertains to one arm and selecting a specific arm means posting a price that is uniformly sampled from the corresponding price bucket (b-bucket is an acronym for bandit with buckets).

To incorporate the competitors’ prices, it is assumed that the arms’ values, i.e., revenues, depend on the prices posted by the other competitors. More precisely, in each time period the competitors’ modal price bucket is forecast using exponential smoothing. The modal price bucket is the bucket that is predicted to contain most of the competitors’ prices. We assume that the optimal choice of price to offer is dependent on this modal price bucket.

In practice this works as follows. At each time step, with probability $\varepsilon$ an exploration step is performed in which an arm is selected randomly. Alternatively, with probability $1-\varepsilon$, an exploitation step is undertaken. In this case, the algorithm selects a price from the price bucket with the highest observed average revenue for the predicted modal price bucket. The exploration parameter $\varepsilon$ is set to 0.2, so that on average 200 time periods are used for exploration and 800 for exploitation.

Competitor b-model

This competitor advocates a bandit formulation of the problem as well, although its design differs conceptually from that of b-bucket and b-grid. Where the aforementioned two competitors assign prices (or prices buckets) to arms, here, an arm pertains to a demand model (b-model is an acronym for bandit with models). The demand models that constitute the four arms are the following:

• Demand Model 1: (Bargain hunters) assumes that the distribution of customers’ willingness to pay (WTP) is normally distributed and that customers select a competitor’s price from the subset of prices that fall below their WTP with probability proportional to $\left( \frac{{\rm WTP}-p_{i}}{{\rm WTP}}\right) ^{b}$, where $p_i$ is the price being offered by competitor i and b is a parameter that influences customers’ price sensitivity. High (low) values of $b>1$ ($<1$) capture customer populations that are highly (in)sensitive to prices close to their reserve price. In general this first demand model captures bargain hunters as in all cases customers will tend to choose low prices where possible.

• Demand Model 2: (Quality seekers) is a variant of the first demand model but the reserve price of a customer is proportional to $\left( 1-\frac{{\rm WTP}-p_{i}}{{\rm WTP}}\right) ^{c}$. This model captures customers who use price as an indicator of quality. The parameter c has a similar interpretation to b above.

• Demand Model 3: (Cheapest price subset) assumes that each customer sees a different random subset of the available prices. Customers are assumed to select the cheapest price that is visible to them. The subset is assumed to include a random number of options uniformly distributed between d and e, which are parameters that can be estimated from the demand data.

The fourth arm alone is used for the first 100 time periods with a relatively high exploration rate to provide sufficient data for estimating the parameters a, b, c, and d of the three demand models by means of simulated annealing. After 100 time periods the reward vectors are reset and the four-armed bandit assumes control of pricing. Similar to the previous two bandit algorithms, with probability $\varepsilon$ an arm is selected randomly and otherwise the most profitable arm is selected.

Optimal prices are chosen based on a forecast of the competitor price (duopoly) or the profile of competitor prices (oligopoly), where we define the competitor price profile to be an ordered list of competitors’ prices. For the oligopoly the competitor price profile is forecast for the next time period using exponential smoothing with trend. In order to estimate the optimal price to charge under each demand model, the algorithm generates a set of potential prices and the projected revenue is evaluated at each price, for the forecast competitor price profile. The price with the highest predicted revenue is assumed to be the best price for this demand model. In an exploitation step, the algorithm selects the arm with the highest predicted revenue and offers the best price for this arm.

Competitor ml

The approach of this competitor is to rely on machine learning techniques to predict demand and optimize prices accordingly (ml is an acronym for machine learning). Much emphasis is put on learning the demand characteristics, as the algorithm dynamically switches back and forth from exploration to exploitation mode over time. In exploration mode, during forty time periods, prices are set according to a cosine function around the mean price level observed to test a variety of price levels and, possibly, confuse competitors. After this learning cycle, demand is modeled using own prices and the competitor prices as covariates by means of a variety of regression models (least-squares, ridge regression, Lasso regression, Bayesian ridge regression, stochastic gradient descent regression, and random forest) and the best model, in terms of demand prediction, is selected through cross-validation.

Subsequently, the model of choice is used during an exploitation cycle of variable length: the length is sampled uniformly between 70 and 150, however, if the revenue earned deteriorates too fast, then, immediately a new exploration cycle is initiated. The price is optimized by discretizing the price space and computing the revenue for all prices. When a new exploration cycle starts, so either when the exploitation cycle was finished or because the revenue deteriorated significantly, all historical data is disregarded for the benefit of capturing shifts and shocks in the market most adequately.

Competitor greedy

This competitor advocates a particularly simple strategy: set the price as the minimum price observed in the previous time period. To avoid a “race to the bottom” with another competitor, the following facility is implemented: if the minimum price observed in the previous period is lower than the 10% percentile of all the prices observed in the last 30 time periods, then the price for the coming period is set as the maximum of this percentile and 5 (i.e., if this 10% percentile is smaller than 5, the price is set to 5).

Competitor wls

The characterizing feature of this competitor is that it aims to maximize own revenue relative to its competitors. More precisely, it attempts to maximize own revenue minus the revenue of the competitor that earns the most revenue. In doing so, it is assumed that demand of competitor k equals $d(p_k,p_{\bar{k}})$, where $p_k$ is the price of competitor k, $p_{\bar{k}}$ is the $(m-1)$-vector with the prices of the competitors of k, and where the notion of time is suppressed. In addition, it is assumed that $d(\cdot ,\cdot )$ is independent of permutations in its second argument, i.e., in the vector $p_{\bar{k}}$. Thus, this algorithm aims to obtain the price that maximizes own revenue compared to the competitors, that is to solve in each time step,

$$\begin{aligned} \max _{p_1} \{p_1 d(p_1,p_{\bar{1}}) - \max \{p_k d(p_k,p_{\bar{k}})\mid k = 2,\ldots ,m\}\}. \end{aligned}$$

(2)

where the competitors are indexed 1 to m (the number of competitors) and wls is indexed 1. Note that $p_1\in p_{\bar{k}}$ for $k\in \{2,\ldots ,m\}$.

The demand function is assumed to be of the form $d(x,y) = a+bx+c\sum _{k=1}^{m-1} y_k$ and the parameters a, b, and c are estimated using weighted least squares (hence the name wls). To capture different time-dependent aspects of demand, various schemes for the weighting of observations are considered and evaluated based on the Median Absolute Error of their historical demand predictions. The best weighting scheme is used in (2) to optimize the price. For this purpose, the price for competitor k in the period to come is predicted based on the median of the historical prices over some window, where the window length is chosen to minimize the Median Absolute Error of historical price predictions.

Finally, for the purpose of exploration, during the first ten periods prices are randomized to guarantee sufficient variance in the observations to estimate the demand models. In addition, when after these ten periods this competitor’s own price is constant for three subsequent periods, the prices are randomized for the next period to induce exploration.