School accountability: can we reward schools and avoid pupil selection?

Abstract

School accountability schemes require measures of school performance, and these measures are in practice often based on pupil test scores. It is well-known that insufficiently correcting these test scores for pupil characteristics may provide incentives for pupil selection. Building further on results from the theory of fair allocation, we show that the trade-off between reward and pupil selection is not only a matter of sufficient information. A school accountability scheme that rewards school performance will create incentives for pupil selection, even under perfect information, unless the educational production function satisfies an (unrealistic) separability assumption. We propose different compromise solutions and discuss the resulting incentives in theory. The empirical relevance of our analysis—i.e., the rejection of the separability assumption and the magnitude of the incentives in the different compromise solutions—is illustrated with Flemish data. The traditional value-added model turns out to be an acceptable compromise.

Appendix

1.1 Proof of proposition 2

A subsidy scheme can satisfy incentives for good administration and no incentives for pupil selection if and only if there exist functions $g:\mathbb {R}\times B\rightarrow \mathbb {R}$ and $h:A\rightarrow \mathbb {R}$, with g strictly increasing in its first argument, such that $f(a,b)=g(h(a),b)$, for all $x=(a,b)$ in X.

If the separability condition holds, it is possible to define a subsidy scheme s such that each school subsidy $s_{j}$ is a strictly increasing function of $h\left( a_{j}\right)$ only. Such a scheme satisfies both axioms. We show the opposite.

Consider a subsidy scheme that satisfies incentives for good administration and no incentives for pupil selection. We show that, for arbitrary administrations $a,a^{\prime }\in A$ and backgrounds $b,b^{\prime }\in B$, we have

$$\begin{aligned} f(a,b)\ge f(a^{\prime },b)\Leftrightarrow f(a,b^{\prime })\ge f(a^{\prime },b^{\prime }). \end{aligned}$$

(19)

This would indeed allow to properly define functions

• 1. $h:A\rightarrow \mathbb {R}$ with $h(a)\ge h(a^{\prime })$ if $f\left( a,b\right) \ge f\left( a^{\prime },b\right)$ for some $b\in B$, and

• 2. $g:\mathbb {R}\times B\rightarrow \mathbb {R}$ with $g(h(a),b)=f(a,b)$ for all $x=(a,b)$,

with g strictly increasing in its first argument.

We proceed by contradiction. Suppose Eq. (19) does not hold, e.g., both $f\left( a,b\right) \ge f\left( a^{\prime },b\right)$ and $f\left( a,b^{\prime }\right) <f\left( a^{\prime },b^{\prime }\right)$ are true for some $a,a^{\prime }\in A$ and $b,b^{\prime }\in B$. (It is easy to verify the other direction using the same logic.) We can use these $a,a^{\prime }\in A$ and $b,b^{\prime }\in B$ to construct four states— $(a,b),\, (a^{\prime },b),\, (a,b^{\prime })$, and $(a^{\prime },b^{\prime })$—for some school (tacitly assuming that school information remains constant for all other schools). We suppress subscripts and use f(a, b) and (with slight abuse of notation) s(a, b) to refer to the output and the subsidy of the school under consideration. Applying incentives for good administration twice, we must have

$$\begin{aligned} s(a,b)-s(a^{\prime },b)\ge 0 \quad \text {and}\quad s(a,b^{\prime })-s(a^{\prime },b^{\prime })<0. \end{aligned}$$

(20)

Applying no incentives for pupil selection twice, we obtain

$$\begin{aligned} s(a,b)=s(a,b^{\prime })\quad \text {and}\quad s(a^{\prime },b)=s(a^{\prime },b^{\prime }), \end{aligned}$$

and, subtracting both equations, we get:

$$\begin{aligned} s(a,b)-s(a^{\prime },b)=s(a,b^{\prime })-s(a^{\prime },b^{\prime }). \end{aligned}$$

(21)

Equation (20) and (21) are incompatible, a contradiction.

1.2 A derivation of the empirical subsidy schemes

The per-capita and uncorrected output schemes are straightforward. We discuss the reference administration, reference background and value added scheme. A subsidy scheme is defined as

$$\begin{aligned} s_{j}= {\texttt {1}} + {\texttt {slope}} \times (\widetilde{y}_{j}- \overline{\widetilde{y}}), \end{aligned}$$

with the slope defined by (17) for each scheme. We focus here on the difference $\widetilde{y}_{j}-\overline{\widetilde{y}}$.

We start from the empirical model

The RA models use a reference administration, say $\widetilde{a}=(\widetilde{ z}_{a},\widetilde{v},\widetilde{\beta }_{b})$, to define the hypothetical output as

$$\begin{aligned} \widetilde{y}_{j}=\overline{y}_{j}-f\left( \widetilde{a},b_{j}\right) = \overline{y}_{j}-\left( \beta _{a}^{\prime }\widetilde{z}_{a}+\widetilde{v}+ \widetilde{\beta }_{b}^{\prime }\overline{z}_{b,j}\right) . \end{aligned}$$

The average hypothetical output is equal to

$$\begin{aligned} \overline{\widetilde{y}}=\overline{y}-\left( \beta _{a}^{\prime }\widetilde{z}_{a}+ \widetilde{v}+\widetilde{\beta }_{b}^{\prime }\overline{z}_{b}\right) , \end{aligned}$$

and the difference $\widetilde{y}_{j}-\overline{\widetilde{y}}$ is indeed equal to

$$\begin{aligned} (\overline{y}_{j}-\overline{y})-\widetilde{\beta }_{b}^{\prime }(\overline{z} _{b,j}-\overline{z}_{b}). \end{aligned}$$

Starting from the same empirical model, the RB models replace $\overline{z} _{b,j}$ by a reference background $\widetilde{b}=\widetilde{z}_{b}$ to get

$$\begin{aligned} \widetilde{y}_{j}=f(a_{j},\widetilde{b})=\widehat{\beta }_{a}^{\prime } \overline{z}_{a,j}+\widehat{v}_{j}+\widehat{\beta }_{b,j}^{\prime } \widetilde{z}_{b}. \end{aligned}$$

The OLS estimate for $\widehat{v}_{j}$ is

$$\begin{aligned} \widehat{v}_{j}=\overline{y}_{j}-\widehat{\beta }_{a}^{\prime }\overline{z} _{a,j}-\widehat{\beta }_{b,j}^{\prime }\overline{z}_{b,j}, \end{aligned}$$

and we can rewrite the hypothetical output as

$$\begin{aligned} \widetilde{y}_{j}=\overline{y}_{j}-\widehat{\beta }_{b,j}^{\prime }( \overline{z}_{b,j}-\widetilde{z}_{b}). \end{aligned}$$

The average is given by

$$\begin{aligned} \overline{\widetilde{y}}_{j}=\overline{y}-\overline{\widehat{\beta } _{b,j}^{\prime }(\overline{z}_{b,j}-\widetilde{z}_{b})}, \end{aligned}$$

and the difference $\widetilde{y}_{j}-\overline{\widetilde{y}}_{j}$ indeed becomes

$$\begin{aligned} (\overline{y}_{j}-\overline{y})-\widehat{\beta }_{b,j}^{\prime }(\overline{z} _{b,j}-\widetilde{z}_{b})+\overline{\widehat{\beta }_{b,j}^{\prime }( \overline{z}_{b,j}-\widetilde{z}_{b})}. \end{aligned}$$

Finally, for the value-added (VA) model we have

$$\begin{aligned} \widetilde{y}_{j}=\widehat{\beta }_{a}^{VA\prime }\overline{z}_{a,j}+ \widehat{v}_{j}^{VA}, \end{aligned}$$

with the OLS estimate of $v_{j}^{VA}$ in (18) given by

$$\begin{aligned} \widehat{v}_{j}^{VA}=\overline{y}_{j}-\widehat{\beta }_{a}^{VA\prime } \overline{z}_{a,j}-\widehat{\beta }_{b}^{VA\prime }\overline{z}_{b,j}. \end{aligned}$$

Plugging in the OLS estimate, corrected output becomes

$$\begin{aligned} \widetilde{y}_{j}=\overline{y}_{j}-\widehat{\beta }_{b}^{VA\prime }\overline{ z}_{b,j}. \end{aligned}$$

Averaging the corrected output, we get

$$\begin{aligned} \overline{\widetilde{y}}=\overline{y}-\widehat{\beta }_{b}^{VA\prime } \overline{z}_{b}, \end{aligned}$$

and the difference $\widetilde{y}_{j}-\overline{\widetilde{y}}$ indeed reduces to

$$\begin{aligned} (\overline{y}_{j}-\overline{y})-\widehat{\beta }_{b}^{VA\prime }(\overline{z} _{b,j}-\overline{z}_{b}). \end{aligned}$$

1.3 First step probit estimates for Table 6

 First-step probit estimates for the sample selection model (model e)

Present	In period 1		In period 2
	Coeff.	$\hbox {p}>{\vert }\hbox {t}{\vert }$	Coeff.	$\hbox {p}>{\vert }\hbox {t}{\vert }$
$\hbox {Math}_{0}$	0.02	0.39	0.30	0.00
Girl	0.04	0.40	0.01	0.86
m_dutch	0.20	0.07	$-0.39$	0.00
f_dutch	0.07	0.49	$-0.01$	0.96
m_edu_sec	$-0.00$	1.00	0.16	0.05
m_edu_high	$-0.01$	0.90	0.20	0.03
m_edu_uni	$-0.09$	0.47	0.30	0.03
f_edu_sec	$-0.05$	0.53	0.07	0.42
f_edu_high	$-0.06$	0.57	0.14	0.18
f_edu_uni	$-0.23$	0.04	$-0.04$	0.74
Duo	$-0.65$	0.00	0.45	0.00
Peer	0.15	0.01	0.34	0.00
Time_math	$-0.12$	0.00	0.29	0.00
Experience	$-0.02$	0.00	0.01	0.00
Class_size	0.00	0.58	$-0.01$	0.41
Pseudo-${R}^2$		0.12		0.14
No. of observations		3578		3578

1. Constant and regional dummies included, but not reported

 First-step probit estimates for the sorting model (model f)

In a catholic school	In period 1		In period 2
	Coeff.	$\hbox {p}>{\vert }\hbox {t}{\vert }$	Coeff.	$\hbox {p}>{\vert }\hbox {t}{\vert }$
$\hbox {Math}_{0}$	0.03	0.48	0.02	0.55
Girl	0.18	0.00	0.20	0.00
m_dutch	$-0.06$	0.62	$-0.04$	0.73
f_dutch	0.03	0.80	0.00	0.97
m_edu_sec	0.06	0.49	0.04	0.63
m_edu_high	0.25	0.02	0.23	0.03
m_edu_uni	0.22	0.12	0.15	0.30
f_edu_sec	0.07	0.43	0.08	0.37
f_edu_high	0.14	0.18	0.14	0.17
f_edu_uni	0.16	0.21	0.21	0.11
Duo	$-0.08$	0.39	0.78	0.00
Peer	1.12	0.00	1.11	0.00
Time_math	$-0.16$	0.00	$-0.20$	0.00
Experience	$-0.01$	0.00	$-0.02$	0.00
Class_size	0.03	0.00	0.02	0.01
m_freemason	0.08	0.68	0.05	0.79
m_christian	0.42	0.02	0.38	0.03
m_jewish	0.23	0.67	0.46	0.40
m_islam	0.59	0.06	0.60	0.06
m_atheist	0.03	0.89	0.02	0.92
m_other	0.05	0.86	0.06	0.80
f_freemason	$-0.38$	0.04	$-0.35$	0.06
f_christian	$-0.02$	0.92	0.02	0.90
f_jewish	$-1.37$	0.01	$-1.41$	0.01
f_islam	$-0.45$	0.01	$-0.45$	0.13
f_atheist	$-0.07$	0.74	$-0.04$	0.87
f_other	$-0.67$	0.01	$-0.74$	0.00
$\hbox {Pseudo}$-$R^{2}$	0.25		0.29
No. of observations	2747		2747

1. Constant and regional dummies included, but not reported