Connected components of Prym eigenform loci in genus three

Abstract

This paper is devoted to the classification of connected components of Prym eigenform loci in the strata $\mathcal H(2,2)^\mathrm{odd}$ and $\mathcal H(1,1,2)$ of the bundle of Abelian differentials $\Omega {\mathcal {M}}_3$ over ${\mathcal {M}}_3$. These loci, discovered by McMullen (Duke Math J 133:569–590, 2006), are $\mathrm{GL}^+(2,\mathbb {R})$-invariant submanifolds of complex dimension 3 of $\Omega {\mathcal {M}}_g$ that project to the locus of Riemann surfaces whose Jacobian variety has a factor admitting real multiplication by some quadratic order $\mathcal {O}_D$. These algebraic varieties are not necessarily irreducible. The main result we show is that for each discriminant D the corresponding locus has one component if $D\equiv 0,4 \text { mod }8$, two components if $D\equiv 1 \text { mod }8$, and is empty if $D\equiv 5 \text { mod }8$. Our result contrasts with the case of Prym eigenform loci in the strata $\mathcal H(1,1)$ in genus 2 [studied by McMullen (Ann Math (2) 165(2):397–456, 2007)] which is connected for every discriminant D.

Appendix: A partial compactification of \(\Omega E_D(\kappa ), \; \kappa \in \{(2,2), (1,1,2)\}\)

The subvarieties $\Omega E_D(2,2)^\mathrm{odd}$ and $\Omega E_D(1,1,2)$ of $\Omega {\mathcal {M}}_3$ are not compact. In order to compute their topological and dynamical invariants (e.g. Euler characteristic, Siegel–Veech constants) it is important to embed these varieties into compact ones such that the complement are divisors. The aim of this section is to provide a partial compactification of $\Omega E_D(\kappa )$ having good properties.

1.1 Degeneration of surfaces

We first introduce some notations.

• (1) We denote by $\mathrm{Prym}_\mathrm{triple}(0,0,0)$ the set of triple tori $\{(X_j,\omega _j,P_j), j=0,1,2\}$, where $(X_j,\omega _j,P_j) \in \mathcal H(0)$, and $(X_1,\omega _1,P_1)$ and $(X_2,\omega _2,P_2)$ are isometric. We denote by $\Lambda _j$ the lattice of $\mathbb {C}$ corresponding to $(X_j,\omega _j)$, that is $\Lambda _j:=\{\int _c\omega _j, \, c \in H_1(X_j,\mathbb {Z})\}$.

  Given a natural number $D \in \mathbb {N}$ satisfying $D \equiv 0,1,4 \text { mod }8$, we define $\Omega E_D(0,0,0)$ to be the subset of $\mathrm{Prym}(0,0,0)$ consisting of triples that satisfy the following condition: given any basis $(u_0,v_0)$ of $\Lambda _0$, there exists a tuple $(w,h,t,e) \in \mathbb {Z}^4$ satisfying

  $$\begin{aligned} \mathcal {P}_{D,\mathrm {triple}}(0,0,0) \left\{ \begin{array}{l} w>0, h>0, 0 \le t < \gcd (w,h), _, \gcd (w,h,t,e)=1 \\ D=e^2+8wh \end{array} \right. \end{aligned}$$

  and a basis $(u_1,v_1)$ of $\Lambda _1$ such that $u_1$ is parallel to $u_0$ and up to a rescaling by $\mathrm{GL}^+(2,\mathbb {R})$, we have $u_0=\lambda , v_0= \imath \lambda$, $u_1=w, v_1=t+\imath h$, where $\lambda =\frac{e+\sqrt{D}}{2}$.

• (2) We denote by $\Omega E_D(2)^*$ the space of triples $(X,\omega ,W)$, where $(X,\omega )$ is a Prym eigenform in $\Omega E_D(2)$ (in particular X is a Riemann surface of genus 2), and W is a Weierstrass point of X which is not the zero of $\omega$.

• (3) We denote by $\Omega E_D(2)^{**}$ the space of tuples $(X,\omega ,W_1,W_2)$, where $(X,\omega )$ is a Prym eigenform in $\Omega E_D(2)$, and $W_1,W_2$ are two Weierstrass points, not a zero of $\omega$.

• (4) Let $\mathrm{Prym}(0^3)$ denote the set of tuples $(X,\omega ,P_0,P_1,P_2)$, where $(X,\omega ,P_0) \in \mathcal H(0)$, and $P_1,P_2 \in X$ are permuted by the (elliptic) involution fixing $P_0$. Given $D=e^2 \in \mathbb {N}$, with $e \ge 3$, we denote by $\Omega E_D(0^3)$ the subset of $\mathrm{Prym}(0^3)$ consisting of elements that satisfy the following condition: there exists $a \in \mathbb {Z}$ such that

  $$\begin{aligned} \left( \mathcal {P}_D(0^3) \right) : \quad 0< a < e/2, \, \gcd (a,e)=1 \end{aligned}$$

  and up to a rescaling by $\mathrm{GL}^+(2,\mathbb {R})$, we have $(X,\omega )\simeq (\mathbb {C}/(\mathbb {Z}\oplus \imath \mathbb {Z}),dz)$, and $P_0,P_1,P_2$ are the projections of $0, -\frac{a}{e},\frac{a}{e}$ respectively.

• (5) For any $D=e^2$, with $e \ge 3$, we denote by $\Omega \tilde{E}_D(0^2)$ the set of surfaces $(\tilde{X},\tilde{\omega })\in \mathcal H(1,1)$ that are double covers of surfaces $(X,\omega ,P_1,P_2) \in \mathcal H(0,0)$ ramified over exactly the two marked points $P_1,P_2$, where $(X,\omega , P_1,P_2)$ satisfies the following condition: there exists $a \in \mathbb {Z}$ such that

  $$\begin{aligned} \left( \mathcal {P}_D(0^2)\right) : \quad 0< a < e/2, \, \gcd (a,e)=1 \end{aligned}$$

  and up to a rescaling by $\mathrm{GL}^+(2,\mathbb {R})$, we have $(X,\omega )\simeq (\mathbb {C}/(\mathbb {Z}\oplus \imath \mathbb {Z}), dz)$, and $P_1,P_2$ are the projections of $-\frac{a}{e},\frac{a}{e}$ respectively.

1.2 Partial compactification of $\Omega E_D(2,2)$

Theorem 7.1

Let $(X,\omega )\in \Omega E_D(2,2)^\mathrm{odd}$ be a Prym eigenform and $\tau$ be the Prym involution of X. Let $\mathcal {S}$ denote the set of horizontal saddle connections joining the two zeros P and Q of $\omega$. We assume that $\mathcal {S}$ is not empty.

Let $\sigma _0$ be a saddle connection of minimal length in $\mathcal {S}$, and $t_0=|\sigma _0|$. Define $(X_t,\omega _t), \, t \in [0,t_0)$ to be the surface in the leaf of the kernel foliation through $(X,\omega )$, which is obtained by contracting $\sigma _0$ by the amount t, that is $(X_t,\omega _t)=(X,\omega )-(t,0)$. Let M be the limit surface as t tends to $t_0$. We have

• (I) If $\sigma _0$ has no twin then $M\in \Omega E_D(4)$.

• (II) If $\sigma _0$ has only one other twin $\sigma _1$ then either (II.a): $\sigma _0$ and $\sigma _1$ are permuted by the Prym involution and $M\in \Omega E_{D'}(2)^*$, with $D' \in \{D,D/4\}$, or (II.b): both $\sigma _0,\sigma _1$ are invariant under the Prym involution and $M \in \Omega \tilde{E}_D(0^2)$.

• (III) If $\sigma _0$ has two other twins $\sigma _1,\sigma _2$ then we can always assume that $\sigma _0$ is invariant under $\tau$. Set $\eta =\sigma _1*(-\sigma _2)$, then either (III.a): $\eta$ is a separating curve and $M \in \Omega E_{D,\mathrm triple}(0,0,0)$, or (III.b): $\eta$ is a non-separating curve and $M \in \Omega E_D(0^3)$.

Note that $\sigma _0$ cannot have more than two twins, and Cases (II.b) and (III.b) can only occur when D is a square.

Remark 7.2

• Since $\tau$ permutes the zeros of $\omega$, $\tau (\sigma _0)$ is also an element of $\mathcal {S}$ which has the same length as $\sigma _0$. Thus if $\sigma _0$ has no twin then $\sigma _0$ must be invariant under $\tau$.

• If $\sigma _0$ and $\sigma _1$ are two twins saddle connections in $\mathcal {S}$ both invariant by $\tau$, then the simple closed curve $\sigma _0*(-\sigma _1)$ must be non-separating (see Lemma 6.5), and consequently D must be a square (see [9, Lem. 9.2(2)]).

• Since $\tau$ induces a permutation of order two on any family of twins saddle connections, if ${\mathcal {S}}$ contains three twins saddle connections, then one of them must be invariant under $\tau$.

Proof

Cases (I) follows from Proposition 5.5. Cases (II.a), and (III.a) were actually proved in [9, Theorem 9.1]. We will only give the proofs of the remaining cases.

Case (II.b) $\sigma _0$ and $\sigma _1$ are twins and both are invariant under $\tau$. Applying the cutting-gluing procedure described in Lemma 6.5 to $\eta :=\sigma _0\cup \sigma _1$, we get a surface $(X',\omega ') \in \mathcal H(1,1)$ which is a double cover of a torus $(X'',\omega '')$. Along this process, P (resp. Q) gives rise to two points $P''_1, P''_2$ (resp. $Q''_1,Q''_2$) in $X''$, and $\eta$ gives rise to two geodesic segments $\eta ''_1, \eta ''_2$, where $\eta ''_i$ joins $P''_i$ to $Q''_i$. By convention, the double cover from $X'$ to $X''$ is ramified over $P''_1$ and $Q''_2$. As $(X,\omega )$ moves in the leaf of the kernel foliation, $(X'',\omega '')$ and $P''_1,P''_2$ are fixed, only $\eta ''_i$ vary. Thus, as t tends to $t_0$, $\eta ''_i$ shrinks to a point, and $(X',\omega ')$ converges in $\mathcal H(1,1)$ to the double cover of $(X'',\omega '')$ ramified over $P''_1$ and $P''_2$. All we need to show is that $(X'',\omega '',P''_1,P''_2)$ satisfies the conditions in $(\mathcal {P}_D(0^2))$.

We first seek a symplectic basis of $H_1(X,\mathbb {Z})^-$. Let $\gamma ''$ be a geodesic segment of minimal length in $X''$ joining $P''_1$ to $P''_2$. The pre-image of $\gamma ''$ in $X'$ consists of two segments, one of them joins a zero of $\omega '$ to a regular point both of which correspond to the zero P of $\omega$. Let $\gamma '$ denote this segment (see Fig. 14). It follows that $\gamma '$ corresponds to a simple closed curve $\gamma$ on X which contains P. By construction, we have $\langle \gamma ,\eta \rangle =\pm 1$ in $H_1(X,\mathbb {Z})$. Note that $\tau (\gamma )$ is a saddle connection joining Q to itself, and $\tau (\gamma )$ is homologous to $-\gamma$. Thus $\gamma \in H_1(X,\mathbb {Z})^-$ as an element of the homology group.

Fig. 14

Finding a symplectic basis adapted to $\gamma ''$, $(X',\omega ')$ is the double cover of $(X'',\omega '')$ ramified over $P''_1$ and $Q''_2$, $(X,\omega )$ is obtained from $(X',\omega ')$ by slitting along $\eta _1,\eta _2$, then identifying the left side of $\eta _1$ with the right side of $\eta _2$, and the right side of $\eta _1$ with the left side of $\eta _2$

Let $(\alpha '',\beta '')$ be a basis of $H_1(X'',\mathbb {Z})$ which is represented by a pair of simple closed curves disjoint from $\gamma ''\cup \eta ''_1\cup \eta ''_2$. The pre-image of $\alpha ''$ (resp. of $\beta ''$) in $X'$ corresponds to two simple closed curves $\alpha _1,\alpha _2$ (resp. $\beta _1,\beta _2$) in X which satisfy $\tau (\alpha _1)=-\alpha _2$ (resp. $\tau (\beta _1)=-\beta _2$). Set $\alpha =\alpha _1+\alpha _2,\beta =\beta _1+\beta _2$, then $\{\alpha ,\beta , \gamma , \eta \}$ is a symplectic basis of $H_1(X,\mathbb {Z})^-$. Since $\omega (\eta )=0$, the arguments in Lemma 6.2 show that up to a rescaling by $\mathrm{GL}^+(2,\mathbb {R})$, we can assume that $\omega (\alpha )=2, \omega (\beta )= 2\imath$, and $\omega (\gamma ) \in \mathbb {Q}+\imath \mathbb {Q}$.

Now, observe that we have $\omega (\alpha )=2\omega ''(\alpha ''), \omega (\beta )=2\omega ''(\beta '')$, and $\omega (\gamma )=\omega ''(\gamma '')$. Thus $(X'',\omega '')$ is the standard torus $(\mathbb {C}/(\mathbb {Z}\oplus \imath \mathbb {Z}),dz)$. Since $\omega ''(\gamma '') \in \mathbb {Q}\oplus \imath \mathbb {Q}$, we can find a basis $(\hat{\alpha }'', \hat{\beta }'')$ of $H_1(X'',\mathbb {Z})$ such that $\hat{\alpha }''$ and $\gamma ''$ are parallel. Applying again a matrix in $\mathrm{SL}(2,\mathbb {Z})$, we can assume that $\omega ''(\hat{\alpha }'')=1, \omega ''(\hat{\beta }'')=\imath$, and $\omega ''(\gamma '')=x$, with $x \in \mathbb {R}_{>0}$. Let $(\hat{\alpha },\hat{\beta },\gamma ,\eta )$ be the symplectic basis of $H_1(X,\mathbb {Z})^-$ with $(\hat{\alpha },\hat{\beta })$ arising from $(\hat{\alpha }'',\hat{\beta }'')$. Note that in this basis, the restriction of the intersection form is given by $\left( {\begin{matrix} 2J &{} 0 \\ 0 &{} J \end{matrix}}\right)$. As an element of $(H_1(X,\mathbb {Z})^-)^*, \, \omega$ corresponds to the vector $(2,2\imath ,x,0)$. By [7, Prop. 4.2], there exists a unique generator T of $\mathcal {O}_D$ which is given by a matrix of the form $\left( {\begin{matrix} e &{} 0 &{} a &{} b\\ 0 &{} e &{} c &{} d \\ 2d &{} -2b &{} 0 &{} 0 \\ -2c &{} 2a &{} 0 &{} 0 \end{matrix}} \right)$ with $(a,b,c,d,e)\in \mathbb {Z}^5, \gcd (a,b,c,d,e)=1$, such that $\omega \cdot T=\lambda \omega$ and $\lambda >0$. The condition $\omega \cdot T =\lambda \omega$ implies that $b=c=d=0$, $\lambda =e$, and $x=2a/e$. It follows that T satisfies the equation $T^2=eT$. Since T is a generator of $\mathcal {O}_D$, we must have $D=e^2$. Note that $\gamma ''$ is an embedded segment, therefore we must have $0<x <1$, which implies that $0< a < e/2$. All the conditions in $(\mathcal {P}_D(0^2))$ are then fulfilled.

Case (III.b) $\sigma _0$ is invariant under $\tau$ and has two other twins $\sigma _1,\sigma _2$ satisfying $\eta :=\sigma _1*(-\sigma _2)$ is a non-separating curve. We first show that $\sigma _1,\sigma _2$ are exchanged by $\tau$. Suppose that it is not the case, then both $\sigma _1,\sigma _2$ are invariant under $\tau$. Applying the cutting-gluing procedure described in Lemma 6.6 to $\sigma _0$ and $\sigma _1$, we then get a surface $(X',\omega ')\in \mathcal H(1,1)$ together with an involution $\tau '$ induced by $\tau$. By construction, $\sigma _2$ becomes a saddle connection in $X'$ joining the two zeros of $\omega '$. The arguments in Lemma 6.6 actually show that the hyperelliptic involution $\iota$ of $X'$ sends $\sigma _2$ into another saddle connection $\sigma '_2$. Since $\sigma '_2$ must correspond to a saddle connection of X, we derive that $\sigma _0$ has three other twins, which is impossible. Thus we can conclude that $\sigma _1,\sigma _2$ are exchanged by $\tau$.

Apply now the cutting-gluing procedure described Lemma 6.3 (see also Fig. 9) to the family $\{\sigma _0,\sigma _1,\sigma _2\}$, we obtain a flat torus $(X',\omega ')$ with three geodesic segments $\eta '_0,\eta '_1,\eta '_2$ which are parallel and have the same length. Let $P'_i,Q'_i$ be the endpoints of $\eta _i$, where $P'_i$ (resp. $Q'_i$) corresponds to P (resp. to Q). By construction, $\tau$ induces an involution $\tau '$ of $X'$ that leaves $\eta '_0$ invariant and exchanges $\eta '_1,\eta '_2$. Let $\hat{P}_i$ be the midpoint of $\eta '_i$. Note that $\tau '$ fixes $\hat{P}_0$ and exchanges $\hat{P}_1,\hat{P}_2$.

As $(X,\omega )$ moves in the leaf of the kernel foliation, $(X',\omega ')$ and $\tau '$ are fixed, only $\eta '_i$ vary. In particular, as t tends to $t_0$, $\eta '_i$ shrinks to the point $\hat{P}_i$. Thus the limit surface belongs to $\mathrm{Prym}(0^3)$. All we need to show is that $(X',\omega ',\hat{P}_0,\hat{P}_1,\hat{P}_2)$ satisfies the conditions in $(\mathcal {P}_D(0^3))$. For this purpose, we first need to specify a symplectic basis of $H_1(X,\mathbb {Z})^-$.

Set $\eta _i:=\sigma _0*(-\sigma _i), \, i=1,2$. Observe that $\tau (\eta _1)=-\eta _2$ and $\tau (\eta _2)=-\eta _1$. Thus $\hat{\eta }:=\eta _1+\eta _2 \in H_1(X,\mathbb {Z})^-$. Let $\gamma '_1$ be a geodesic segment of minimal length from $\hat{P}_1$ to $\hat{P}_0$. Let $\gamma '_2=\tau '(\gamma '_1)$. Reconstruct $(X,\omega )$ from $(X',\omega ')$ (see Fig. 9), we see that $\gamma '_1$ and $\gamma '_2$ correspond respectively to two simple closed geodesics $\gamma _1,\gamma _2$ that are exchanged by $\tau$. Let $\hat{\gamma }=\gamma _1+\gamma _2$, we then have $\hat{\gamma } \in H_1(X,\mathbb {Z})^-$, and $\langle \hat{\gamma },\hat{\eta }\rangle =2$ (up to a convenient choice of orientations).

Let $\alpha ,\beta$ be a basis of $H_1(X',\mathbb {Z})$ which can be represented by simple closed curves disjoint from $\eta '_0\cup \eta '_1\cup \eta '_2\cup \gamma '_1\cup \gamma '_2$. Since $\tau '$ is an elliptic involution of $X'$, we have $\tau '(\alpha )=-\alpha$ and $\tau '(\beta )=-\beta$ in $H_1(X',\mathbb {Z})$. It follows that $\alpha$ and $\beta$ correspond to two elements of $H_1(X,\mathbb {Z})^-$ which satisfy $\langle \alpha ,\beta \rangle =1$. Thus $\{\alpha ,\beta ,\hat{\gamma },\hat{\eta }\}$ is a symplectic basis of $H_1(X,\mathbb {Z})^-$, in which the restriction of the intersection form is given by the matrix $\left( {\begin{matrix} J &{} 0 \\ 0 &{} 2J \end{matrix}} \right)$.

Since $\omega (\hat{\eta })=0$, the arguments in Lemma 6.2 show that up to a rescaling by $\mathrm{GL}^+(2,\mathbb {R})$, we can assume that $\omega (\alpha )=1, \omega (\beta )= \imath$, and $\omega (\hat{\gamma }) \in \mathbb {Q}+\imath \mathbb {Q}$. We can then choose a basis $(\hat{\alpha },\hat{\beta })$ of $H_1(X',\mathbb {Z})$ such that $\hat{\alpha }$ and $\gamma '_i$ are parallel. Note that $\{\hat{\alpha },\hat{\beta },\hat{\gamma },\hat{\eta }\}$ is also a symplectic basis of $H_1(X,\mathbb {Z})^-$. Applying an appropriate element of $\mathrm{SL}(2,\mathbb {Z})$, we can assume that $\omega$ corresponds to a vector $(1,\imath ,x,0)$, with $x \in \mathbb {R}_{>0}$, with respect to this basis. The rest of the proof then follows from the same arguments as Case (II.b). $\square$

1.3 Partial compactification of $\Omega E_D(1,1,2)$

Theorem 7.3

Let $(X,\omega ) \in \Omega E_D(1,1,2)$ be a Prym eigenform in $\mathcal H(1,1,2)$, and $\tau$ be the Prym involution of X. We denote by Q the double zero of $\omega$, and by $R_1,R_2$ the simple zeros. Let $\mathcal {S}$ denote the set of horizontal saddle connections with distinct endpoints in X. We assume that $\mathcal {S}$ is not empty.

Let $\sigma _0$ be a saddle connection of minimal length in $\mathcal {S}$, and $t_0=|s_0|$. Note that we can always assume that $R_1$ is an endpoint of $\sigma _0$. For any $t \in [0,t_0)$, let $(X_t,\omega _t)$ be the surface in the kernel foliation leaf through $(X,\omega )$ which is obtained by reducing the length of $\sigma _0$ by the amount t, that is $(X_t,\omega _t)=(X,\omega )-(t,0)$. Let M be the limit surface as t tends to $t_0$. We have

• (I) If $\sigma _0$ joins $R_1$ to Q and has no twin nor double-twin then $M\in \Omega E_D(4)$.

• (II) If $\sigma _0$ joins $R_1$ to Q and has a double-twin then $M \in \Omega \tilde{E}_D(0^2)$.

• (III) If $\sigma _0$ joins $R_1$ to Q and has a twin $\sigma _1$, set $\eta :=\sigma _0*(-\sigma _1)$, then either (III.a): $\eta$ is separating and $M\in \Omega E_{D,\mathrm triple}(0,0,0)$, or (III.b): $\eta$ is non-separating and $M \in \Omega E_D(0^3)$

• (IV) If $\sigma _0$ joins $R_1$ to $R_2$, then either (IV.a): $\sigma _0$ has no twin and $M \in \Omega E_D(2,2)^\mathrm{hyp}$, or (IV.b): $\sigma _0$ has a twin and $M\in \Omega E_{D'}(2)^{**}$ with $D'\in \{D, D/4\}$.

Note that Cases (II) and (III.b) can only occur when D is a square.

Proof

Case (I) is actually proved in Proposition 5.5. We will give a sketch of proof for the remaining cases.

Case (II) $\sigma _0$ has a double-twin $\sigma _1$. Recall that $\sigma _1$ joins $R_1$ to $R_2$ and $\omega (\sigma _1)=2\omega (\sigma _0)$. Note also that $\sigma _1$ is unique. Set $\eta =\sigma _0*\tau (\sigma _0)*(-\sigma _1)$. By Lemma 6.5, we know that $\eta \in H_1(X,\mathbb {Z})^-$ is non-separating and $\omega (\eta )=0$. Applying the cutting-gluing procedure described in Lemma 6.7 to the curve $\eta$, we obtain a surface $(X',\omega ') \in \mathcal H(1,1)$ together with two geodesic segments $\eta _1,\eta _2$ corresponding to $\eta$. The midpoints of $\eta _1$ and $\eta _2$ are the zeros of $\omega '$. The arguments in Lemma 6.7 show that as $(X,\omega )$ moves in a leaf of the kernel foliation, $(X',\omega ')$ is fixed, only $\eta _1,\eta _2$ vary. Thus, as t tends to $t_0$, $\eta _i$ shrinks to a zero of $\omega '$. Hence the limit surface is $(X',\omega ')$.

By construction, $\tau$ induces an involution $\tau '$ of $X'$ which has two fixed points. Therefore $\tau '$ is not the hyperelliptic involution of $X'$. It follows that $(X',\omega ')$ is a double cover of a flat torus $(X'',\omega '')$ ramified over two points $P''_1, P''_2$. We can now use the same arguments as in Theorem 7.1 Case (II.b) to conclude that $(X',\omega ') \in \Omega \tilde{E}_D(0^2)$.

Case (III.a) $\sigma _0$ has a twin $\sigma _1$ and $\eta =\sigma _0*(-\sigma _1)$ is separating. Applying the cutting-gluing procedure in Lemma 5.7 to $\eta$ and $\tau (\eta )$, we get a triple of tori $\{(X_j,\omega _j), j=0,1,2\}$. The Prym involution $\tau$ induces an involution on this family of tori which leaves $(X_0,\omega _0)$ invariant and exchanges $(X_1,\omega _1)$ and $(X_2,\omega _2)$. By this construction, $\eta$ and $\tau (\eta )$ give rise to a slit $\eta _j$ on $X_j$ such that $|\eta _0|=2|\eta _1|=2|\eta _2|=2|\sigma _0|$. As $(X,\omega )$ moves in a leaf of the kernel foliation, the family $\{(X_j,\omega _j), \, j=0,1,2\}$ is fixed, only the slits $\eta _j$ vary. Thus as t tends to $t_0$, the slit $\eta _j$ shrinks to a point $P_j\in X_j$. Clearly, the triple of marked flat tori $\{(X_j,\omega _j,P_j), \, j=0,1,2\}$ belongs to $\mathrm {Prym}_\mathrm{triple}(0,0,0)$. We can now use the arguments in [9, Lem. 11.2] to conclude that $\{(X_j,\omega _j,P_j), \, j=0,1,2\} \in \Omega E_{D,\mathrm triple}(0,0,0)$.

Case (III.b) $\sigma _0$ has a twin $\sigma _1$ and $\eta =\sigma _0*(-\sigma _1)$ is non-separating. Applying the cutting-gluing procedure in Lemma 6.4 to the curves $\eta$ and $\tau (\eta )$, we obtain a flat torus $(X',\omega ')$ together with three slits $\eta '_0,\eta '_1,\eta '_2$ such that $|\eta '_0|=2|\eta '_1|=2|\eta '_2|$. In Fig. 10, $\eta '_0,\eta '_1,\eta '_2$ are respectively the segments $\overline{R''_1R''_2}, \overline{R'_1Q'_1}, \overline{R'_2Q'_2}$. Note that $\tau$ induces an elliptic involution of $X'$ that leaves $\eta '_0$ invariant and exchanges $\eta '_1,\eta '_2$.

Let $Q''$ denote the midpoint of $\eta '_0$, and $Q'_1,Q'_2$ be respectively the endpoint of $\eta '_1$ and $\eta '_2$ that correspond to the double zero Q of $\omega$. Note that $Q''$ also corresponds to Q. As $(X,\omega )$ moves in a leaf of the kernel foliation, $(X',\omega '), Q'',Q'_1,Q'_2$ are fixed, only the slits $\eta '_i$ vary. When t tends to $t_0$, $\eta '_0$ shrinks to $Q''$, $\eta '_1$ and $\eta '_2$ shrink to $Q'_1$ and $Q'_2$ respectively. Thus the limit surface is $(X',\omega ',Q'',Q'_1,Q'_2) \in \mathrm{Prym}(0^3)$. Using similar arguments as the proof of Theorem 7.1 Case (III.b), we get that $(X',\omega ',Q'',Q'_1,Q'_2) \in \Omega E_D(0^3)$.

Case (IV.a) $\sigma _0$ joins $R_1$ to $R_2$ and has no twin. The arguments in Proposition 5.5 show that we can collide $R_1$ and $R_2$ by collapsing $\sigma _0$. Therefore, as t tends to $t_0$, $(X_t,\omega _t)$ converges (in $\Omega \mathfrak {M}_3$) to a surface $(\hat{X},\hat{\omega })$ in $\mathcal H(2,2)$. Since the set of Prym eigenforms is closed in $\Omega \mathfrak {M}_g$ (see [14]), we have $(\hat{X},\hat{\omega }) \in \Omega E_D(2,2)$.

Let $\hat{\tau }$ be the Prym involution of $\hat{X}$. Recall that the Prym involution $\tau$ on X exchanges $R_1,R_2$. Since $R_1$ and $R_2$ collide in the limit, $\hat{\tau }$ fixes the resulting double zero of $\hat{\omega }$. Thus $\hat{\tau }$ fixes both zeros of $\hat{\omega }$. It follows that $(\hat{X},\hat{\omega })$ is the standard orienting double cover of a (meromorphic) quadratic differential in $\mathcal {Q}(1^2,-1^2)$. But it is well-known that such double covers belong to the component $\mathcal H^\mathrm{hyp}(2,2)$ (see [6]). Therefore, we can conclude that $(\hat{X},\hat{\omega })$ belongs to $\Omega E_D(2,2)^\mathrm{hyp}$. By Proposition 2.3, we also know that $(\hat{X},\hat{\omega })$ is a unramified double cover of an eigenform in $\mathcal H(2)$. In particular, the $\mathrm{GL}^+(2,\mathbb {R})$-orbit of $(\hat{X},\hat{\omega })$ is closed.

Case (IV.b) $\sigma _0$ joins $R_1$ to $R_2$ and has a twin $\sigma _1$ (which also joins $R_1$ to $R_2$). Let $\eta :=\sigma _0*(-\sigma _1)$. $\square$

Claim 8

$\eta$ is non-separating.

Proof

Assume that $\eta$ is separating. By cutting along $\eta$, then closing up the boundaries of the resulting components, we obtain a slit torus $(X_1,\omega _1)$ with a slit $\eta _1$, and a surface $(X_2,\omega _2)\in \mathcal H(2)$ with a marked geodesic segment $\eta _2$. The Prym involution $\tau$ induces the involutions $\tau _1$ on $X_1$ and $\tau _2$ on $X_2$. Observe that $\tau _1$ leaves $\eta _1$ invariant.

Since $X_1$ is an elliptic curve, $\tau _1$ must have four fixed points, three of which are not contained in $\eta _1$. Since $X_1{\setminus } \eta _1$ is a subsurface of X, we derive that those three fixed points of $\tau _1$ are also fixed points of $\tau$. Recall that by definition, $\tau$ has four fixed points, and one of which must be the double zero of $\omega$. Therefore, the double zero of $\omega$ (which is also the double zero of $\omega _2$) is the unique fixed point of $\tau$ in $X_2{\setminus } \eta _2$. It follows that $\tau _2$ has exactly two fixed points: the unique zero of $\omega _2$ and the midpoint of $\eta _2$. In particular, $\tau _2$ is not the hyperelliptic involution $\iota _2$ of $X_2$. On the other hand, the unique zero of $\omega _2$ is fixed by both of $\tau _2$ and $\iota _2$, and the differentials of $\tau _2$ and $\iota _2$ at this point are both equal to $-\mathrm {Id}$. Therefore, we must have $\tau _2=\iota _2$, which is a contradiction. We can now conclude that $\eta$ is non-separating. $\square$

Claim 9

$\sigma _0$ and $\sigma _1$ are exchanged by $\tau$.

Proof

Since $\eta$ is non-separating, cutting X along $\eta$, then closing up the boundary of the resulting surface, we obtain a surface $(X',\omega ') \in \mathcal H(2)$ with to marked geodesic segments $\eta _1,\eta _2$ arising from $\eta$. Note that $\eta _1$ and $\eta _2$ are parallel and have the same length.

By construction, $\tau$ induces an involution $\tau '$ on $X'$. Assume that both $\sigma _0$ and $\sigma _1$ are invariant under $\tau$. A careful inspection on the action of $\tau$ in a neighborhood of $\eta$ shows that $\tau '$ maps $\eta _1$ to $\eta _2$. Recall that $\tau$ has four fixed points in X, two of which are contained in $\eta$ (i.e. the midpoints of $\sigma _0$ and $\sigma _1$). It follows that $\tau '$ has only two fixed points, which correspond to the fixed points of $\tau$ not contained in $\eta$. We derive in particular that $\tau '$ is not the hyperelliptic involution of $X'$. But since we have ${\tau '}^*\omega '=-\omega '$, the same argument as in the previous Claim shows that we also get a contradiction. $\square$

Fig. 15

Finding a symplectic basis for $H_1(X,\mathbb {Z})^-$: $(X,\omega )$ is obtained from $(X',\omega ')$ by identifying the left side of $\eta _1$ with the right side of $\eta _2$, and the right side of $\eta _1$ with the left side of $\eta _2$. Note that $\beta _{21}$ and $\beta _{22}$ are homologous in $X'$ but not in X. The Prym involution of X is induced by the hyperelliptic involution of $X'$. A symplectic basis of $H_1(X,\mathbb {Z})^-$ is given by $\{\alpha _1,\beta _1,\alpha _2,\beta _{21} +\beta _{22}\}$

Let $(X',\omega '), \eta _1,\eta _2$ be as in Claim 9. Let $P_1,P_2$ denote respectively the midpoints of $\eta _1,\eta _2$. A direct consequence of Claim 9 is that both $\eta _1,\eta _2$ are invariant under the involution $\tau '$ on $X'$ which is induced by $\tau$. It follows that $\tau '$ has exactly 6 fixed points: the four fixed points of $\tau$ together with $P_1$ and $P_2$. Thus $\tau '$ is the hyperelliptic involution of $X'$.

We can now use the arguments in [9, Lem. 8.7] to prove that $(X',\omega ')$ belongs to $\Omega E_{D'}(2)$, with $D' \in \{D, D/4\}$. Let $(\alpha _1,\beta _1,\alpha _2,\beta _2)$ be the symplectic basis of $H_1(X,\mathbb {Z})^-$, where $\beta _2=\beta _{21}+\beta _{22}$, as shown in Fig. 15. We can view $\omega$ as a vector in $H^1(X,\mathbb {C})^-\simeq \mathbb {C}^4$. Let (x, y, u, v) be the row vector corresponding to $\omega$ with respect to this basis.

Note that the intersection form in $H_1(X,\mathbb {Z})^-$ is given by the matrix $\left( {\begin{matrix} J &{} 0 \\ 0 &{} 2J \end{matrix}}\right)$. There exists a generator T of $\mathcal {O}_D \subset \mathrm{End}(\mathrm{Prym}(X,\tau ))$ whose matrix in the basis $\{\alpha _1,\beta _1,\alpha _2,\beta _2\}$ has the form

$$\begin{aligned} T=\left( {\begin{matrix} e &{}\quad 0 &{}\quad 2a &{}\quad 2b \\ 0 &{}\quad e &{}\quad 2c &{}\quad 2d \\ d &{}\quad -b &{}\quad 0 &{}\quad 0 \\ -c &{}\quad a &{}\quad 0 &{}\quad 0 \\ \end{matrix}} \right) \end{aligned}$$

with $(a,b,c,d,e) \in \mathbb {Z}^5$ and $\gcd (a,b,c,d,e)=1$, such that $\omega \cdot T =\lambda \omega$, with $\lambda \in \mathbb {R}_{>0}$. Observe that by construction, we have $\omega (\beta _2)=\omega (\beta _{21})+\omega (\beta _{22})=2\omega (\beta _{21})$ (since we have $\beta _{22}=-\tau (\beta _{21})$ and $\tau ^*\omega =-\omega$). Consider now $\alpha _1,\beta _1,\alpha _2,\beta _{21}$ as 1-cycles on $X'$. Observe that $\{\alpha _1,\beta _1,\alpha _2,\beta _{21}\}$ is a symplectic basis of $H_1(X',\mathbb {Z})$. Again, by construction, we have

$$\begin{aligned} \left\{ \begin{array}{l} \omega '(\alpha _1)=\omega (\alpha _1)=x,\\ \omega '(\beta _1)=\omega (\beta _1)=y,\\ \omega '(\alpha _2)= \omega (\alpha _2)=u,\\ \omega '(\beta _{21})=\omega (\beta _{21})=1/2\omega (\beta _2)=v/2, \end{array} \right. \end{aligned}$$

which means that $\omega '$ corresponds to the row vector (x, y, u, v / 2) in $H^1(X',\mathbb {C})$. Let $T'$ be the following matrix

$$\begin{aligned} T':=\left( {\begin{matrix} e &{}\quad 0 &{}\quad 2a &{}\quad b\\ 0 &{}\quad e &{}\quad 2c &{}\quad d \\ d &{}\quad -b &{}\quad 0 &{}\quad 0\\ -2c &{}\quad 2a &{}\quad 0 &{}\quad 0\\ \end{matrix}} \right) . \end{aligned}$$

Note that $T'$ is self-adjoint with respect to the intersection form of $H_1(X',\mathbb {Z})$. It is straightforward to check that the condition $\omega \cdot T=\lambda \omega$ is equivalent to $\omega '\cdot T'=\lambda \omega '$. Hence $(X',\omega ')$ is an eigenform in $\mathcal H(2)$. Thus $(X',\omega ') \in \Omega E_{D'}(2)$, for some discriminant $D'$. Note that $\mathcal {O}_{D'}$ must be generated by a matrix $T''\in \mathrm{End}(\mathbf{Jac}(X'))$ such that $T'=kT''$, with $k\in \mathbb {Z}$. Thus we must have $k=\gcd (2a,2c,b,d,e)$. Since we have $\gcd (a,b,c,d,e)=1$, it follows that $k\in \{1,2\}$, which implies that $D' \in \{D,D/4\}$.

To conclude, we finally remark that as $(X,\omega )$ moves in a leaf of the kernel foliation, $(X',\omega ')$, and $P_1,P_2$ are fixed. Hence, as t tends to $t_0$, $\eta _1$ and $\eta _2$ shrink to $P_1$ and $P_2$, and the limit surface belongs to $\Omega E_{D'}(2)^{**}$. The proof of Theorem 7.3 is now complete.