Relationship Between Data Smoothing and the Regularization of Inverse Problems

Abstract

We investigate the practice of regularization (also termed damping) in inverse problems, meaning the use of prior information to supplement observations, in order to suppress instabilities in the solution caused by noisy and incomplete data. Our focus is on forms of regularization that create smooth solutions, for smoothness is often considered a desirable—or at least acceptable—attribute of inverse theory solutions (and especially tomographic images). We consider the general inverse problem, in its continuum limit. By deconstruction into the part controlled by the regularization and the part controlled by the data kernel, we show the general solution depends on a smoothed version of the back-projected data as well as a smoothed version of the generalized inverse. Crucially, the smoothing function that controls both is the solution to the simple data smoothing problem. We then consider how the choice of regularization shapes the smoothing function, in particular exploring the dichotomy between expressing prior information either as a constraint equation (such as a spatial derivative of the solution being small) or as a covariance matrix (such as spatial correlation falling off at a specified rate). By analyzing the data smoothing problem in its continuum limit, we derive analytic solutions for different choices of regularization. We consider four separate cases: (1) the first derivative of the solution is close to zero, (2) the prior covariance is a two-sided declining exponential, (3) the second derivative of the solution is close to zero, and (4) the solution is close to its localized average. First-derivative regularization is put forward as having several attractive properties and few, if any, drawbacks.

Appendix: Derivations of Smoothing Kernels and Covariances for Case Studies

1.1 Case 1: First-Derivative Minimization

The operator ${ \mathcal{L}} = \varepsilon {\text{d}}/{\text{d}}x$ has translational invariance, so we expect that the smoothing kernel a(x, x′) = a(x − x′) will depend only upon the separation distance (x − x′) (as also will C _h , P _h , Q _h , and R). Without loss of generality, we can set x′ = 0, so that Eq. (13) becomes

$$\left( { - \varepsilon^{2} \frac{{{\text{d}}^{2} }}{{{\text{d}}x^{2} }} + 1} \right) a(x) = \delta (x).$$

(42)

Here, we utilize the relationship that $\left( {{\text{d}}/{\text{d}}x} \right)^{\dag } = - {\text{d}}/{\text{d}}x$. The solution to this well-known 1D screened Poisson equation is

$$a\left( x \right) = \frac{{\varepsilon^{ - 1} }}{2}\exp \left( { - \varepsilon^{ - 1} \left| x \right|} \right).$$

(43)

This solution can be verified by substituting it into the differential equation:

$$\begin{aligned} \frac{{{\text{d}}a}}{{{\text{d}}x}} = - \frac{{\varepsilon^{ - 2} }}{2}\text{sgn} \left( x \right) {\text{exp}}\left( { - \varepsilon^{ - 1} \left| x \right|} \right)\quad {\text{and }}\quad \frac{{{\text{d}}^{2} a}}{{{\text{d}}x^{2} }} = \frac{{\varepsilon^{ - 3} }}{2} {\text{exp}}\left( { - \varepsilon^{ - 1} \left| x \right|} \right) - \varepsilon^{ - 2} \delta \left( x \right) \hfill \\ {\text{so}}\quad - \varepsilon^{2} \frac{{\varepsilon^{ - 3} }}{2} {\text{exp}}\left( { - \varepsilon^{ - 1} \left| x \right|} \right) - \varepsilon^{2} \left( { - \varepsilon^{ - 2} } \right) \delta \left( x \right) + \frac{{\varepsilon^{ - 1} }}{2}\exp \left( { - \varepsilon^{ - 1} \left| x \right|} \right) = \delta (x). \hfill \\ \end{aligned}$$

(44)

Here, we have relied on the fact that (d/dx)|x| = sgn(x) and $\left( {{\text{d}}/{\text{d}}x} \right) {\text{sgn}}(x) = 2\delta \left( x \right)$. Note that a(x) is a two-sided declining exponential with unit area and decay rate ɛ ⁻¹. Because of the translational invariance, the integral in Eq. (11) has the interpretation of a convolution. The solution is the observed data d(x) convolved with this smoothing kernel:

$$m\left( x \right) = a\left( x \right)*d(x).$$

(45)

The variance C _h of the prior information satisfies Eq. (15a):

$$- \sigma_{d}^{ - 2} \varepsilon^{2} \frac{{{\text{d}}^{2} }}{{{\text{d}}x^{2} }} C_{h} (x) = \delta (x).$$

(46)

This is a 1D Poisson equation, with solution

$$C_{h} \left( x \right) = \sigma_{d}^{2} \frac{{\varepsilon^{ - 2} }}{2}\left( {C_{0} - \left| x \right|} \right)\quad {\text{with}}\;C_{0} {\text{ arbitrary}}.$$

(47)

This solution can be verified by substituting it into the differential equation:

$$\begin{aligned} \frac{{{\text{d}}C_{h} }}{{{\text{d}}x}} = - \sigma_{d}^{2} \frac{{\varepsilon^{ - 2} }}{2} {\text{sgn}}\left( x \right)\quad {\text{and }}\quad \frac{{{\text{d}}^{2} C_{h} }}{{{\text{d}}x^{2} }} = - \sigma_{d}^{2} \varepsilon^{ - 2} \delta \left( x \right) \hfill \\ {\text{thus}}\quad - \sigma_{d}^{ - 2} \varepsilon^{2} \frac{{{\text{d}}^{2} }}{{{\text{d}}x^{2} }}C_{h} \left( x \right) = - \sigma_{d}^{ - 2} \varepsilon^{2} \left( { - \sigma_{d}^{2} \varepsilon^{ - 2} } \right) \delta \left( x \right) = \delta \left( x \right). \hfill \\ \end{aligned}$$

(48)

The covariance C _h (x − x′) implies that the errors associated with neighboring points of the prior information equation m(x) = 0 are highly and positively correlated, and that the degree of correlation declines with separation distance, becoming negative at large separation.

Finally, we note that the operator ${\mathcal{L}} = \varepsilon {\text{d}}/{\text{d}}x$ is not self-adjoint, so that it is not the continuous analogue of the symmetric matrix C ^−1/2 _h . As described earlier, we can construct a symmetric operator by introducing a unitary transformation. ${\mathcal{L}}$ is antisymmetric in x, but we seek a symmetric operator, so the correct transformation is the Hilbert transform, ${\mathcal{H}}$, that is, the linear operator that phase-shifts a function by π/2. It obeys the rules ${\mathcal{H}}^{\dag } = - {\mathcal{H}}$, ${\mathcal{H}}^{\dag } {\mathcal{H}} = 1$, and ${\mathcal{H}}\left( {{\text{d}}/{\text{d}}x} \right) = \left( {{\text{d}}/{\text{d}}x} \right){\mathcal{H}}$. The modified operator ${\mathcal{L}}_{\text{sa}} = \varepsilon { \mathcal{H}}{\text{d}}/{\text{d}}x$ is self-adjoint and satisfies ${\mathcal{L}}_{\text{sa}}^{\dag } {\mathcal{L}}_{\text{sa}} = {\mathcal{L}}^{\dag } {\mathcal{L}}.$

1.2 Case 2: Exponentially Decaying Covariance

For a covariance described by a two-sided declining exponential function,

$$C_{h} \left( {x - x'} \right) = \varepsilon^{ - 2} \exp \left( { - \eta \left| {x - x'} \right|} \right) = \frac{{2\varepsilon^{ - 2} }}{\eta } \frac{\eta }{2}\exp \left( { - \eta \left| {x - x'} \right|} \right).$$

(49)

By comparing Eqs. (42) and (43), we find that this prior covariance is the inverse of the operator

$${\mathcal{L}}^{\dag } {\mathcal{L}} = \frac{{\eta \varepsilon^{2} }}{2}\left( { - \eta^{ - 2} \frac{{{\text{d}}^{2} }}{{{\text{d}}x^{2} }} + 1} \right).$$

(50)

The smoothing kernel solves the equation

$$\gamma^{2} \left( { - \beta^{ - 2} \gamma^{ - 2} \frac{{{\text{d}}^{2} }}{{{\text{d}}x^{2} }} + 1} \right)a\left( x \right) = \delta \left( x \right)$$

$${\text{with}}\quad \beta^{2} = 2\eta \varepsilon^{ - 2} \quad {\text{and}}\quad \gamma^{2} = \left( {1 + \frac{{\eta \varepsilon^{2} }}{2}} \right).$$

(51)

By analogy to Eqs. (42) and (43), the smoothing kernel is

$$a\left( x \right) = \gamma^{ - 2} \frac{\beta \gamma }{2}\exp \left( { - \beta \gamma \left| x \right|} \right).$$

(52)

An operator ${\mathcal{L}}$ that reproduces the form of ${\mathcal{L}}^{\dag } {\mathcal{L}}$ given in Eq. (50) is

$${\mathcal{L}} = \lambda \left( {\eta^{ - 1} \frac{\text{d}}{\text{d}x} + 1} \right)\quad {\text{with}}\quad \lambda^{2} = \eta /2\varepsilon^{ - 2}.$$

(53)

The function P _h solves Eq. (15b), ${\mathcal{L}}^{\dag } P_{h} = \delta (x)$, which for the operator in (30) has the form of a one-sided exponential,

$$P_{h} \left( x \right) = \alpha \lambda^{ - 1} H( - x)\exp \left( {\eta x} \right).$$

(54)

Here, H(x) is the Heaviside step function. Because of the translational invariance, the inner product in Eq. (14) relating P _h to C _h is a convolution. That, together with the rule that the adjoint of a convolution is the convolution backwards in time, implies that $C_{h} \left( t \right) = P_{h} \left( { - t} \right)*P_{h} \left( t \right) = P_{h} \left( t \right){ \star }P_{h} (t)$, where ${ \star }$ signifies cross-correlation. The reader may easily verify that the autocorrelation of Eq. (31) reproduces the formula for C _h given in (25). Unfortunately, its Hilbert transform cannot be written as a closed-form expression, so no simple formula for the symmetrized form of P _h , analogous to C ^1/2_h , can be given.

1.3 Case 3: Second-Derivative Minimization

The smoothing kernel a(x) satisfies the differential equation

$$\left( {\varepsilon^{2} \frac{{\text{d}^{4} }}{{\text{d}x^{4} }} + 1} \right)a(x) = \delta (x).$$

(55)

This well-known differential equation has solution (Hetenyi 1979; see also Menke and Abbott 1989; Smith and Wessel 1990; Menke 2014)

$$a(x) = V { \exp }\left( { - \left| x \right|/\lambda } \right) \left\{ {\cos \left( {\left| x \right|/\lambda } \right) + \sin \left( {\left| x \right|/\lambda } \right)} \right\},$$

$$\lambda = \left( {2\varepsilon } \right)^{1/2} \quad {\text{and}}\quad V = \frac{{\lambda^{3} }}{{8\varepsilon^{2} }}.$$

(56)

The variance C _h of the prior information satisfies Eq. (15a):

$$\sigma_{d}^{ - 2} \varepsilon^{2} \frac{{{\text{d}}^{4} }}{{{\text{d}}x^{4} }} C_{h} (x) = \delta (x),$$

(57)

and by analogy to (47) has solution

$$C_{h} \left( x \right) = - \sigma_{d}^{2} \frac{{\varepsilon^{ - 2} }}{12}\left( {C_{0} - \left| {x^{3} } \right|} \right)\quad {\text{with}}\;C_{0} {\text{ arbitrary}}.$$

(58)

This solution can be verified by substituting it into the differential equation:

$$\frac{{{\text{d}}^{3} C_{h} }}{{{\text{d}}x^{3} }} = \sigma_{d}^{2} \frac{{\varepsilon^{ - 2} }}{2} {\text{sgn}}\left( x \right)\quad {\text{and}}\quad \frac{{{\text{d}}^{2} C_{h} }}{{{\text{d}}x^{2} }} = \sigma_{d}^{2} \varepsilon^{ - 2} \delta \left( x \right),$$

$${\text{thus}}\quad \sigma_{d}^{ - 2} \,\varepsilon^{2} \frac{{{\text{d}}^{4} }}{{{\text{d}}x^{4} }}C_{h} \left( x \right) = \sigma_{d}^{ - 2} \varepsilon^{2} \left( {\sigma_{d}^{2} \varepsilon^{ - 2} } \right) \delta \left( x \right) = \delta \left( x \right).$$

(59)

This function implies a steep drop off in covariance between neighboring points and increasingly great anticorrelation with distance.

1.4 Case 4: Damping towards Localized Average

From Eq. (37), we find that the smoothing kernel a(x) satisfies

$${\mathcal{L}}^{\dag } {\mathcal{L} }a + a = \varepsilon^{2} \left[ {\delta \left( x \right) - s\left( x \right)} \right]*\left[ {\delta \left( x \right) - s\left( x \right)} \right]*a + a = \delta \left( x \right).$$

(60)

We now make use of the fact that the operator ${\mathcal{L}}_{s} = 1 - \eta^{ - 2} \text{d}^{2} /\text{d}x^{2}$ is the inverse to convolution by s(x). Applying ${\mathcal{L}}_{s}$ twice to (37) yields the differential equation

$$\left( {1 + \varepsilon^{2} } \right)\eta^{ - 4} \frac{{{\text{d}}^{4} a}}{{{\text{d}}x^{4} }} - 2\eta^{ - 2} \frac{{{\text{d}}^{2} a}}{{{\text{d}}x^{2} }} + a = f(x)\quad {\text{with}}\quad f(x) = {\mathcal{L}}_{s} {\mathcal{L}}_{s} \delta \left( x \right).$$

(61)

We solve this equation by finding its Green function [that is, solving (39) with f(x) = δ(x)] and then by convolving this Green function by the actual f(x). This Green function can be found using Fourier transforms, with the relevant integral given by Equation 3.728.1 of Gradshteyn and Ryzhik (1980) (which needs to be corrected by dividing their stated result by a factor of 2). The result is

$$a\left( x \right) = \left( {1 - AD} \right) \delta \left( x \right) - A \left\{ {S\;\sin \left( {\eta q\left| x \right|/r} \right) - {C\text{ cos}}\left( {\eta q\left| x \right|/r} \right)} \right\} {\text{exp}}\left( { - \eta p\left| x \right|/r} \right),$$

(62)

where

$$S = \left( {\frac{\eta }{r}} \right)^{4} p\left\{ {\left( {p^{4} - q^{4} } \right) - 2q^{2} \left( {p^{2} + q^{2} } \right)} \right\},$$

$$C = \left( {\frac{\eta }{r}} \right)^{4} q\left\{ {\left( {p^{4} - q^{4} } \right) + 2p^{2} \left( {p^{2} + q^{2} } \right)} \right\},$$

$$A = \varepsilon^{2} \eta^{ - 4} \times \frac{2}{\pi }\left( {\frac{{\eta^{4} }}{{\varepsilon^{2} + 1}}} \right) \times \left( {\frac{\pi }{4uv}} \right) \times 2\left( {\frac{\eta }{r}} \right)$$

$${\text{or}}\; A = \left( {\frac{{\varepsilon^{2} }}{{\varepsilon^{2} + 1}}} \right)\left( {\frac{\eta }{uvr}} \right),$$

$$D = 4\left( {\frac{\eta }{r}} \right)^{3} pq\left( {p^{2} + q^{2} } \right),$$

$$u = \frac{{2\varepsilon \eta^{2} }}{{\varepsilon^{2} + 1}} \quad {\text{and}}\quad v = \frac{{\eta^{2} \left( {\varepsilon^{2} + 1} \right)^{1/2} }}{{r^{2} }},$$

$$r = \left( {\varepsilon^{2} + 1} \right)^{1/2} \quad {\text{and}}\quad p = \left( {\frac{r + 1}{2}} \right)^{1/2} \quad {\text{and}}\quad q = \left( {\frac{r - 1}{2}} \right)^{1/2}.$$

(63)

We determine the area under the smoothing kernel by taking the Fourier transform of (61):

$$\left( {\left( {1 + \varepsilon^{2} } \right)\eta^{ - 4} k^{4} - 2\eta^{ - 2} k^{2} + 1} \right) a(k) = 1 - 2\eta^{ - 2} k^{2} + \eta^{ - 4} k^{4}$$

(64)

and evaluating it at zero wavenumber. Thus, a(k = 0) = 1; that is, the area is unity.