Pareto optimality solution of the multi-objective photogrammetric resection-intersection problem

Abstract

Reconstruction of architectural structures from photographs has recently experienced intensive efforts in computer vision research. This is achieved through the solution of nonlinear least squares (NLS) problems to obtain accurate structure and motion estimates. In Photogrammetry, NLS contribute to the determination of the 3-dimensional (3D) terrain models from the images taken from photographs. The traditional NLS approach for solving the resection-intersection problem based on implicit formulation on the one hand suffers from the lack of provision by which the involved variables can be weighted. On the other hand, incorporation of explicit formulation expresses the objectives to be minimized in different forms, thus resulting in different parametric values for the estimated parameters at non-zero residuals. Sometimes, these objectives may conflict in a Pareto sense, namely, a small change in the parameters results in the increase of one objective and a decrease of the other, as is often the case in multi-objective problems. Such is often the case with error-in-all-variable (EIV) models, e.g., in the resection-intersection problem where such change in the parameters could be caused by errors in both image and reference coordinates. This study proposes the Pareto optimal approach as a possible improvement to the solution of the resection-intersection problem, where it provides simultaneous estimation of the coordinates and orientation parameters of the cameras in a two or multistation camera system on the basis of a properly weighted multi-objective function. This objective represents the weighted sum of the square of the direct explicit differences of the measured and computed ground as well as the image coordinates. The effectiveness of the proposed method is demonstrated by two camera calibration problems, where the internal and external orientation parameters are estimated on the basis of the collinearity equations, employing the data of a Manhattan-type test field as well as the data of an outdoor, real case experiment. In addition, an architectural structural reconstruction of the Merton college court in Oxford (UK) via estimation of camera matrices is also presented. Although these two problems are different, where the first case considers the error reduction of the image and spatial coordinates, while the second case considers the precision of the space coordinates, the Pareto optimality can handle both problems in a general and flexible way.

Appendix

Summary of the steps of the algorithm

Read input data

The coordinates of the points on the photo-planes:

$$x_{j}^{(i)},y_{j}^{(i)} ,\text{}j = 1, 2,\text{...}n,\text{ }i = 1,2,\text{...}m$$

The space coordinates:

$$X_{j},Y_{j},Z_{j}, j = 1, 2,\text{...}n,$$

where n is the number of points on a photo-plane, and m is the number of the photo-planes.

Defining the objective functions

• a) for the photo-planes:

  $$\begin{array}{lll} G_{\text{xy}}\left(\pi _1,\pi _2,\text{...}\pi _m\right)\\ \quad= \displaystyle\sum _{i=1}^m \sum _{j=1}^n w_{x_j}\left(x_j^{(i)}-p_x\left(\pi _i,X_j,Y_j,Z_j\right)\right)^2\\ \quad\quad{\kern6pt} +w_{y_j}\left(y_j^{(i)}-p_y\left(\pi _i,X_j,Y_j,Z_j\right)\right)^2 \end{array} $$

• b) for the space coordinates

  • use one-point intersection to express the space coordinates explicitely, see Eq. 14:

    $$\begin{array}{lll} G_{\text{XYZ}}\left(\pi _1,\pi _2,\text{...}\pi _m\right)\\ \quad= \sum\limits_{j=1}^n W_{X_j}\left(X_j^{(i)}-p_X\left(\pi _1,\pi _2,\text{...}\pi _m,x_j^{(i)},y_j^{(i)}\right)\right)^2\\ \quad\quad+W_{Y_j}\left(Y_j^{(i)}-p_Y\left(\pi _1,\pi _2,\text{...}\pi _m,x_j^{(i)},y_j^{(i)}\right)\right)^2\\ \quad\quad+W_{Z_j}\left(Z_j^{(i)}-p_Z\left(\pi _1,\pi _2,\text{...}\pi _m,x_j^{(i)},y_j^{(i)}\right)\right)^2 \end{array} $$

  • alternatively use implicit expression of the space coordinates as constraint while minimizing the adjustments of the space coordinates, see Eq. 20.

    $$\begin{array}{rll} \text{ }G_{\text{XYZ}} \left(\pi _1,\pi _2,\text{...}\pi _m\right)&=&\sum\limits_{j=1}^n W_{X_j}\Delta \text{X}_j^2+W_{Y_j}\Delta \text{Y}_j^2\\ &&+W_{Z_j}\Delta \text{Z}_j^2, \end{array} $$

with the constraints

$$\begin{array}{lll} P_X\left(\pi _i,X_j+\Delta \text{X}_j,Y_j+\Delta \text{Y}_j,Z_j+\Delta \text{Z}_j,x_j^{(i)},y_j^{(i)}\right)\\ \quad=0, j = 1\text{..}n \end{array} $$

$$\begin{array}{lll} P_Y\left(\pi _i,X_j+\Delta \text{X}_j,Y_j+\Delta \text{Y}_j,Z_j+\Delta \text{Z}_j,x_j^{(i)},y_j^{(i)}\right)\\ \quad=0, j = 1\text{..}n. \end{array} $$

Computing the dimensionless form of the conflicting objective functions

• a) Minimize $\,G_{\text {xy} }\,\text {to} \,\text {get} \,\pi _1^{(\text {xy})},\pi _2^{(\text {xy})},\text {...}\pi _m^{(\text {xy})}$

• b) Minimize $G_{\text {XYZ} }\,\text {to} \,\text {get} \,\pi _1^{(\text {XYZ})},\pi _2^{(\text {XYZ})},\text {...}\pi _m^{(\text {XYZ})}$

• c) The maximum values of the objective functions

  $$G_{\text{xymax}}=G_{\text{xy}}\left(\pi _1^{(\text{XYZ})},\pi _2^{(\text{XYZ})},\text{...}\pi _m^{(\text{XYZ})}\right)$$
  $$G_{\text{XYZmax}}=G_{\text{XYZ}}\left(\pi _1^{(\text{xy})},\pi _2^{(\text{xy})},\text{...}\pi _m^{(\text{xy})}\right)$$

• d) Compute the dimensionless forms

  $$\tilde{G}_{\text{xy}}\left(\pi _1,\pi _2,\text{...}\pi _m\right)=\frac{G_{\text{xy}}\left(\pi _1,\pi _2,\text{...}\pi _m\right)-G_{\text{xymin}}}{G_{\text{xymax}}-G_{\text{xymin}}}$$

  and

  $$\tilde{G}_{\text{XYZ}}\left(\pi _1,\pi _2,\text{...}\pi _m\right)=\frac{G_{\text{XYZ}}(\pi _1,\pi _2,\text{...}\pi _m)-G_{\text{XYZmin}}}{G_{\text{XYZmax}}-G_{\text{XYZmin}}}$$

Computing the Pareto set

• a) Set discrete values λ _k $\in$[0, 1], k $=$ 1,2, ...N

• b) Minimize the mono-objective function for all λ _k

  $$\begin{array}{rll} \tilde{G}\left(\pi _1,\pi _2,\text{...}\pi _m,\lambda _k\right)&=& \lambda _k \tilde{G}_{\text{XYZ}}\left(\pi _1,\pi _2,\text{...}\pi _m\right)\\ &&+\left(1-\lambda _k\right)\tilde{G}_{\text{xy}}\left(\pi _1,\pi _2,\text{...}\pi _m\right) \end{array} $$

  to get the Pareto-set of ( λ _k , Π1(k), Π2(k), ...Π m(k)) , k $=$ 1,2, ...N

Computing the Pareto front

• a) Set the interpolation functions: Π ₁($\lambda$), Π ₂($\lambda$)... Π _m ($\lambda$) from the discrete values

• b) Substitute these functions into the objective functions,

  $$\tilde{G}_{\text{xy}}(\lambda )=\tilde{G}_{\text{xy}}\left(\pi _1(\lambda ),\pi _2(\lambda ),\text{...}\pi _m(\lambda )\right)$$

  and

  $$\tilde{G}_{\text{XYZ}}(\lambda )=\tilde{G}_{\text{XYZ}}\left(\pi _1(\lambda ),\pi _2(\lambda ),\text{...}\pi _m(\lambda )\right)$$

  Then we get the Pareto-front represented in param- etric form: $\tilde {G}_{\rm xy}$(λ) - $\tilde {G}_{\rm XYZ}$(λ)

Selecting a single solution

• a) $\lambda$ = 0

  $$\begin{array}{lll} \textrm{we get }\tilde{G}_{\min }= \tilde{G}_{\text{xymin}}\textrm{ therefore the point} \\ \quad\textrm{ of the Pareto }-\textrm{ front for }\lambda =0\, \textrm{is}\\ \quad\quad{} \left(\tilde{G}_{\text{XYZmax}},\text{ }\tilde{G}_{\text{xymin}}\right). \end{array} $$

• b) $\lambda$ = 1

  $$\begin{array}{lll} \textrm{we get }\tilde{G}_{\min }= \tilde{G}_{\text{XYZmin}}\textrm{ therefore the point} \\ \quad{} \textrm{ of the Pareto - front - front for }\lambda =1\, \textrm{is}\\ \quad\quad \left(\tilde{G}_{\text{XYZmin}}, \tilde{G}_{\text{xymax}}\right). \end{array} $$

  Consequently to minimize the error of the coordinates of the photo-planes we should select a point of the Pareto-front represented by the parameter λ ^∗ $<<$ 1, and vica versa to minimize the error of the space coordinates one should select a point of the Pareto-front with λ ^∗ $>>$ 0.This is therefore a trade-off job for the decision maker.

• c) compute the camera parameters $\pi _i^*$ employing the selected $\lambda ^*$ as $\pi _i^*(\lambda ^*)$ for i $=$ 1,2,$\ldots$m.

Selecting the Pareto-balanced solution

This solution can minimize the overall errors of the coordinates of photo-planes as well as the space coordinates. The point of the Pareto-front representing this solution is the closest point to the ideal point (0, 0), which represents zero error for $\tilde {G}_{\rm xy}$ as well as for $\tilde {G}_{\rm XYZ}$.

• a) use L ₁norm

  $$\underset{\lambda }{\min }\,\tilde{G}_{\text{xy}}(\lambda )+\tilde{G}_{\text{XYZ}}(\lambda )\longrightarrow \lambda ^{\text{*}}$$

• b) alternatively use L ₂norm

  $$\underset{\lambda }{\min }\sqrt{\left(\tilde{(G}_{\text{xy}}(\lambda )\right)^2+\left(\tilde{G}_{\text{XYZ}}(\lambda )\right)^2}\longrightarrow \lambda ^{\text{*}}$$