Light Field Estimation in the Ultraviolet Spectrum

Abstract

Ultraviolet cameras are becoming well used, with their applications in botany, dermatology, and recently in photography. In this paper, we develop a novel method of light field estimation from an ultraviolet image. Our UV light field imaging model is obtained by exploiting the radiometry, the optic, and the acquisition geometry. First, we develop an optic simulation model for an UV camera with a thin lens. That model permits to reconstruct a scene image from the correspondent light field. Then, we define a variational formalism that integrates an image of a scene acquired by an UV camera, the depth map of that scene, and the optic simulation model in order to estimate the light field. The experimental results show that it is possible to estimate the light field in the UV spectrum with accuracy.

1 Introduction

The human eye perceives a small portion of the light spectrum between 380 nm and 780 nm, called the visible spectrum. To go beyond the visible spectrum, the ultraviolet (UV) cameras are well placed to provide new information about the visible scene. The UV spectrum is separated into three bands: the ultraviolet A between 380 nm and 315 nm, the ultraviolet B between 315 nm and 280 nm, and the ultraviolet C between 280 nm and 100 nm. The UV imaging may be used in various applications including dermatology [1,2,3], botany [4, 5], criminology [6], biology [7, 8], volcanology [9, 10] and for artistic purposes. The UV imaging starts with the acquisition of 2D images by using an UV camera and UV light source. The choice of the acquisition system depends on the target application and therefore the target UV band. Recently, light field cameras were introduced to the consumer market as a technology facilitating novel user experiences [12]. However, current cameras have been manufactured for sensing light field in visible spectrum [11, 13]. It seems that there are no available cameras, models or methods allowing the acquisition of light field in the UV spectrum. In this paper, we propose a method allowing the light field acquisition in the UV spectrum by using an UV camera. Such a light field can be applied in all the domains cited previously in order to reconstruct images where a user can choose the plane of focus position [14] or to allow 3D visualisation of the scene in the UV spectrum [15, 16]. This paper does not focus on the design of a special camera for the UV light field acquisition, but the inference of a light field by using a 2D UV image and a depth map captured by a depth camera. The remainder of this paper is organised as follows. Firstly, the general light field representation and its scene depth encoding are explained in Sect. 2. Secondly, the optic simulation model of the UV thin lens camera is introduced in Sect. 3. Thirdly, the light field inference method from the UV image and the depth map is described in Sect. 4. Finally, the results of light field inference and images reconstructed by using the optic simulation model are shown and discussed in Sect. 5.

Fig. 1.

Projection of UV light rays inside an UV camera.

2 What is a Light Field?

A light field represents all light rays propagating within a scene [17]. For a wavelength $\lambda$, and at an instant $\tau$ a light ray is parametrised by four coordinate (a, b, c, d) under the assumption that the radiance is constant along the light ray [18]. These four coordinate can be represented by the ray intersection coordinate with two parallel planes spaced by a distance $d_{0}$. In Fig. 1, a light ray intersects the plane AB at coordinate (a, b) and crosses the plane CD at coordinate (c, d). This representation of the light field only considers the UV light rays reflected by an object. This consideration imposed some limitation on the UV bands studied in this paper. In fact, the smaller the wavelength the more it is scattered by the medium and the object of the scene, as well as the optical device of a UV camera. Therefore, the UV light in band B and C is mostly scattered and not reflected. Hence, we limit this study to the ultraviolet A band of the UV spectrum which is less scattered and therefore the light amount reaching the camera is higher than the two other bands. The signal-to-noise ratio is also higher. A light ray reflected from a scene point, carries a radiance R emitted by this point. This radiance corresponds to the intensity of a light ray in the UV spectrum which is defined by the plenoptic function L(a, b, c, d) [17]. Dansereau and Bruton [19] showed that all light rays emitted from a scene point P, which is located at a depth z, represent a hyperplane in the light field 4D space formed by (a, b, c, d). We call that plane a subspace which is described by:

$$\begin{aligned} \left\{ \begin{array}{l l} c=\frac{z-z_{l}}{z-z_{l}-d_{0}}a-\frac{z}{d_{1}}\left( 1-\frac{z-z_{l}}{z-z_{l}-d_{0}}\right) x_{p}\\ d=\frac{z-z_{l}}{z-z_{l}-d_{0}}b-\frac{z}{d_{1}}\left( 1-\frac{z-z_{l}}{z-z_{l}-d_{0}}\right) y_{p} \end{array} \right. \end{aligned}$$

(1)

where $(x_{p},y_{p})$ is the perspective coordinate of the scene point P. This coordinate is determined by the intersection between the image plane Ip and a non-deflected ray, emitted by P and crosses the camera aperture on its centre. Note that $d_1$ is the distance between the image plane and the aperture plane, and $z_l$ is the distance between the plan aperture and the plane CD.

The Eq. (1) defines lines that identify the subspace. These lines have the same orientation $\theta$ which can be determined from the lines slope which can also be written as $\tan (\theta )$. Therefore, the slope angle $\theta$ is given by:

$$\begin{aligned} \theta =\tan ^{-1}\big (\frac{z-z_{l}}{z-z_{l}-d_{0}}\big ) \end{aligned}$$

(2)

The Eq. (2) shows that the subspace formed by the light rays emitted by a scene point depends on the depth z of this point. Hence, the light field is built by a set of subspaces which are characterised by the scene depth.

3 Thin Lens Optic Simulation Model

In order to derive the optic simulation model of a UV camera, we analyse the light ray travelling towards the image plane Ip illustrated in Fig. 1. First, the light ray crosses the aperture plane Ap at coordinate $(a_{c},a_{d})$. By using basic geometry principles, the coordinate $(a_{c},a_{d})$ is given by:

$$\begin{aligned} \left\{ \begin{array}{l l} a_{c}=\frac{z_{l}+d_{0}}{d_{0}}c-\frac{z_{l}}{d_{0}}a \\ a_{d}=\frac{z_{l}+d_{0}}{d_{0}}d-\frac{z_{l}}{d_{0}}b \end{array} \right. \end{aligned}$$

(3)

The emerging light ray is deflected by the UV camera optical device and projected on Ip at the coordinate $(i_{x},i_{y})$ as depicted in Fig. 1. The optical device of the UV camera is assumed to be a thin lens. Such a hypothesis has already been used in [20,21,22] for cameras in visible spectrum and can be extended to UV cameras because they are based on the same imaging model. A thin lens, with a focal length f, focuses the light rays originating from a point $P_{f}$ at depth $z_{f}$, in one point of Ip located at a distance $d_{1}$ from the lens. This property is described by:

$$\begin{aligned} \frac{1}{f}=\frac{1}{d_{1}}+\frac{1}{z_{f}} \end{aligned}$$

(4)

Another characteristic of a thin lens is that a light ray crossing the lens on its center is not deflected. By using basic geometry principles and the mentioned properties, the imaging point at coordinate $(i_{x},i_{y})$ is determined by:

$$\begin{aligned} \left\{ \begin{array}{l l} i_{x}=\frac{z_{l}\left( d_{1}-f\right) -fd_{1}}{fd_{0}}a-\frac{\left( d_{0}+z_{l}\right) \left( d_{1}-f\right) -fd_{1}}{d_{0}f}c \\ i_{y}=\frac{z_{l}\left( d_{1}-f\right) -fd_{1}}{fd_{0}}b-\frac{\left( d_{0}+z_{l}\right) \left( d_{1}-f\right) -fd_{1}}{d_{0}f}d \end{array} \right. \end{aligned}$$

(5)

When a light ray intersects Ip, its radiance is transformed into the irradiance Ir defined as follows:

$$\begin{aligned} \begin{aligned} Ir(i_{x},i_{y})= L(a,b,c,d)~k\frac{\cos (\alpha )\cos (\beta )^{3}}{d_{1}^{2}}\Big (\frac{z_{l}+d_{0}}{d_{0}}\Big )^{2} dc\,dd \end{aligned} \end{aligned}$$

(6)

$$\begin{aligned} \alpha = cos^{-1}\Big (\frac{z_{l}}{\sqrt{(a_{c}-c)^{2}+(a_{d}-d)^{2}+z_{l}^2}}\Big ) \end{aligned}$$

(7)

$$\begin{aligned} \beta = cos^{-1}\Big (\frac{d_{1}}{\sqrt{(i_{x}-a_{c})^{2}+(i_{y}-a_{d})^{2}+d_{1}^2}}\Big ) \end{aligned}$$

(8)

where k is a weight, and $\alpha$ (resp. $\beta$) refers to the angle between the incident (resp. the emerging) light ray and Ap normal. The variables dc and dd are the width and the height of an infinitesimal rectangular area around the light ray intersection with the plane CD. Note that the radiance L in the Eq. (6) is already multiplied by a factor $\eta$, which models the attenuation caused by the scattering of UV light rays. Thus, for small wavelengths, $\eta$ is small, and the provided images are dark.

In [13, 23], the authors assume that the total energy I generated on a point at coordinate (x, y) on Ip is a linear combination of the irradiance deposited by each ray projected on (x, y). This total energy is given by:

$$\begin{aligned} I(x,y)= \int _{a} \int _{b} \int _{c} \int _{d}\alpha _{x,y,a,b,c,d}L(a,b,c,d)\,da\,db\,dc\,dd \end{aligned}$$

(9)

This latter equation corresponds to the optic simulation model of a camera with a thin lens. In the following sections, I is considered as an UV image formed on Ip. The coefficient $\alpha _{x,y,a,b,c,d}$ is a weight which transforms the radiance into irradiance. The attenuation factor $\eta$; and instead of being multiplied by the radiance; can be modeled in the weight $\alpha$. Thereby, for small wavelengths, $\eta$ is small, and UV light rays are stopped by the lens, which produce dark images. Note that $\alpha _{}$ also selects light rays projected at (x, y) according to two functions. The first function determines if the light rays are projected on (x, y). It is a unit impulse function $\delta ()$ which is one when a light ray intersects Ip on (x, y); otherwise, it is zero. It is satisfied by filtering the light rays in function of their trajectory. This filtering operation can be seen as the numerical simulation micro-lenses [13] embedded in light field cameras. The second function models the camera aperture and informs if a light ray passes through it. We choose to model the aperture as a Gaussian. The Gaussian model has the advantage to approximate the effects of the phenomena caused by the optical device and happened during the image formation, as the diffraction and the dispersion [24]. This Gaussian weighs the rays passing outside of the camera aperture, of radius r, by a value close to zero; otherwise, it generates higher weights. The weight $\alpha _{x,y,a,b,c,d}$ is determined by:

$$\begin{aligned} \alpha _{x,y,a,b,c,d}=\frac{1}{2\pi (\frac{r}{3})^{2}}\delta ((i_{x}-x)^{2}+(i_{y}-y)^{2}) ~ e^{\big (-\frac{a_{c}^{2}+a_{d}^{2}}{2(\frac{r}{3})^{2}}\big )} k\frac{\cos (\alpha )\cos (\beta )^{3}}{d_{1}^{2}}\big ( \frac{z_{l}+d_{0}}{d_{0}}\big )^{2} \end{aligned}$$

(10)

4 Light Field Inference

To reconstruct a light field, we need an image $I_{b}$ of a scene captured by an UV camera and a depth map of the same scene acquired by a depth camera. These data are then used to solve an inverse problem. This problem is the following: Given a 2D image, we have to determine a light field. In order to solve such a problem, we have to fulfil two requirements. First, by using (9), an accurate reconstruction of $I_{b}$ must be calculated from a light field estimated from $I_{b}$. This requirement is modelled by determining the light field which minimises a reconstruction error described by:

$$\begin{aligned} F_{E}(L)= \int _{x} \int _{y} \Big (I_{b}(x,y) - \int _{a} \int _{b} \int _{c} \int _{d} \alpha _{x,y,a,b,c,d}L(a,b,c,d)\,da\,db\,dc\,dd\Big )^{2}dx\,dy \end{aligned}$$

(11)

We assume that the radiance of light rays emitted from a scene point are almost equal over consecutive pixels receiving that rays as in [23]. In the light field 4D space, it means that the radiance of each light ray represented in a subspace (1) varies smoothly. This allows us to formulate the second requirement as: the light field must minimise the radiance variation between the light rays contained in a subspace. Such a requirement is modelled by the minimisation of the light field gradient in a subspace. These subspaces are identified by the gradient orientation $\theta$ defined in (2). The smoothness requirement can be expressed as:

$$\begin{aligned} \begin{aligned}&F_{O}(L_{a}, L_{b}, L_{c}, L_{d})= \int _{a} \int _{b} \int _{c} \int _{d} ((L_{a}(a,b,c,d) +L_{b}(a,b,c,d))\cos (\theta ) \\&+ (L_{c}(a,b,c,d) +L_{d}(a,b,c,d))\sin (\theta ))^{2}da\,db\,dc\,dd \end{aligned} \end{aligned}$$

(12)

In order to estimate a light field which fulfils these two requirements, Eqs. (11) and (12) have to be linearly combined. The strength of the second requirement is modulated by a weight $\gamma$ as in the following equation:

$$\begin{aligned} min_{L}\;F_{E}(L)+\gamma F_{O}(L_{a}, L_{b}, L_{c}, L_{d}) \end{aligned}$$

(13)

In this variational framework, the image $I_{b}$ is given by the UV camera and the scene depth map is given by a depth camera, e.g. the Kinect camera [25]. The UV camera intrinsic parameters are found by using data provided by the manufacturer and the calibration procedure described in [26]. The only unknown variable is the light field L. To solve this inference problem, we choose use the partial derivative equation (PDE) of (13) by using the Gateaux derivates. The PDE of (13) corresponds to the following equation:

$$\begin{aligned} \begin{aligned}&\int _{x}\int _{y}\Big (\alpha _{x,y,m,n,p,q}I_{b}(x,y)-\alpha _{x,y,m,n,p,q}\\&\int _{a}\int _{b}\int _{c}\int _{d} \alpha _{x,y,a,b,c,d}L(a,b,c,d)\,da\,db\,dc\,dd\Big )dx\,dy\\&+\gamma \big (\left( L_{mm}(m,n,p,q)+2L_{mn}(m,n,p,q)+L_{nn}(m,n,p,q)\right) \cos (\theta )^{2} \\&+2 \sin (\theta )\cos (\theta )\big ((L_{mp}(m,n,p,q)+L_{mq}(m,n,p,q) +L_{np}(m,n,p,q)\\&+L_{nq}(m,n,p,q)\big ) +L_{pp}(m,n,p,q)+2L_{pq}(m,n,p,q)\\&+L_{qq}(m,n,p,q)\big )\sin (\theta )^{2} =0 \end{aligned} \end{aligned}$$

(14)

where $L_{mn}$ is the second derivative of L with regard to m and n. To numerically solve this PDE, one can discretise it. The resulting system of linear equations can be written in matrix format. The matrices representing the discrete image $I_{b}$ and light field L have a resolution of $X\times Y$ and $A\times B\times C\times D$ respectively, as well as a sampling step of one along their respective dimensions. Let $i_{b}$ and l be vectors obtained by the vectorisation of those two matrices. We note T, a projection matrix of dimension $XY\times ABCD$, where each row indexes a pixel location in $i_{b}$ and each column indexes the light ray position in l. Let G be a derivation matrix of dimension $ABCD\times ABCD$ which allows calculating the partial derivatives in (14) by using the explicit finite difference as a numerical scheme. In matrix form, the PDE is written as:

$$\begin{aligned} T^{t}i_{b}-T^{t}Tl+\gamma G l=0 \end{aligned}$$

(15)

where $T^{t}$ is the transpose of T. Assuming $(T^{t}T-\gamma G)$ is invertible, the light field solution of the PDE is given by the equation:

$$\begin{aligned} l=(T^{t}T-\gamma G)^{-1}(T^{t}i_{b}) \end{aligned}$$

(16)

5 Results and Discussion

The light field estimation method and the optic simulation model are tested on several UV images acquired by a UV monochrome camera, Sony XC-EU50, and depth maps acquired by a Kinect camera. The Table 1 shows the intrinsic parameters of the UV camera and the lens, Pentax TV lens 25 mm. The resolution of plane AB resolution is $720\times 480$ and the resolution of plane CD is variable. In the experiments, k is set to one and the distance $d_{1}$ is equal to $d_{0}$. Moreover, the CD plane is placed on the lens so that $z_{l}$ is zero. The depth maps acquired by the Kinect camera contains gaps where the depth is set to zero. These gaps are filled in with depth values by using an edge confined diffusion algorithm similar to the one explained in [27]. The light field estimation algorithm has been implemented in C++ and parallised by using OpenMP on a 5200 kernels PC cluster of MOIVRE laboratory. We tested the light field estimation on various UV images and depth maps. A subset of images is illustrated in Fig. 2. The performance evaluation of the proposed light field estimation method, in the absence of ground truth light fields, comes back to evaluate reconstructed images. Two measures are used to study the influence of the value of the weight $\gamma$, in (13), and the resolution of CD plane on the quality of the estimated UV light field. Firstly, the mean absolute error (MAE) is measured between the original UV image $I_{b}$ and an UV image I reconstructed by applying Eq. (9) on the light field estimated from $I_{b}$ and the camera parameters in Table 1. The MAE is defined by:

$$\begin{aligned} MAE=\frac{1}{XY}\left. \sum _{x=1}^{X}\sum _{y=1}^{Y}\left| I_{b}(x,y)-I(x,y)\right| \right. \end{aligned}$$

(17)

Table 1. Intrinsic parameters of the camera Sony XC-EU50 and the Pentax TV lens 25 mm.

Fig. 2.

Greyscale UV images and greyscale depth maps captured with an UV camera and a Kinect camera. In the depth map, the darkest to the brightest gray-scale means close depths to further depths with regard to the camera position. In all these scenes, the depth varies between 50 and 123 cm. The scene (a) is composed of a plane object and (b) is an apple placed on a cubic metal object in front of a plane object. The scene (c) contains several plane objects and the UV camera focuses on a banana in the background. The figure (d) is the picture of a person face on which was added sunscreen on a side of the face.

Secondly, the spatial information of I and $I_{b}$ are compared. The error of the spatial content reconstruction is measured by the mean cosine of gradient orientation differences $\varDelta$ between edges of I and $I_{b}$. This measure is noted CADE and is given by:

$$\begin{aligned} CADE=\frac{1}{N}\sum _{n=1}^{N}\cos \left( \varDelta (n)\right) \end{aligned}$$

(18)

where N is the total number of pixel edges and $\varDelta (n)$ is the gradient orientation difference between the edge pixel n of I and $I_{b}$.

The Table 2 gives the measures of MAE and CADE for different values of the weight $\gamma$ and a resolution of plane CD of $7 \times 7$.

Table 2. CADE and MAE measured for several values of $\gamma$ on images reconstructed from light fields estimated by using images in Fig. 2. The CADE and the MAE are rounded to two digits of precision.

The MAE has low values around 0.70 and the CADE is close to 0.85 for values of $\gamma$ between 100 and 0.5 so that $\gamma$ does not influence the light field reconstruction when it is not zero. For $\gamma$ equal to zero, the MAE corresponds to a high reconstruction error and the CADE is close to zero which means that I does not resemble $I_{b}$. These observations imply that the second requirement in (13) greatly enhances the quality of the estimated light field and $\gamma$ must be non-null. However, a value of $\gamma$ greater than zero does not seem to influence the light field quality so that we arbitrarily set it to one in the following experiments.

Table 3. Measured CADE and MAE for several CD plane resolutions ($C \times D$) on image reconstructed from light fields estimated by using images in Fig. 2. The CADE and the MAE are rounded to two digits of precision.

Table 3 shows the MAE and the CADE measured for five resolutions of plane CD. The CADE gives an error around 0.85. Such a difference shows that there are errors in the spatial content of the reconstructed image I; however, they are sufficiently small to say that I is close to $I_{b}$. The measured MAE shows that each pixel has a small reconstruction error around 0.71. From these two measures, one observes that our method allows a good reconstruction of $I_{b}$ from the estimated light field. The MAE and the CADE seem invariant for the different CD resolutions. Such an invariance is due to the UV camera depth of field and the configuration of the scene captured.

The estimated light field can be used to reconstruct images with different intrinsic parameters than the ones presented in Table 1. Therefore, the plane of focus and the blur introduced in the image can be changed. Also, variation of these parameters can be used to focus areas of an image at a precise depth. The images in Figs. 3 and 4 are reconstructed from the thin lens optic simulation model of the UV camera in (9) for a $5 \times 5$ UV resolution, an aperture radius of 37 mm and various focal lengths.

Fig. 3.

Focus on the foreground (a) and on the background (b) of the scene in Fig. 2c

Fig. 4.

Focus on the face of the person in the foreground (a) and the background (b) of the scene of Fig. 2d

In Fig. 3, artefacts are visible on the object in foreground which is slightly blurred even when it is in focus. These artefacts are due to bad depth values estimated on this object. However, in Fig. 4, no artefacts are visible because of the accurate depth map estimation from the Kinect camera. In these images, a natural blur variation, depending on $z_{f}$, is observed. Such a behaviour is realistic because it mimics the focus of a real camera.

6 Conclusion

In this paper, a light field estimation method by using an UV camera is presented. This method relies on solving an inverse problem based on the light field properties and the thin lens optic simulation model of an UV camera. The quality of a light field is assessed by the comparison of an original image and an image reconstructed from an estimated light field. We make it possible to acquire an UV light field and produce from it UV images with different acquisition configurations. This finding opens the door to the computational photography and strengthens the use of UV in its traditional applications. In future work, we will apply this estimation method in order to create a multi-spectral light field by estimating light fields from an ultraviolet image, an infrared image and a colour image in visible spectrum. Such a light field inference will increase the quantity of information contained in a light field so that it will increase the number of possible applications. In a future study, we will also find a quality assessment method which evaluates the inferred light field quality in a much better way than the CADE and the MAE of a reconstructed image.