Multispectral Constancy Based on Spectral Adaptation Transform

Abstract

The spectral reflectance of an object surface provides valuable information about its characteristics. Reflectance reconstruction from multispectral images is based on certain assumptions. One of these assumptions is that the same illumination is used for system calibration and image acquisition. We propose the novel concept of multispectral constancy, achieved through a spectral adaptation transform, which transforms the sensor data acquired under an unknown illumination to a generic illuminant-independent space. The proposed concept and methods are inspired from the field of computational color constancy. Spectral reflectance is then estimated by using a generic linear calibration. Results of reflectance reconstruction using the proposed concept show that it is efficient, but highly sensitive to the accuracy of illuminant estimation.

1 Introduction

Multispectral imaging refers to the acquisition of image data at specific intervals in the electromagnetic spectrum by the use of spectral filters. These spectral filters are sensitive to specific wavelengths and allow acquisition of data in form of channels. Color images contain three channels while a multispectral image consists of more than three channels. With advancement in the sensor technology, use of multispectral imaging for indoor scene acquisition under controlled conditions has also increased. The advantage of multispectral imaging is the ability to acquire more spectral information of a scene, which can be used for spectral reflectance reconstruction [1] of the object’s surfaces. A constraint associated with such use of multispectral imaging is the need of system calibration for the imaging environment when performing the spectral reconstruction [2]. A system being calibrated for one type of conditions needs to be re-calibrated if it has to be used in another imaging environment. One of the major limitations is the need for having the same illuminant for spectral reconstruction system’s calibration and the image acquisition. This limitation is a major challenge and hurdle for the use of multispectral imaging in outdoor environments.

In this paper, we introduce the concept of multispectral constancy, as an extension of the computational color constancy [3] but in higher spectral dimension. By multispectral constancy, we refer to the representation of imaged surfaces without the effect of the scene illuminant. In other words, the surfaces appear as if they are taken under a canonical illuminant. To achieve multispectral constancy, we introduce a spectral adaptation transform (SAT). The closest concept to SAT in the literature is spectral adaptation being introduced by Fairchild [4] and is applied on the full spectral reflectances, while we generalize it to the sensor measurements. By attaining multispectral constancy, multispectral imaging can be used for any illuminant without the requirement of re-calibration when imaging environment is changed.

This paper is organised as follows. Section 2 formalises the problem of spectral acquisition, including the calibration and spectral reconstruction as a linear problem. We propose to add a term to the usual calibration, which discards the illumination change. Section 3 defines our simulation and experimental protocol. Results are analyzed in Sect. 4 before we conclude.

2 Spectral Adaptation Transform

In the context of a simplified noiseless imaging model, a pixel captured in the imaging system is the combination of spectral reflectance of the surface $r(\lambda )$, spectral power distribution of illuminant $e(\lambda )$ and spectral sensitivity of the $n^{th}$ spectral channel $m_n(\lambda )$. This formation for the visible wavelength spectrum $\omega$ is defined as

$$\begin{aligned} f_n = \int _{\omega } r(\lambda ) e(\lambda ) m_n(\lambda ) d\lambda \end{aligned}$$

(1)

In practice, we can formulate a discrete version of Eq. 1 as $\mathbf F = \mathbf{REM}$. Considering the spectral sampling of 10 nm within the wavelength spectrum of 400 nm to 720 nm and N number of spectral filters, $\mathbf F$ is $S\times N$ matrix (S is the number of spectral samples being acquired), $\mathbf R$ is $S\times 33$ matrix of surface reflectance, $\mathbf E$ is the diagonal matrix ($33 \times 33$) of the scene illuminant and $\mathbf M$ is $33\times N$ matrix, consisting of spectral sensitivities of the filters.

Here we consider two cases of image acquisition. One is with a canonical illumination $\mathbf E _c$ and the other case is when an unknown illuminant $\mathbf E _{ill}$ is used. We present both cases in parallel in Eq. 2.

$$\begin{aligned} \mathbf F _{ill} = \mathbf R {} \mathbf E _{ill}{} \mathbf M \qquad \text {;} \qquad \mathbf F _c = \mathbf R {} \mathbf E _c\mathbf M \end{aligned}$$

(2)

To perform the spectral reflectance estimation $\hat{\mathbf{R }}$ from the imaged data in both of the above mentioned cases, we can apply a generalized inverse, denoted by ${^+}$, as in Eq. 3.

$$\begin{aligned} \hat{\mathbf{R }} = \mathbf F _{ill}{} \mathbf M ^{+} \mathbf E _{ill}^{+} \qquad \text {;} \qquad \hat{\mathbf{R }} = \mathbf F _c\mathbf M ^{+}{} \mathbf E _c^{+} \end{aligned}$$

(3)

For the reconstruction of spectral reflectance, knowledge of illumination is required. There are many methods to estimate the sensor sensitivity $\mathbf M$ for a given camera [5, 6]. However, it is not an easy task to measure the scene illuminant every time along with the image acquisition.

The spectral reconstruction is performed by training a calibration matrix $\mathbf W = \mathbf M ^{+}{} \mathbf E ^{+}$ [7]. This calibration is specific for a given illumination $\mathbf E _c$. By using the calibration matrix $\mathbf W$, the equations for spectral reflectance reconstruction become $\hat{\mathbf{R }} = \mathbf F _{ill} \mathbf W _{ill}$ and $\hat{\mathbf{R }} = \mathbf F _c \mathbf W _c$ for both cases, respectively. We propose to transform the acquired image $\mathbf F _{ill}$ into a canonical representation $\mathbf F _c$. In this way, $\mathbf W _c$ can be used for the spectral reconstruction from a multispectral image, being taken under any illumination. We call this concept as the multispectral constancy. It is achieved through SAT, which is in the form of a diagonal matrix $\mathbf A _{SAT}$ and is applied as,

$$\begin{aligned} \mathbf F _c = \mathbf A _{SAT}{} \mathbf F _{ill} . \end{aligned}$$

(4)

In Eq. 4, SAT being applied to the acquired spectral data enables the estimation of reflectance spectra from an image being taken under any illuminant and can be used as,

$$\begin{aligned} \hat{\mathbf{R }}=\mathbf W _c\mathbf A _{SAT}{} \mathbf F _{ill} \end{aligned}$$

(5)

With the use of SAT as in Eq. 5, the requirement of having same illuminant for spectral reconstruction system’s calibration and the image acquisition, is no longer required since the acquired spectral image is transformed before the estimation of spectra. Obtaining $\mathbf A _{SAT}$ and $\mathbf W$ are explained in Sects. 3.3 and 3.4, respectively.

3 Methodology and Experimental Setup

3.1 Simulation Pipeline

Figure 1 shows the pipeline of experimental framework for the spectral reconstruction, based on multispectral constancy. This pipeline consists of sensor simulation, acquisition of spectral data, SAT, spectral reconstruction and the evaluation of results. These blocks are explained in the following sections.

Fig. 1.

Pipeline for evaluation of the proposed framework of multispectral constancy. Spectral measurements from Macbeth ColorChecker are used as the reflectance data. Illuminants $\mathbf E _c$ and $\mathbf E _{ill}$ are used for training and testing, respectively. To achieve multispectral constancy, SAT is applied to the acquired spectral data after estimation of illuminant in the sensor domain. Result from spectral reconstruction of transformed data is evaluated with the measured spectra of the Macbeth ColorChecker.

3.2 Sensor

To implement and validate the proposed idea of multispectral constancy, we use reflectance data in the wavelength range of 400 nm to 720 nm, from the GretagMacbeth ColorChecker [8]. We apply equi-Gaussian filters [9] for simulation of the spectral filters and use 8 filters in the experiments. By increasing the number of filters, more noise is introduced in the image which affects the spectral reconstruction. This effect was observed by Wang et al. [7]. Radiance data is created by using the illuminants E and D65 and then the simulated sensors are used to acquire the multispectral data.

3.3 Multispectral Constancy Through SAT

In order to achieve multispectral constancy, SAT is applied to the spectral image after the estimation of illuminant in the sensor domain as in Eq. 5. For estimation of the illuminant in spectral images of natural scenes, we propose using the Max-Spectral Algorithm and the Spectral Gray-Edge Algorithm. These algorithms are extensions of Max-RGB Algorithm [10] and the Gray-Edge Algorithm [11, 12]. The extension of these algorithms from color to spectral is discussed in detail in [9, 13]. Once the illuminant is estimated, then we propose to apply the SAT in form of a diagonal correction to the acquired data, so that it appears as if being taken under a canonical illuminant. Such a diagonal transform was initially proposed by Von Kries [14]. We extend this transform into the spectral domain. For N number of channels, SAT is defined as in Eq. 6.

$$\begin{aligned} \begin{pmatrix} F_1^c \\ F_2^c \\ \vdots \\ F_N^c \end{pmatrix} = \begin{pmatrix} k_1 &{} 0 &{} \dots &{} 0 \\ 0 &{} k_2 &{} \dots &{} 0 \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ 0 &{} 0 &{} \dots &{} k_N \end{pmatrix} . \begin{pmatrix} F_1^u \\ F_2^u \\ \vdots \\ F_N^u \end{pmatrix} \end{aligned}$$

(6)

Here, $F_n^u$ is the pixel of $n^{th}$ channel, taken under an unknown illuminant while $F_n^c$ is the transformed images so that it appears to be taken under a canonical illuminant. $k_n$ is the correction parameter for the channel n, which is obtained from the illuminant estimation (IE).

3.4 Spectral Reflectance Reconstruction

As explained in Sect. 2, a training matrix is required for the spectral reconstruction from camera data. This matrix $\mathbf W$ is called the calibration matrix. It is obtained by using measured reflectance spectra $\mathbf R _t$ and the acquisition of radiance data $\mathbf F _t$, using the same camera. We use reflectance data from the GretagMacbeth ColorChecker [8] $\mathbf R _t$ and illuminant E for calibration of $\mathbf W$.

There are many methods being proposed for spectral reconstruction. We use linear methods to keep the proposed system simple and robust [2]. Those linear methods include the linear least square regression, principal component analysis [15] and Wiener estimation [16]. We evaluated the performance of these three methods and got similar results. We decided to use the Wiener estimation method because it is robust to noise and fulfills the criteria of being linear. It is defined as

$$\begin{aligned} \mathbf W = \mathbf R _t \mathbf R _t^T (\mathbf M \mathbf E )^T ((\mathbf M {} \mathbf E ) \mathbf R _t \mathbf R _t^T (\mathbf M {} \mathbf E )^T + \mathbf G )^{-1} \end{aligned}$$

(7)

Here, $\mathbf R _t \mathbf R _t^T$ and $\mathbf G$ are the autocorrelation matrices of training spectra and additive noise, respectively. G is in the form of a diagonal matrix consisting of the variance of noise ${\sigma }^2$.

One of the major shortcomings of the linear methods for spectral reconstruction is the assumption that the same illuminant is used for system’s calibration and acquisition of data. We are interested in the development of a spectral reconstruction system which does not require the same illuminant for system’s calibration and data acquisition. This can be achieved through the illuminant estimation in spectral images and then applying SAT to the acquired data as in Eq. 5. The matrix $\mathbf W$ is obtained by using the training reflectance spectra and radiance with the canonical illuminant $\mathbf E _c$ (we use illuminant E). This matrix is used for the spectral reconstruction from the spectral data being taken under any lighting conditions.

We evaluate the validity of our proposed idea by performing an experiment on the measured reflectance data of Macbeth ColorChecker for spectral reflectance reconstruction. Those results are provided in Sect. 4.

3.5 Evaluation

To measure the performance of spectral reconstruction, we compare the reconstruction $\hat{\mathbf{r }}$ for each color patch of the Macbeth ColorChecker with the measured reflectance $\mathbf r$, through root mean square error (RMSE) and goodness of fit coefficient (GFC) [17] as in Eqs. 8 and 9 respectively.

$$\begin{aligned} \text {RMSE} = \sqrt{\frac{1}{33}\sum _{j=1}^N ({r_j - \hat{r}_j})^2} \end{aligned}$$

(8)

$$\begin{aligned} \text {GFC} = \frac{{ \mathbf r ^T \hat{\mathbf{r }}}}{\sqrt{(\mathbf r ^T\mathbf r )( \hat{\mathbf{r }}^T\hat{\mathbf{r }})}} \end{aligned}$$

(9)

Besides that, we also provide spectral reconstruction results for the reflectance patches of the Macbeth ColorChecker in the form of graphs so that the overall performance of spectral reconstruction can be visually analyzed.

4 Results and Discussion

In this section we analyze the validity of our proposed idea of multispectral constancy through SAT. We also investigate the influence of illuminant estimation on the results of spectral reconstruction. We use three different noisy estimates of illuminant for testing the proposed framework to assess the influence of erroneous illuminant estimation.

For training of matrix $\mathbf W$, 24 patches of Macbeth ColorChecker are used within the visible wavelength range (400–720 nm). For spectral reconstruction, we use different scenarios, which are given in Table 1.

Table 1. Description of the experiments being performed. Adequate calibration of system is performed by using the same illuminant. In no correction case, we use different illuminants and do not apply any correction. In ideal correction, SAT is applied by assuming efficient estimation of the illuminant while in naïve correction, SAT is applied using the illuminant E, while D65 is used for image acquisition. In next three experiments, we apply SAT using the three different noisy estimate of illuminants.

For testing the influence of error in the illuminant estimation, we use three different noisy estimates. Angular error ($\varDelta A$) is calculated in term of radians between the vectors of original illuminant $\mathbf e$ and the estimated illuminant $\hat{\mathbf{e }}$ as

(10)

For checking the effect of error in illuminant estimation, use three different randomly generated estimates of illuminant with $\varDelta A$ of 0.0210, 0.1647 and 0.3658 rad. We apply these erroneous illuminants in the sensor domain to the acquired spectral image of Macbeth ColorChecker according to Eq. 6. First we evaluate spectral reconstruction and then analyze the performance of SAT. Figures 2, 3 and 4 show results obtained from the 24 reflectance patches of the Macbeth ColorChecker in the visible spectrum. These results are discussed in the following sections.

4.1 Reflectance Estimation

With the use of linear method for spectral reconstruction (Wiener estimation [16]), we evaluate the performance of the algorithm using the adequate calibration of system. We measure the performance with both illuminants E and D65. They provide similar results. We show results of adequate calibration with illuminant D65 in Figs. 2, 3 and 4. This is the best reconstruction that can be obtained with this given number of sensors and sensor configuration. We investigate the performance of Wiener estimation when different illuminants are used for training and testing and no SAT is applied. We also perform SAT with illuminant E in sensor domain and the camera data being acquired with illuminant D65. We call this naïve correction. With adequate calibration, the best spectral reconstruction results we could obtain with Wiener estimator, provided an average RMSE of 0.001 and average GFC of 0.9993 over the 24 patches of Macbeth ColorChecker. With no correction being applied and using different illuminants for training and acquisition, average RMSE and GFC were 0.0118 and 0.9898 respectively. To validate the idea of multispectral constancy through SAT, the error in spectral reconstruction must be low as compared to the error in case of applying no correction. We evaluate the performance of SAT in Sect. 4.2.

Table 2. Mean RMSE and mean GFC from spectral reconstruction of 24 patches of the Macbeth ColorChecker.

4.2 SAT Performance

Fig. 2.

First 9 reflectance patches from Macbeth ColorChecker. In each figure, there are curves of measured reflectance, spectral reconstruction from adequate calibration, no correction, ideal correction, naïve correction, reflectance reconstruction results after applying SAT from simulated illuminants having $\varDelta A$ of 0.0210, 0.1647 and 0.3658, respectively.

Fig. 3.

Reflectance patches 10–18 from the Macbeth ColorChecker. In each figure, there are curves of measured reflectance, spectral reconstruction from adequate calibration, no correction, ideal correction, naïve correction, reflectance reconstruction results after applying SAT from simulated illuminants having $\varDelta A$ of 0.0210, 0.1647 and 0.3658, respectively.

Fig. 4.

Reflectance patches 19–24 from the Macbeth ColorChecker.

Figures 2, 3 and 4 show the spectral reconstruction results from the multispectral data acquired using eight spectral filters. Although exact spectral reconstruction is not possible with a reduced number of bands, Wiener estimation method is still able to make a close match when adequate calibration is applied. For testing the performance of SAT, we calibrate the spectral reconstruction system for illuminant E and acquire the radiance data under the illuminant D65. It is obvious from the spectral reconstruction results that the overall accuracy of our proposed framework is dependent on the accuracy of illuminant estimation, as would be expected. With efficient illuminant estimation, SAT performs almost the same as in the case of adequate calibration. However, it is interesting to note that although there is overlapping between the spectral reflectance reconstruction curves in the case of adequate calibration and ideal correction as seen in Figs. 2, 3 and 4, the difference in RMSE and GFC is not as close as expected (see Table 2). This leads to opening the discussion about efficiency of SAT and the question of whether SAT should be optimised as well in order to get better results. Another problem to be investigated is the required efficiency of both SAT and the illuminant estimation for applications like object detection and classification on the basis of their spectral properties. However, the closeness in result proves that if efficient illuminant estimation is performed and SAT is applied, we can attain close results as compared with adequate calibration. The advantage of using multispectral constancy is that there is no requirement of measuring the scene illuminant explicitly. The only factor which remains important in our proposed idea is the efficient estimation of illuminant in the spectral image.

The dependence of SAT on the accuracy of illuminant estimation is evident when spectral reconstruction is performed. SAT being applied along with erroneous estimation of illuminant in the sensor domain causes error in the spectral reconstruction. It is interesting to note that even the naïve correction is able to perform well in comparison with the result of applying no correction, which makes the role of illuminant estimation as a significant factor for our proposed idea of multispectral constancy through SAT. SAT itself also needs to be investigated so that an efficient framework for the transformation of acquired spectral image into the illuminant free representation (multispectral constancy) can be achieved.

5 Conclusion

This work formalizes the concept of multispectral constancy, which permits multispectral image acquisition, independent of the illumination. Multispectral constancy is achieved via a spectral adaptation transform, which changes data representation from the actual sensor domain, towards a canonical one, where calibration applies. Simulation results show that a diagonal SAT permits to achieve similar reflectance reconstruction than when the samples are acquired under the illumination being used for calibration. However, when the spectral adaptation transform is evaluated based on an estimate of illumination, error in illuminant estimates makes the performance drop down significantly.

It is still to be investigated that what level of accuracy is required for the illuminant estimation to make this concept useful for computer vision applications. It is also important to recall that these results are compiled based on simulation of noiseless sensor data acquisition. Further work shall investigate these directions and define the limits of using this approach.