PSO-Based Single Image Defogging

Abstract

This paper proposes a novel defogging algorithm based on particle swarm optimization (PSO) to adaptively and automatically select parameter values. Owing to the lack of enough information to solve the equation of image degradation model, existing defogging methods generally introduce some parameters and set these values fixed. Inappropriate parameter setting leads to difficulty in obtaining the best defogging results for different input foggy images. In this paper, we mainly focus on the way to select optimal parameter values for image defogging. The proposed method is applied to two representative defogging algorithms by selecting the two main parameters and optimizing them using the PSO algorithm. A comparative study and qualitative evaluation demonstrate that the better quality results are obtained by using the proposed method.

1 Introduction

Most automatic systems assume that the input images have clear visibility, therefore removing the effects of bad weather from these images is an inevitable task. In the past decades, extensive research efforts have been conducted to remove fog or haze from a single input image. Most of these methods [1,2,3,4] intend to recover scene radiance using the image degradation model that describe the formation of a foggy image. Tan [1] removed fog by maximizing the local contrast of the restored image. Nishino et al. [2] proposed a Bayesian probabilistic method that estimates the scene albedo and depth from a foggy image with energy minimization of a factorial Markov random field. He et al. [3] estimated the transmission map and the airlight of the degradation model using the dark channel prior. Tarel et al. [4] introduced an atmospheric veil to restore image visibility based on the fast median filter. However, these methods are controlled by a few parameters with fixed values that cannot be automatically adjusted for different foggy images.

In recent years, people are quite interested in automatic fog removal, which is useful in applications such as surveillance video [5], intelligent vehicles [6], and outdoor object recognition [7]. In this paper, we thus focus on the PSO-based adaptive parameter adjustment for single image defogging.

It is widely agreed that PSO algorithm can be used for digital image processing, such as image retrieval [8], image segmentation [9], etc. However, due to the lack of proper objective criterion of defogging effect as the fitness function, present defogging algorithms seldom use genetic algorithm to effectively remove fog from a single image. Though the objective image quality evaluation methods have achieved some promising results, they are just applied to assess the quality of degraded image, such as image denosing results and image deblurring results. The aim of defogging algorithm is to recover color and details of the scene from input foggy image. Unlike image quality assessment, the fog can not be addressed like a classic image noise or degradation which may be added and then removed. Meanwhile, there is no easy way to have a reference no-fog image, and the quality evaluation criteria of degraded image, such as the structural similarity (SSIM) [10], the peak signal-to-noise ratio (PSNR) [11], and the mean square error (MSE), are not suitable for assessing image defogging effects. This makes the problem of adaptive parameter adjustment of defogging algorithm not straightforward to solve.

In this paper, we mainly focus on the way to select parameter values for single image defogging. Here, the defogging effect assessment index presented in our previous work [12] is taken as the fitness function of the proposed PSO-based method to adaptively adjust the parameter values for different input foggy images. In Sect. 2 of the paper, the limitation of existing methods are described. Section 3 introduces the proposed parameter value selection approach to single image defogging. Experimental results and conclusions are presented in Sects. 4 and 5, respectively.

2 Limitation of the Existing Defogging Methods

Most current defogging methods recover the scene radiance by solving the image degradation model. Since the model contains three unknown parameters and the solving process is an ill-posed inverse problem, it is thus inevitable to introduce many application-based parameters that used in various assumptions for image defogging. A large quantity of experimental results shows that the selection of the algorithm parameters has direct influence on the final defogging effect. However, there exists a major problem for the parameter setting in most defogging algorithms, i.e. the parameters always have fixed values in the defogging algorithms.

In our experiments, we find that the fixed parameter values caused that the fog removal algorithms just have good defogging effect for a certain kind of foggy image, and the algorithms may not work well for the images captured under other foggy conditions. For example, He’s algorithm [3] has mainly three parameters to control: $\omega$ which alters the amount of haze kept at all depths, c the patch size for estimating transmission map, and t$_{{\textit{0}}}$ restrict the transmission to a lower bound to make a small amount of fog preserve in very dense fog regions. All these parameters have fixed value suggested by the authors, such as the fog parameter $\omega$ , which is set to be 0.95 in the algorithm [3]. Our experimental results given in Fig. 1 using He’s algorithm [3] show that, if $\omega$ is adjusted downward, more fog will be kept, and vice versa. Using $\omega = 0.95$ keeps a slight amount of fog effect around at all depths. However, the experiments show that $\omega$ sometimes needs to be decreased when an image contains substantial sky regions, otherwise the sky region may wind up having artifacts. An example showing the need to decrease $\omega$ is presented in Fig. 1. The defogging result with $\omega = 0.95$ is shown in Fig. 1(b). One can clearly see that the sky looks contoured since the fog removed by He’s algorithm was too strong in this region, and the defogging result has no obvious visibility improvement with $\omega = 0.65$ . If setting $\omega = 0.12$ , the sky region becomes brighter and smoother, which makes the whole image look more natural.

Fig. 1.

Fog removal results with fixed parameter value. (a) Original image. (b) Unpleasing contour effect with $\omega = 0.95$. (c) No obvious visibility improvement with $\omega = 0.65$. (d) Smoother sky region using $\omega = 0.12$

Most defogging algorithms have introduced some parameters, which lead to user interaction and make the final defogging effect hard to control as well. For example, Tarel’s algorithm [4] is controlled by five parameters [4] in which p is the percentage of removed atmospheric veil, $s_v$ the assumed maximum size of white objects in the image, b the white balance control for global or local process, $s_i$ the maximum size of adapted smoothing to soften the noise amplified by the restoration, and g an extra factor during final gamma correction. Figure 2 shows the defogging results obtained with different parameter values using Tarel’s algorithm [4]. One can clearly see that the restoration is too light with $p=0.7$, $b=0.5$, $s_v=19$, $s_i=1$, $g=1.3$ and too strong with $p=0.99$, $b = 0$, $s_v = 19$, $s_i = 1$, $g = 1$. It seems better with $p=0.96$, $b=0.5$, $s_v=19$, $s_i=1$, $g=1.3$. On the right, most of the colors are removed due to a too large value of $b = 1$, and $b = 0.5$ leads to better results. It is obvious that the controllability can be greatly improved and the user-interaction can be also largely reduced if defogging algorithm has no more than two parameters. Therefore, distinguishing between main parameters which directly affect the results and other less important parameters which can be considered as fixed values, and then automatically adjusting their values are very important for the defogging algorithms.

Fig. 2.

Fog removal results with many fixed parameters. (a) Original image. (b) Defogging result with p = 0.7, b = 0.5, s$_{{\textit{v}}}$ = 19, s$_{{\textit{i}}}$ = 1, g = 1.3. (c) Defogging result with p = 0.99, b = 0, s$_{\text {\textit{v}}}$ = 19, s$_{{\textit{i}}}$ = 1, g = 1. (d) More pleasing result with p = 0.96, b = 0.5, s$_{\text {\textit{v}}}$ = 19, s$_{{\textit{i}}}$ = 1, g = 1.3. (e) Less pleasing result with p = 0.99, b = 1, s$_{\text {\textit{v}}}$ = 19, s$_{{\textit{i}}}$ = 1, g = 1

In order to solve the “fixed parameter value” problem, on the one hand we should choose two main parameters and optimize them for different defogging algorithms, and on the other hand we must use an effective defogging effect assessment index to automatically determine the parameter values for different input image. However, the defogging evaluation results obtained using image quality evaluation criteria is often inconsistent with human visual conception, and the existing enhancement assessment methods are mainly from image contrast and do not consider the color restoration effect of the defogging results.

It can thus be seen that “fixed parameter value” is the most common problem existing in most fog removal methods. However, it is hard to realize adaptive adjustment of the algorithm parameters by the defogging results due to the lack of an effective assessment index. Therefore, a PSO-based parameter value selection algorithm using proper defogging effect assessment index is proposed in this paper to select the parameter values for single image defogging.

3 The Proposed Parameter Value Selection Approach

Since most defogging algorithms may produce oversaturated or undersaturated colors with fixed parameters, the two main parameters that directly affect the defogging results are first selected. And then a PSO algorithm combined with a proper defogging assessment index is used here to enable these defogging methods to adaptively set the required two parameters for different input images, and enhance the visibility of scenes with natural color and the best defogging effect. In this section, the detailed descriptions of the two main parameters selection approach to various representative defogging algorithms are first given in Subsect. 3.1. Then, the assessment index for measuring defogging effect is reported in Subsect. 3.2. Finally, the PSO-based parameter value selection method is presented in Subsect. 3.3.

3.1 Optimal Parameter Selection

The task of optimal parameter selection is to distinguish between two main parameters which directly affect the results and other less important parameters which can be considered as fixed values for various defogging methods. There are two ways to select the two main parameters: one is to analyze the related parameters from the perspective of physical mechanism, and the other is to tune one of algorithm parameters by fixing the rest to see whether the defogging results have significant change. Two representative fog removal methods [3, 4] are taken as examples to describe the parameter selection process in this paper. Since He’s method [3] is recognized as one of the most effective ways to remove fog, and Tarel’s method [4] is regarded as one of the fastest defogging algorithms at present.

For He’s method [3], a small amount of haze for distance objects is kept to make the final defogging results seem more natural and preserve the feeling of depth as well. There are two key parameters for the fog preservation purpose: $\omega {} {} {} (0< \omega < 1)$ and ${t_0}{} {} (0< {t_0} < 1)$ . Other parameters have less influence on the final defogging effect, and can thus be regarded as less important parameters. For example, the influence of the large patch size c can be effectively reduced by the soft matting used in He’s method [13]. Therefore, the patch size suggested by the author is reasonable. Figure 1 shows He’s defogging results using different $\omega$. The fog removal results using different ${t_0}$ or patch size c are shown in Fig. 3. One can clearly see that ${t_0}$ has significant influence on the final results, while the patch size c on the contrary has little influence.

Fig. 3.

He’s defogging results with different parameter values. (a) The influence of two main parameters on final results with $\omega = 0.95$ , c = 7. From left to right: original foggy image with size of 400 $\times$ 300, the results obtained with $t_{0} = 0.1$, 0.5, 1, respectively. (b) The influence of parameter c on the final results with $\omega = 0.95$ , $t_{\text {0}} = 0.5$. From left to right: the results obtained with c = 30, 15, 9, 3, respectively

For Tarel’s method [4], the most important two steps that determine the final defogging effect are atmospheric veil remove and gamma correction. Since the value of p(0 < p < 1) controls the amount of atmospheric veil that can be removed, this parameter is useful to compromise between highly restored visibility where colors may appear too dark, and less restored visibility where colors are clearer. The parameter g(0 < g < 10) is used to perform gamma correction to achieve more colorful result. Experimental results show that the larger the value of g, the clearer the defogging result is. Compared with the two parameters p and g, other parameters b, $s_{v}$ and $s_{i}$ have less effect on the final defogging results, as shown in Fig. 4.

Fig. 4.

Tarel’s defogging results with different parameter values. (a) The influence of two main parameters on the final results with b = 0.5, sv = 17, s$_{{\textit{i}}}$ = 1. From left to right: original foggy image with size of 370 $\times$ 400, the results obtained with (p = 0.1, g = 3), (p = 0.95, g = 3), (p = 0.1, g = 3), (p = 0.1, g = 7), respectively. (b) The influence of other three parameters on final results with p = 0.1, g = 5. From left to right: the results obtained with (b  = 0, sv = 17, s$_{\text {\textit{i}}}$ = 1), (b = 1, sv =  17, s$_{\text {\textit{i}}}$ = 1), (b = 0.5, sv = 5, s$_{\text {\textit{i}}}$ = 1), (b = 0.5, sv = 30, s$_{\text {\textit{i}}}$ = 1), (b  = 0.5, sv = 17, s$_{\text {\textit{i}}}$ = 10), respectively

From the two examples showed in Figs. 3 and 4, we can deduce that the two main parameters generally have significant influence on the final results, and also have definite physical meanings. While other parameters, such as the patch size or the window size of smoothing, have much less effect on the results and can thus be considered as fixed values. The conclusions can be applied to other defogging algorithms to select the key parameters for them.

3.2 Measurement of Defogging Effect

The CNC index, an effective defogging evaluation indicator proposed in our previous work [12] is used here to guide the parameter adjustment process. For the input foggy image x and its corresponding fog removal imagey, the CNC index is obtained after carrying out the following steps: (i) compute the ratee of visible edges after and before fog removal, (ii) calculate the image color naturalness index (CNI) and color colourfulness index (CCI) to measure the color naturalness of the defogging image y, and (iii) Combine the three components e, CNI and CCI to yield an overall defogging effect measure:

$$\begin{aligned} \mathrm{{CNC}}(\mathbf{{x}}, \mathbf{{y}}) = h\left( {e(\mathbf{{x}}, \mathbf{{y}}), {} {} {} \mathrm{{CNI}}(\mathbf{{y}}), {} {} \mathrm{{CCI}}(\mathbf{{y}})} \right) \end{aligned}$$

(1)

For the overall variation trend of the three indexes, the statistical results show that the peak of CNI curve stands for the most natural result, but it is not necessarily the best defogging effect. However, the best effect must have good naturalness (high CNI value). When the image is overenhanced, the color is distorted, and CNI goes down rapidly. For e and CCI, they achieve the best effect before reaching their peaks. When the image is overenhanced, the curves continue ascending. After reaching their peaks, these curves begin to go down. Therefore, if the uptrend of e and CCI (from their best effect points to their curve’s peaks) can be largely counteracted by the downtrend of CNI, and the peak of CNC curve can be more close to the real best effect point. Meanwhile, the value variation of CNI is small, while that of e and CCI is relatively big. Thus, the effect of e and CCI on the CNC index needs to be weakened. The CNC index between image x and y, i.e. the function h in (1) can be defined as

(2)

As explained above, a good result is described by the large value of CNC. Therefore, the optimal value of the two main parameters of defogging algorithm scan be obtained when the CNC index (2) achieves the largest value.

3.3 Parameter Value Selection Using Particle Swam Optimization

The particle swarm optimization (PSO) is one of the most important swarm intelligence paradigms [14], and its searching is based on the simulations of social behaviors such as animals herding, fish schooling, and birds flocking where the swarm search for food in a collaborative manner. The PSO is easy to implement, and therefore, it is employed to solve the optimization problems in many applications. Using PSO, the two main parameters of various defogging algorithms can be automatically determined on the basis of the CNC index. The simple but effective PSO-based parameter selection consists in the following steps:

Parameter Initialization. Initialize the beginning parameters, such as swarm populations, and the number of training iterations. Also, the particles are randomly located and the movement vector is randomly assigned. To prevent the blind search of particles, the particle’s position and velocity are constrained in a certain range.

The task using the PSO is to find the best combination of the two main parameters according to an objective criterion. Therefore, the initial position of a particle can be expressed as an 1-by-2 matrix of random value RV, and each element of $RV \in (0, 1)$. $X_{\min }$ and $X_{max}$ are the minimum and maximum range values for the position of the two parameters. The $X_{\text {min}}$ is set to 0, and the value of $X_{\max }$ depends on the value ranges of the two parameters. For example, the value ranges of parameters p and g in Tarel’s method are (0, 1) and (0, 10), respectively, so the 2 parameter values can be written as $p = RV$ and $g = 10 \times RV$.

Fitness Function. Let Gbest be the best known position of the entire swarm and let Pbest be the best known position of particle i. Store Gbest and all Pbest locations at the current iteration by using an evaluation process employing the fitness function for all particles.

For image defogging, the fitness is measured by the CNC index [see Eq. (2)] of a defogged image obtained by the method presented above, because a color image with good defogging effect includes many visible edges and its color must be natural and colorful. Thus, the best fitness value is the one with the largest CNC value. Therefore, using the CNC index, the Gbest and Pbest can be obtained by selecting the highest fitness.

Termination Condition. If the number of training iterations is terminated or the accuracy is satisfied, then output Gbest and Pbest locations, and the algorithm terminates. Otherwise, go to Step (4). When the number of training iterations is reached, the PSO computation is terminated.

In our experiment, the swarm populations sizepop is set to 20 and the number maxgen of training iterations is set to 35, the total number of parameter p or g can thus be calculated as $2\times sizepop+ (maxgen-1) \times sizepop= 2 \times 20 + (35-1) \times 20 = 720$. The final optimal value of p or g corresponds to the maximum value of the 720 candidate particle values indicates the selected value for the two main parameters. The enhanced image with the two final parameter values is our final fog removal result.

Iterative Optimization. Calculate the movement vectors in Eq. (3) for all particles. Next, modify the locations of all particles utilizing Eq. (4) and then go to Step (2). The movement vector (location) is specified as follows:

$$\begin{aligned} {\mathrm{{V}}_i}(t\,+\,1) = w{\mathrm{{V}}_i}(t) + {c_1} \times {r_1} \times (Pbes{t_i} - {\mathrm{{X}}_i}(t)) + {c_2} \times {r_2} \times (Gbest - {\mathrm{{X}}_i}(t)) \end{aligned}$$

(3)

where ${\mathrm{{V}}_i} = ({V_{i1}}, {V_{i2}}, ..., {V_{im}}) \in {\mathfrak {R}^m}$ , and the particle’s initial velocity is 0.5 $\times$ RV. ${\mathrm{{V}}_i}(t\,+\,1)$ represents the movement vector of particle i at the (t+1)th iteration, w indicates the inertia weight (w = 1 in our experiments), $c_{\text {1}}$ and $c_{\text {2}}$ denote the acceleration coefficients which are random numbers in [0,1], we set $c_{\text {1}} = c_2= 0.5$ for all results reported in this paper. $r_1$ and $r_{2}$ are also two randomly generated values in [0,1]. Moreover, in Eq. (3), the first term $w{\mathrm{{V}}_i}(t)$ denotes the particle’s inertia, the second term ${c_1} \times {r_1} \times (Pbes{t_i} - {\mathrm{{X}}_i}(t))$ indicates the particle’s cognition-only model, and the third term ${c_2} \times {r_2} \times (Gbest - {X_i}(t))$ stands for the particle’s social-only model. The location of particle i is modified by Eq. (4).

$$\begin{aligned} {\mathrm{{X}}_i}(t\,+\,1) = {\mathrm{{X}}_i}(t) + {\mathrm{{V}}_i}(t\,+\,1) \end{aligned}$$

(4)

where ${\mathrm{{X}}_i}(t\,+\,1)$ represents the location of particle i at the $(t+1)$th iteration, which is used to estimate the optimal value of the two main parameters. ${\mathrm{{V}}_i}(t\,+\,1)$ denotes the movement vector of particle i at the (t+1)th iteration. Hence, the new location of particle i is to add its current location vector to its movement vector. Therefore, we can deduce that the iterative optimization operation is designed to suit our needs regarding good image defogging effect with relatively large CNC index value and maintaining the diversity at the same time.

4 Experimental Results

The publicly available dataset frida2 [14] is used to evaluate image defogging methods. Although the testing images in this dataset are very similar in view structure, the reason why we use the this dataset images as our testing images is that the dataset contains synthetic no-fog images and associated foggy images for 66 diverse road scenes, while capturing the fog and no-fog natural images with the same scenes is very hard in most cases. The absolute difference (AD) on the images between defogged images and target images without fog is used as performance metric, and good results are described by small value of AD. To verify the effectiveness and validity of the proposed parameter value selection method, three criteria have been considered: (i) generation number influence, (ii) qualitative comparison, and (iii) quantitative evaluation. In the experiments, all the results are obtained by executing Matlab R2008a on a PC with 3.10 GHz Intel® CoreTM i5-2400 CPU.

4.1 Generation Number Evaluation

The final condition of the proposed PSO-based algorithm is to reach the given number maxgen of training iterations. To evaluate the influence of the iteration number maxgen used in the proposed method, some group experiments are performed by varying the iteration number maxgen from 20 to 50, the parameter values and the resulting AD metics of a test image are presented in Table 1. One can clearly see that the results are visually and statistically close (the value range of AD are [60.7332, 61.9503] for He’s method, and [42.2918, 43.7579] for Tarel’s method) when varying maxgen from 20 to 50. It demonstrates that the influence of the iteration number is very limited in the proposed method. The experiments on a large quantity of other test images also confirm the observations.

Table 1. AD metics of the two representative defogging methods under maxgen from 20 to 50

4.2 Qualitative Comparison

For further evaluation of the PSO-based method for selecting the most proper parameter values, the image dataset provide by Tarel et al. [15] is used to validate the accuracy of the parameter selection, since this dataset provides the original image without fog and the image with fog for 66 road scenes simultaneously. For each image, we obtain the fog removal results with the default and the auto-adaptive parameter values for He’s and Tarel’s algorithms. We also give the corresponding fog-free images as the reference images for comparison. An illustrative example is shown in Fig. 5. One can clearly see that the defogging results obtained using the auto-adaptive values achieve a better enhancement effect compared to the results obtained using the default values for both defogging algorithms. Note that the color of He’s results seems much dark than that of no-fog reference images, and there are some halo artifacts in Tarel’s results. All these problems are caused by the defogging algorithm itself, not the proposed parameter selection method. The accuracy of the PSO-based parameter selection can be validated by determining the most proper parameter values and producing the best results for each defogging method.

Fig. 5.

Visual comparison of defogging results for public database frida2 [14]. (a) Foggy images. (b) Fog-free images. (c) He’s results obtained using default parameter values ($\omega = 0.95$, $t_{0} = 0.1$, $c=3$). (d) He’s results obtained using auto-adaptive parameter values. (e) Tarel’s results obtained using default parameter values ($p = 0.95$, $b = 0.5$, $s_v = 9$, $s_{i} = 1$, $g = 1.3$). (f) Tarel’s results obtained using auto-adaptive parameter values

4.3 Quantitative Evaluation

To quantitatively assess the proposed parameter value selection method, we compute the AD index value for the images in Fig. 5, and the statistical results are shown in Table 2. One can notice that the AD value obtained by the auto-adaptive parameter is smaller than that of default parameters for both defogging algorithms, which means that the better defogging effect can be obtained by using the proposed method. This confirms our observations in Fig. 5.

Table 2. AD index between enhanced images and no-fog images for the testing images in Fig. 5

Fig. 6.

AD index results for the 66 test images in public database frida2 [15]. (a) He’s statistical results. (b) Tarel’s statistical results (Color figure online)

The AD index is also tested for more test images in public database frida2 (66 images). Figure 6(a) shows the statistical results of the AD for He’s method and Fig. 6(b) shows the AD results for Tarel’s method. In Fig. 6, circle “” stands for foggy image, circle “” stands for the defogging image obtained using the default parameter values, and circle “” stands for the defogging image obtained by the auto-adaptive parameter values. The horizontal axes are the AD index values and vertical axes are the image number index. It is clear that the ADs of adaptive parameter results are smaller than that of other results for both defogging methods. This indicates that the fog removal results obtained by the proposed parameter value selection method have better defogging effect for the public image database compared to the other results. This is also consistent with the assessment results of AD and human visual perception.

5 Conclusions

In this paper, a novel PSO-based parameter value selection method was proposed. Different from the most defogging methods which generally fix the parameter values, the proposed approach can help defogging methods automatically select optimal parameter values for different foggy images. In the proposed method, the two main parameters which directly affect the results are first distinguished from other less important parameters which can be considered as fixed values. Then, the two parameter values are adaptively determined by using the PSO algorithm. The proposed parameter selection method has been applied to two representative defogging algorithms, which demonstrated the superior performance of the proposed scheme in both qualitatively and quantitatively. Although the proposed method provided a new way to solve the parameter adjustment problem for single image defogging, in the future, we will try to investigate the parameter value selection issue based on more advanced assessment index, since the CNC index may not be the best one to measure image defogging effect.