Keywords

1 Introduction

Foreign objects (as shown in Fig. 1) such as wrenches, rubber pieces of tires, screws, stones, and metal, are named foreign object debris (FOD) which may cause foreign object damage and seriously threaten flight safety [7]. On July 25, 2000, a concord flight operated by Air France was crashed by a metal strip and one hundred and thirteen people lost their lives in this accident [4]. Therefore, FOD detection and classification are very important for preventing such concord disaster.

Fig. 1.
figure 1

Examples of FOD on the airport (collected from the internet). (a) Wrench; (b) Tire debris; (c) Screw; (d) Stone; (e) Metal; and (f) Others.

Full size image

Since 2000, a variety of countries began to develop technology for FOD detection. So far there are four detection systems have been successfully applied in airports in field worldwide. They are Tarsier Radar system developed by United Kingdom (UK) [2], FODetect system by Israel [3], FOD Finder system by United States (US) [1], and iFerret system by Singapore [26]. These systems employed different methods and sensors for detecting FOD and giving promising results on the detection of FOD, but they cannot directly classify FOD. Even though some systems can obtain the FOD images, they will not automatically complete the FOD classification. If the characteristics of different types of foreign objects are used to help the classification of FOD, it will show great significance on the prevention of FOD damage and can provide the utilization rate of the airport runway.

In recent years, many methods for the classification of FOD on airports have been developed. As we know, a detection or classification algorithm is often influenced by the feature extraction algorithm and the classifier performance [14, 15]. Therefore, most of the FOD classification approaches mainly focus the study on the novel feature extraction algorithm or the efficient classifier. Wang et al. [29] proposed a foreign object debris detection and identification method, in which the Gabor features are extracted and the nearest neighbor (NN) classifier is used to obtain the final classification results. Niu et al. [25] proposed a FOD detection system which based on Gabor wavelets and support vector machine (SVM). They firstly used Gabor wavelets to extract useful features for describing FOD images and then employed the SVM for the classification. Next, they proposed an improved version using Garbor wavelets and SVM for FOD classification [24]. Xiang et al. [31] proposed a FOD recognition method based on depth features and Adaboost classifier. Han et al. [11, 12] proposed novel FOD classification algorithms based on low-level features and subspace features. Cao et al. [5] proposed a FOD detection framework using region proposal technique and convolution neural networks (CNN).

Although the above methods have achieved promising results, there is much room to improve on the classification accuracy, especially, the features to characterize many different types of FOD are remains to be further investigated. In this paper, a novel FOD classification framework based on the integrated visual features and extreme learning machine (ELM) classifier is developed. The proposed algorithm not only considers color and texture features for the feature extraction of FOD images, but also employs ELM for an efficient classification. The experimental results on both various feature extraction methods and different classifiers demonstrate the performance of our proposed FOD classification algorithm.

The organization of this paper are presented as follows. In Sect. 2, we present the proposed classification algorithm, including the integrated visual feature extraction and the classification using ELM. In Sect. 3, we evaluate the proposed classification algorithm on the FOD dataset from different perspectives. We also compare the proposed algorithm with other state-of-the-art FOD classification methods. Finally, the conclusions of this paper are provided in Sect. 4.

2 Methods

A novel FOD classification algorithm based on the integrated visual features and extreme learning machine is proposed in this paper. The overview of the proposed FOD classification algorithm is shown in Fig. 2.

Fig. 2.
figure 2

The overview of the proposed FOD classification algorithm.

Full size image

2.1 Image Preprocessing

In this paper, the resolution of original FOD images obtained by the FOD detection system is \(1024\times 1024\) pixels. Actually, foreign object only occupied a small area on the whole image. To obtain stable and accurate classification result, we perform image preprocessing on original FOD images. First, an edge detection method is used to find the edge of the foreign objects and then the FOD images are normalized to \(64\times 64\) for the classification [11, 12].

2.2 Integrated Visual Feature Extraction

Color feature. Color is an effective tool for describing human visual properties and it is often used to represent objects in image segmentation and classification. Due to the influences of angle, illumination and shadow, it usually produces variations in the color descriptions and thus leads to the color feature extraction is difficult [21]. However, color has been applied in image classification with high robustness since color do not depend on the deformation and scale variation of the image. In 2007, Weijer et al. [30] proposed a color naming feature extraction method in which the contribution is assigning linguistic color labels to image pixels.

Texture feature. Scale-invariant feature transform (SIFT) proposed by David Lowe [23] is an effective algorithm to characterize local texture features of images. It has been demonstrated by Lowe that the SIFT keypoint descriptors exhibit high performance for local feature representation with invariance of image scale and rotation, illumination change, and image noise. The SIFT feature extraction algorithm contains four steps which are detection of scale-space extrema; localization of keypoints; assignment of orientation; and calculation of keypoint descriptors.

Another useful local texture feature extraction algorithm is histograms of oriented gradient (HOG) which is proposed by Dalal and Triggs [9]. In their method, the image is firstly divided into many small spatial regions (cells). Secondly, the histogram of gradient orientation for each larger spatial regions (blocks) is calculated. The combined histograms are then formed the representations. Finally, after the contrast normalization on all of the cells in the block, the HOG descriptors are obtained.

Feature extraction. In the proposed algorithm, the scale invariant feature transform descriptors [23] and the histograms of oriented gradient descriptors [9] for texture feature as well as the color names feature descriptors [30] are extracted as visual features to characterize the FOD image.

In the actual extraction of color names feature, SIFT feature and HOG feature, we firstly sample the feature at multiple scales with a grid spacing of 4. Secondly, a dictionary with size 100 has been learned. Further, the locality-cnstrained linear coding (LLC) at 2-level spatial pyramid is employed to calculate the feature descriptor for each region [21]. Finally, we combine the color names feature, SIFT feature and HOG feature into an integrated visual feature vector to characterize the FOD image.

2.3 Classification Using ELM

Extreme learning machine proposed by Huang et al. [18] is a single-hidden layer feedforward neural network (SLFN), which has several new features when compared to the traditional neural networks: it has only one hidden layer; the hidden layer need not be tuned; input weights and hidden layer biases of ELM can be chosen randomly; and the output weights of ELM are determined analytically. These special features make the learning of ELM simple and efficient, and this leads to it has both universal approximation [17] and classification capabilities. Therefore, up to now ELM has been widely applied in both academia and industry such as to solve the clustering [19], regression [13], classification [15], detection [28], and feature learning problems [16, 20].

For M samples \(\left( \mathbf {x}_{i},\mathbf {t}_{i} \right) \), L hidden nodes and active function g(x), the output of standard SLFN can be formulated as

$$\begin{aligned} \sum _{i=1}^{L} \beta _{i} g_{i} \left( \mathbf {x}_{j} \right) =\sum _{i=1}^{L} \beta _{i} g \left( \mathbf {w}_{i} \cdot \mathbf {x}_{j} + b_{i} \right) = \mathbf {o}_{j}, j=1,\ldots ,M, \end{aligned}$$
(1)

where \(\mathbf {w}_{i}\) is the weight vector connecting the ith hidden node and the input nodes, while \(\beta _{i}\) is the weight vector between the ith hidden node and the output nodes. \(b_{i}\) is the threshold of ith hidden node. The standard SLFN can approximate the M samples with zero error, that is to say there exist \(\beta _{i}\), \(\mathbf {w}_{i}\) and \(b_{i}\) satisfy

$$\begin{aligned} \sum _{i=1}^{L} \beta _{i} g \left( \mathbf {w}_{i} \cdot \mathbf {x}_{j} + b_{i} \right) = \mathbf {t}_{j}, j=1,\ldots ,M. \end{aligned}$$
(2)

These equations can be rewritten as

$$\begin{aligned} \mathbf {H}\beta =\mathbf {T}, \end{aligned}$$
(3)

where \(\mathbf {H}\) is the output matrix of hidden layer. We can easily get

$$\begin{aligned} \beta =\mathbf {H}^{\dagger }\mathbf {T}, \end{aligned}$$
(4)

where \(\mathbf {H}^{\dagger }\) is the Moore-Penrose generalized inverse of \(\mathbf {H}\).

Training. Before the training, the feature matrix F for the training FOD images is constructed based on the integrated visual feature extraction. The procedure of training using ELM is presented as follows. Firstly, we randomly choose the parameters of hidden layer nodes \(\mathbf {w}_{i}\) and \(b_{i}\). Secondly, according to \(\mathbf {w}_{i}\) and \(b_{i}\), the output matrix \(\mathbf {H}\) is obtained using the feature matrix F. Thirdly, \(\beta \) is obtained with the usage of \(\mathbf {H}\) and the ground truth (labels) of the training FOD images and finally to finish the ELM training.

Testing. After the ELM training, the parameters \(\mathbf {w}_{i}\), \(b_{i}\) and \(\beta \) are obtained that means the ELM classifier for FOD classification has been constructed. In the testing, the integrated visual features of the testing FOD images are firstly extracted. Then we compute the output matrix \(\mathbf {H}\) according to \(\mathbf {w}_{i}\) and \(b_{i}\). Furthermore, the classification of FOD is carried out using \(\beta \).

3 Experimental Results and Discussion

In this section, the performance of the proposed FOD classification algorithm is evaluated using a FOD dataset of which all images are collected by the FOD detection subsystem [11]. The FOD dataset contains five categories for a total of 320 images. It includes 81 images of wrench, 74 images of plastic pip, 64 images of tire debris, 50 images of fuel-tank cap, and 51 images of metal bar. As mentioned before, an edge detection method is used to find the edge of the foreign objects and then the FOD images are normalized to \(64\times 64\). Examples of the FOD images used in the experiments are shown in Fig. 3.

Fig. 3.
figure 3

Examples of FOD images for the classification. (a) Wrench; (b) Plastic pip; (c) Tire debris; (d) Fual-tank cap; and (e) Metal bar.

Full size image

As presented by Han et al. in [11], it is difficult to perform classification on this FOD dataset because that there exist many artefacts such as large intra-class variability, small interclass dissimilarity, occlusions, scale variation, and background clutter in the images. Therefore, in this paper we extract various of types of object characteristics to effectively represent the foreign objects. In the proposed FOD classification algorithm, the color names, SIFT, and HOG features are extracted for each FOD image and then all features are combined into an integrated visual feature vector to characterize the FOD image. Subsequently, we use ELM to classify each image into different FOD categories. Conveniently, we note the proposed classification algorithm as IVF-ELM.

To evaluate the performance of the proposed classification algorithm, a series of experiments are conducted using different feature extraction methods and different classifiers in terms of classification accuracy. In the experiments, we randomly select only 33.3% of images as the testing set and the remainders are used for the training set. The experiments are repeated for 10 times and the average result is computed as the final classification accuracy.

In order to evaluate the effectiveness of the integrated visual features, the classification accuracy of the proposed classification algorithm using integrated visual features as well as using only color names feature, SIFT feature or HOG feature are summarized in Table 1. All the results are obtained using the ELM classifier. From the results, it can be seen that the classification accuracy using the integrated visual features outperforms all that using only the single feature, although the result of wrench is slightly lower than that of HOG. This is because the proposed classification algorithm which uses the integrated visual features to consider the color names, SIFT, and HOG of the FOD, provides a more comprehensive classification performance.

In addition, taking the results of color names, SIFT, and HOG in Table 1 for comparison, it also can be seen that different features show different classification results on various foreign objects. For example, although color names feature is an effective technique to characterize object visual properties, it obtained the lowest result on metal bar which has variations on intra-class description and illumination condition. Since SIFT and HOG features play important roles on local texture feature description, HOG obtained the highest accuracy on wrench and tire debris, and SIFT obtained the highest accuracy on plastic pip and fuel-tank cap. Whereas the integrated visual features obtained the optimal classification result almost in all FOD classes.

Table 1. Comparative results of the proposed algorithm using different feature extraction methods in terms of the classification accuracy.
Full size table
Table 2. Comparative results of the proposed algorithm using different classifiers in terms of the classification accuracy.
Full size table

Furthermore, to evaluate the performance of the ELM classifier, we conduct another experiment using the proposed classification algorithm with different classifiers such as the nearest neighbor, SVM, and ELM. Figure 4 shows the comparison of the proposed algorithm using the different classifiers mentioned above. The quantitative results are given in Table 2. From the results, it can be seen that no matter what the feature extract method is used, ELM performs better than NN and SVM by providing the best classification accuracy. It also can be seen that no matter what the classifier is used, the results based on the integrated visual features higher than those of based on the single feature. The results demonstrate the effectiveness of the use of the integrated visual features and the ELM classifier.

Fig. 4.
figure 4

Comparison of the proposed classification algorithm using the different feature extraction methods and the different classifiers.

Full size image

Moreover, in order to evaluate the overall performance of the proposed classification algorithm, we compare it with other state-of-the-art methods, e.g., the FOD classification method based on low-level features proposed by Han et al. [11] and the classification and regression method proposed by Liaw and Wiener [22]. For convenience, we refer above compared methods as LLFM and CRM, respectively. Comparison of the different classification methods are presented in Table 3. The best result is highlighted using bold font. The results of the LLFM and CRM methods are extracted from the paper of [11]. As seen from Table 3, the proposed IVF-ELM algorithm outperforms the competing methods in terms of the classification accuracy. The accuracy of the proposed IVF-ELM algorithm increases at least 5.04% compared with other state-of-the-art methods.

Table 3. Comparison between the proposed algorithm and other state-of-the-art methods in terms of the classification accuracy.
Full size table

Actually, in this paper, we propose a simple and efficient FOD classification algorithm. The validity of the extracted integrated visual features (including color names, SIFT, and HOG) and the ELM classifier have been demonstrated through the experiments presented above. However, there exist some limitations which can be further improved. Firstly, the integrated visual features can be optimized by the feature dimension reduction or the feature selection [6, 10] to search the most efficient features for characterizing FOD. Secondly, in the experiments we find that other classifiers, e.g., SVM, exhibits the same efficient classification ability in a certain type of FOD, and therefore, an ensemble classifier [8, 27], which combines various classifier properties, can be considered to improve the performance of the proposed FOD classification algorithm on a large dataset.

4 Conclusion

In this paper, a novel FOD classification algorithm using integrated visual features and extreme learning machine is proposed. The integrated visual features, which contribute from color names, SIFT, and HOG, are extracted in the proposed algorithm for the FOD images after the image preprocessing. Next, extreme learning machine, an efficient classifier is employed to classify the FOD images. In the experiment section, we conduct a series of experiments using different feature extraction methods and different classifiers to evaluate the performance of the proposed classification algorithm. The experimental results show that the proposed algorithm outperforms other state-of-the-art methods. The classification accuracy is enhanced by the use of the integrated visual features and the ELM classifier.

Next, we will employ some feature learning algorithms for the feature optimization and therefore to promote the FOD classification performance. Furthermore, we will plan to use a ensemble classifier on a bigger FOD dataset to evaluate our proposed classification algorithm in the future research.