Skip to main content
SpringerLink
Log in

Detection of surface defects on pharmaceutical solid oral dosage forms with convolutional neural networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Deep-learning-based approaches have proven to outperform other approaches in various computer vision tasks, making application-focused machine learning a promising area of research in automated visual inspection. In this work, we apply deep learning to the challenging real-world problem domain of automated visual inspection of pharmaceutical products. We focus on investigating whether compact network architectures, adhering to performance, resource, and accuracy requirements, are suitable for usage in the pharmaceutical visual inspection domain. We propose a compact and efficient convolutional neural network architecture design for segmentation and scoring of surface defects, which we evaluate on challenging real-world datasets from the pharmaceutical product-inspection domain. In comparison with other related segmentation approaches, we achieve state-of-the-art performance in terms of defect detection as well as real-time computational efficiency. Compared to the nearest best-performing architecture we achieve state-of-the-art performance with merely 3% of the parameter count, an approximately 8-fold increase in inference speed, and increased surface defect detection performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. SPINE, www.sensum.eu/product/spine

  2. https://hci.iwr.uni-heidelberg.de/content/weakly-supervised-learning-industrial-optical-inspection

  3. https://www.vicos.si/resources/kolektorsdd

  4. https://www.sensum.eu/sensumsodf-dataset

References

  1. Berman A (2004) Reducing medication errors through naming, labeling, and packaging. J Med Syst 28:9–29

    Article  Google Scholar 

  2. FDA-21CFR206 (2015) Imprinting of solid oral dosage form drug products for human use

  3. JPMA (2018) Pharmaceutical administration and regulations in Japan

  4. Karloff AC, Scott NE, Muscedere R (2008) A flexible design for a cost effective, high throughput inspection system for pharmaceutical capsules. IEEE international conference on industrial technology, pages 1–4,

  5. Bukovec M, žpiclin, F. Pernuš, and B. Likar, (2007) Automated visual inspection of imprinted pharmaceutical tablets. Measur Sci Technol 18:2921–2930

    Article  Google Scholar 

  6. Islam MJ, Ahmadi M, Sid-Ahmed MA (2008) Image processing techniques for quality inspection of gelatin capsules in pharmaceutical applications. 10th international conference on control, automation, robotics and vision, pp. 862–867

  7. Mehle Andraž, Bukovec Marko, Likar Boštjan, Tomaževič Dejan (2016) Print registration for automated visual inspection of transparent pharmaceutical capsules. Mach Vis Appl 27:1087–1102

    Article  Google Scholar 

  8. Ziga Špiclin, Boštjan Likar, and Franjo PernuŠ (2010) Real-time print localization on pharmaceutical capsules for automatic visual inspection. IEEE international conference on industrial technology, pp. 279–284

  9. LeCun Yann, Bengio Yoshua, Hinton Geoffrey (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  10. Hearst Marti A (1998) Support vector machines. IEEE Intell Syst 13(4):18–28

    Article  Google Scholar 

  11. Ye J (2007) Least squares linear discriminant analysis. In: Proceedings of the 24th international conference on machine learning, ICML ’07, New York, NY, USA. ACM, pp. 1087–1093

  12. Breiman L, Friedman J, Stone CJ, Olshen RA (1984) Classification and regression trees. The Wadsworth and Brooks-Cole statistics-probability series. Taylor & Francis, UK

    Google Scholar 

  13. Cover T, Hart P (2006) Nearest neighbor pattern classification. IEEE Trans Inf Theor 13(1):21–27

    Article  Google Scholar 

  14. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. In: Proceedings of the IEEE, pp. 2278–2324

  15. Mehle A, Likar B, Tomaževič D (2017) In-line recognition of agglomerated pharmaceutical pellets with density-based clustering and convolutional neural network. In: 2017 Fifteenth IAPR international conference on machine vision applications (MVA), pp. 9–12

  16. Weimer D, Scholz-Reiter B, Shpitalni M (2016) Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection. In: CIRP annals—manufacturing technology 65(1), pp. 417–420

  17. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  18. Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Computer Vision – ECCV 2018, pp. 833–851, Cham: Springer International Publishing

  19. Zhao Hengshuang (2017) Jianping Shi. Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, Xiaojuan Qi

  20. Badrinarayanan V, Handa A, Cipolla R (2015) Segnet: a deep convolutional encoder-decoder architecture for robust semantic pixelwise labelling. arXiv:1505.07293

  21. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. pp. 234–241

  22. AO Martins Luiz, LC Pádua Flávio, and EM Almeida Paulo (2010) Automatic detection of surface defects on rolled steel using computer vision and artificial neural networks. In: IECON 2010-36th annual conference on IEEE industrial electronics society, pp. 1081–1086. IEEE

  23. Masci J, Meier U, Ciresan D, Schmidhuber J, Fricout G(2012) Steel defect classification with max-pooling convolutional neural networks. In: The 2012 international joint conference on neural networks (IJCNN), pp. 1–6. IEEE

  24. Daniel Soukup and Reinhold Huber-Mörk (2014) Convolutional neural networks for steel surface defect detection from photometric stereo images. In: International symposium on visual computing. Springer, pp. 668–677

  25. Faghih-Roohi S, Hajizadeh S, Núñez A, Babuska R, Bart De Schutter (2016) Deep convolutional neural networks for detection of rail surface defects. In: Neural Networks (IJCNN), 2016 international joint conference on, pp. 2584–2589. IEEE

  26. Chen S, Feng J, Zou L (2010) Study of fabric defects detection through gabor filter based on scale transformation. In: Image analysis and signal processing (IASP), 2010 international conference on, pp. 97–99. IEEE

  27. Kai-Ling Mak P, Peng, and KFC Yiu, (2009) Fabric defect detection using morphological filters. Image Vis Comput 27(10):1585–1592

    Article  Google Scholar 

  28. Polzleitner W (2001) Defect detection on wooden surface using gabor filters with evolutionary algorithm design. In: Neural Networks, 2001. Proceedings. IJCNN’01. international joint conference on, volume 1, pp. 750–755. IEEE

  29. Chen P-H, Ho S-S (2016) Is overfeat useful for image-based surface defect classification tasks?. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 749–753

  30. Krizhevsky A, Sutskever I, Hinton GE (2016) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (Eds.) Advances in neural information processing systems 25, pages 1097–1105. Curran Associates, Inc

  31. Lin Hui, Li Bin, Wang Xinggang, Shu Yufeng, Niu Shuanglong (2019) Automated defect inspection of led chip using deep convolutional neural network. J Intell Manuf 30:1–10

    Article  Google Scholar 

  32. Wang B, Cao G, Shang Y, Zhou L, Zhang Y, Li X (2018) Single-column cnn for crowd counting with pixel-wise attention mechanism. Neural Computing and Applications

  33. Zhu L, Li C, Yang Z, Yuan K, Wang S (2019) Crowd density estimation based on classification activation map and patch density level. Neural Computing and Applications

  34. Mohamed IS, Capitanelli A, Mastrogiovanni F, Rovetta S, Zaccaria R (2019) Detection, localisation and tracking of pallets using machine learning techniques and 2d range data. Neural Computing and Applications

  35. Garcia-Garcia A, Orts S, Oprea S, Martinez VV, Rodríguez J (2017) A review on deep learning techniques applied to semantic segmentation

  36. Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), volume 2, pp. 2169–217

  37. Lucchi A, Li Y, Boix X, Smith K, Fua P (2011) Are spatial and global constraints really necessary for segmentation?. In: 2011 international conference on computer vision, pp. 9–16

  38. Rački D, Tomaževič D, Skočaj D (2018) A compact convolutional neural network for textured surface anomaly detection. In: 2018 IEEE winter conference on applications of computer vision (WACV), pp. 1331–1339

  39. Martinel N, Foresti GL, Micheloni C (2018) Wide-slice residual networks for food recognition. In: 2018 IEEE Winter conference on applications of computer vision (WACV), pp. 567–576

  40. Onchis Darian M, Gillich Gilbert-Rainer (2021) Stable and explainable deep learning damage prediction for prismatic cantilever steel beam. Comput Ind 125:103359

    Article  Google Scholar 

  41. Huang Y, Qiu C, Wang X, Wang S, Yuan K (2020) A compact convolutional neural network for surface defect inspection. Sensors 20(7):1974

    Article  Google Scholar 

  42. Lin Zesheng, Ye Hongxia, Zhan Bin, Huang Xiaofeng (2020) An efficient network for surface defect detection. Appl Sci 10:6085

    Article  Google Scholar 

  43. Jianbo Yu, Zheng Xiaoyun, Liu Jiatong (2019) Stacked convolutional sparse denoising auto-encoder for identification of defect patterns in semiconductor wafer map. Comput Ind 109:121–133

    Article  Google Scholar 

  44. Tabernik Domen, Šela Samo, Skvarč Jure, Skočaj Danijel (2020) Segmentation-based deep-learning approach for surface-defect detection. J Intell Manuf 31:759–776

    Article  Google Scholar 

  45. Dezhong Z, Wenhu W, Jinhua Z, Ruisong J, Kang C, Qichao J (2018) Measurement point sampling method for inspection of parts with free-form surfaces. Adv Mech Eng 10(11):1687814018809577

    Google Scholar 

  46. Kim S, Kim W, Noh Y, Park FC (2017) Transfer learning for automated optical inspection. In: 2017 international joint conference on neural networks (IJCNN)

  47. Chen L-C, Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation. arXiv arXiv:1706.05587

  48. Dong Xinghui, Taylor Christopher J, Cootes Tim F (2020) Defect detection and classification by training a generic convolutional neural network encoder. IEEE Trans Signal Process 68(1):6055–6069

    Article  MathSciNet  Google Scholar 

  49. Yang Yatao, Yang Runze, Pan Longhui, Ma Junxian, Zhu Yishuang, Diao Tao, Zhang Li (2020) A lightweight deep learning algorithm for inspection of laser welding defects on safety vent of power battery. Comput Ind 123:103306

    Article  Google Scholar 

  50. Liu Yang, Yuan Yachao, Balta Cristhian, Liu Jing (2020) A light-weight deep-learning model with multi-scale features for steel surface defect classification. Materials 13(20):1–13

    Google Scholar 

  51. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp. 4510–4520

  52. Bergmann P, Fauser M, Sattlegger D, Steger C (2019) MVTec AD - A comprehensive real-world dataset for unsupervised anomaly detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  53. Shah S, Ghosh P, Davis L, Goldstein T (2018) Stacked u-nets: a no-frills approach to natural image segmentation, 04

  54. Kiseǐák J, Lu Y, Svihra J, Peter S, Stehlík M (2020) SPOCU: scaled polynomial constant unit activation function. Neural Computing and Applications, 07

  55. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp. 1026–1034

  56. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. International Conference on Learning Representations, 12

  57. Russakovsky Olga, Deng Jia, Hao Su, Krause Jonathan, Satheesh Sanjeev, Ma Sean, Huang Zhiheng, Karpathy Andrej, Khosla Aditya, Bernstein Michael, Berg Alexander, Li Fei Fei (2014) Imagenet large scale visual recognition challenge. Int J Comput Vis 115:09

    MathSciNet  Google Scholar 

  58. Lin T-Y, Maire M, Belongie S, Perona P, Ramanan D, Dollár P, Zitnick C (2014) Microsoft coco: common objects in context. 05

  59. Wang Tian, Chen Yang, Qiao Meina, Snoussi Hichem (2018) A fast and robust convolutional neural network-based defect detection model in product quality control. The Int J Adv Manuf Technol 94(9):3465–3471

    Article  Google Scholar 

  60. Cognex ViDi (commercial software). Cognex (2018) visionpro vidi: Deep learning-based software for industrial image analysis. url https: www.cognex.com/products/machine-vision/vision-software/visionpro-software

  61. Liu G, Yang N, Guo L, Guo S, Chen Z (2020) A one-stage approach for surface anomaly detection with background suppression strategies. Sensors, 20(7)

Download references

Acknowledgements

This work was supported by the Ministry of Economic Development and Technology (MGRT), Republic of Slovenia; the European Union, European Regional Development Fund (ERDF) under grant 631-63|2017|1; and Sensum, Computer Vision Systems.

Author information

Authors and Affiliations

  1. Sensum, Computer Vision Systems, Tehnološki park 21, SI-1000, Ljubljana, Slovenia

    Domen Rački & Dejan Tomaževič

  2. Faculty of Electrical Engineering, University of Ljubljana, Tržaška cesta, SI-1000, Ljubljana, Slovenia

    Dejan Tomaževič

  3. Faculty of Computer and Information Science, University of Ljubljana, Večna-pot, SI-1000, Ljubljana, Slovenia

    Danijel Skočaj

Authors
  1. Domen Rački
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dejan Tomaževič
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Danijel Skočaj
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Domen Rački.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rački, D., Tomaževič, D. & Skočaj, D. Detection of surface defects on pharmaceutical solid oral dosage forms with convolutional neural networks. Neural Comput & Applic 34, 631–650 (2022). https://doi.org/10.1007/s00521-021-06397-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06397-6

Keywords

Search

Navigation