Skip to main content
SpringerLink
Log in

Analyzing the Potential of Zero-Shot Recognition for Document Image Classification

  • Conference paper
  • First Online:
Document Analysis and Recognition – ICDAR 2021 (ICDAR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12824))

Included in the following conference series:

Abstract

Document image classification is one of the most important components in business automation workflow. Therefore, a range of different supervised image classification methods have been proposed, which rely on a large amount of labeled data, which is rarely available in practice. Furthermore, retraining of these models is necessary upon the introduction of new classes. In this paper, we analyze the potential of zero-shot document image classification based on computing the agreement between the images and the textual embeddings of class names/descriptions. This enables the deployment of document image classification models without the availability of any training data at zero training cost, alongside providing seamless integration of new classes. Our results show that using zero-shot recognition achieves significantly better than chance performance on document image classification benchmarks (49.51% accuracy on Tobacco-3482 in contrast to 10% random classifier accuracy and 39.22% on RVL-CDIP dataset in contrast to 6.25% random classifier accuracy). We also show that the representation learned by a vision transformer using image-text pairs is competitive to CNNs by training a linear SVM on top of the pre-computed representations which achieves comparable performance to state-of-the-art convolutional networks (85.74% on Tobacco-3482 dataset in contrast to 71.67% from ImageNet pretrained ResNet-50). Even though the initial results look encouraging, there is still a large gap to cover for zero-shot recognition within the domain of document images in contrast to natural scene image classification, which achieves comparable performance to fully supervised baselines. Our preliminary findings pave the way for the deployment of zero-shot document image classification in production settings.

This work was supported by the BMBF project DeFuseNN (Grant 01IW17002) and the NVIDIA AI Lab (NVAIL) program.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Afzal, M.Z., et al.: Deepdocclassifier: document classification with deep convolutional neural network. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 1111–1115 (2015). https://doi.org/10.1109/ICDAR.2015.7333933

  2. Afzal, M.Z., Kölsch, A., Ahmed, S., Liwicki, M.: Cutting the error by half: investigation of very deep CNN and advanced training strategies for document image classification. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 883–888. IEEE (2017)

    Google Scholar 

  3. Asim, M.N., Khan, M.U.G., Malik, M.I., Razzaque, K., Dengel, A., Ahmed, S.: Two stream deep network for document image classification. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1410–1416. IEEE (2019)

    Google Scholar 

  4. Baevski, A., Zhou, H., Mohamed, A., Auli, M.: wav2vec 2.0: a framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477 (2020)

  5. Chen, N., Blostein, D.: A survey of document image classification: problem statement, classifier architecture and performance evaluation. IJDAR 10(1), 1–16 (2007)

    Article  Google Scholar 

  6. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 (2020)

  7. Chen, Y., et al.: Bayesian optimization in AlphaGo. arXiv preprint arXiv:1812.06855 (2018)

  8. Cosma, A., Ghidoveanu, M., Panaitescu-Liess, M., Popescu, M.: Self-supervised representation learning on document images. arXiv preprint arXiv:2004.10605 (2020)

  9. Das, D., Lee, C.G.: Zero-shot image recognition using relational matching, adaptation and calibration. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2019)

    Google Scholar 

  10. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  11. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  12. Ferrando, J., et al.: Improving accuracy and speeding up document image classification through parallel systems. In: Krzhizhanovskaya, V.V., et al. (eds.) ICCS 2020. LNCS, vol. 12138, pp. 387–400. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50417-5_29

    Chapter  Google Scholar 

  13. Grill, J.B., et al.: Bootstrap your own latent: a new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 (2020)

  14. Harley, A.W., Ufkes, A., Derpanis, K.G.: Evaluation of deep convolutional nets for document image classification and retrieval. In: International Conference on Document Analysis and Recognition (ICDAR) (2015)

    Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  16. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  17. Kang, L., Kumar, J., Ye, P., Li, Y., Doermann, D.: Convolutional neural networks for document image classification. In: 2014 22nd International Conference on Pattern Recognition, pp. 3168–3172. IEEE (2014)

    Google Scholar 

  18. Kölsch, A., Afzal, M.Z., Ebbecke, M., Liwicki, M.: Real-time document image classification using deep CNN and extreme learning machines. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 1318–1323. IEEE (2017)

    Google Scholar 

  19. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  20. Kumar, J., Doermann, D.: Unsupervised classification of structurally similar document images. In: 2013 12th International Conference on Document Analysis and Recognition, pp. 1225–1229. IEEE (2013)

    Google Scholar 

  21. Kumar, J., Ye, P., Doermann, D.: Learning document structure for retrieval and classification. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR 2012), pp. 1558–1561. IEEE (2012)

    Google Scholar 

  22. Kumar, J., Ye, P., Doermann, D.: Structural similarity for document image classification and retrieval. Pattern Recogn. Lett. 43, 119–126 (2014)

    Article  Google Scholar 

  23. Noce, L., Gallo, I., Zamberletti, A., Calefati, A.: Embedded textual content for document image classification with convolutional neural networks. In: Proceedings of the 2016 ACM Symposium on Document Engineering, pp. 165–173 (2016)

    Google Scholar 

  24. Radford, A., et al.: Learning transferable visual models from natural language supervision. Image 2, T2 (2021)

    Google Scholar 

  25. Reddy, K.V.U., Govindaraju, V.: Form classification. In: Yanikoglu, B.A., Berkner, K. (eds.) Document Recognition and Retrieval XV, vol. 6815, pp. 302–307. International Society for Optics and Photonics, SPIE (2008). https://doi.org/10.1117/12.766737

  26. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  27. Sarkar, P.: Learning image anchor templates for document classification and data extraction. In: 2010 20th International Conference on Pattern Recognition, pp. 3428–3431. IEEE (2010)

    Google Scholar 

  28. Shin, C., Doermann, D., Rosenfeld, A.: Classification of document pages using structure-based features. Int. J. Doc. Anal. Recogn. 3(4), 232–247 (2001)

    Article  Google Scholar 

  29. Shin, C.K., Doermann, D.S.: Document image retrieval based on layout structural similarity. In: IPCV, pp. 606–612 (2006)

    Google Scholar 

  30. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  31. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261 (2016)

  32. Vaswani, A., et al.: Attention is all you need. arXiv preprint arXiv:1706.03762 (2017)

Download references

Author information

Authors and Affiliations

  1. German Research Center for Artificial Intelligence (DFKI), 67663, Kaiserslautern, Germany

    Shoaib Ahmed Siddiqui, Andreas Dengel & Sheraz Ahmed

  2. TU Kaiserslautern, 67663, Kaiserslautern, Germany

    Shoaib Ahmed Siddiqui & Andreas Dengel

  3. DeepReader GmbH, 67663, Kaiserlautern, Germany

    Sheraz Ahmed

Authors
  1. Shoaib Ahmed Siddiqui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andreas Dengel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sheraz Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shoaib Ahmed Siddiqui .

Editor information

Editors and Affiliations

  1. Universitat Autònoma de Barcelona, Barcelona, Spain

    Josep Lladós

  2. Lehigh University, Bethlehem, PA, USA

    Daniel Lopresti

  3. Kyushu University, Fukuoka-shi, Japan

    Seiichi Uchida

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Siddiqui, S.A., Dengel, A., Ahmed, S. (2021). Analyzing the Potential of Zero-Shot Recognition for Document Image Classification. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12824. Springer, Cham. https://doi.org/10.1007/978-3-030-86337-1_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86337-1_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86336-4

  • Online ISBN: 978-3-030-86337-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation