Skip to main content
SpringerLink
Log in

Unsupervised Discovery of Sign Terms by K-Nearest Neighbours Approach

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 Workshops (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12536))

Included in the following conference series:

Abstract

In order to utilize the large amount of unlabeled sign language resources, unsupervised learning methods are needed. Motivated by the successful results of unsupervised term discovery (UTD) in spoken languages, here we explore how to apply similar methods for sign terms discovery. Our goal is to find the repeating terms from continuous sign videos without any supervision. Using visual features extracted from RGB videos, we show that a k-nearest neighbours based discovery algorithm designed for speech can also discover sign terms. We also run experiments using a baseline UTD algorithm and comment on their differences.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    github.com/bootphon/tdev2.

References

  1. Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1302–1310 (2017)

    Google Scholar 

  2. McKee, D., McKee, R., Alexander, S.P., Pivac, L.: The online dictionary of New Zealand sign language (2015). http://nzsl.vuw.ac.nz

  3. Dunbar, E., et al.: The zero resource speech challenge 2017. In: Proceeding of IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 323–330 (2017)

    Google Scholar 

  4. Jansen, A., Durme, B.V.: Efficient spoken term discovery using randomized algorithms. In: Proceeding of IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 401–406 (2011)

    Google Scholar 

  5. Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUS. arXiv preprint arXiv:1702.08734 (2017)

  6. Kamper, H.: Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models. In: Proceeding of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6535–3539 (2019)

    Google Scholar 

  7. Kamper, H., Elsner, M., Jansen, A., Goldwater, S.: Unsupervised neural network based feature extraction using weak top-down constraints. In: Proceeding of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5818–5822 (2015)

    Google Scholar 

  8. Kamper, H., Jansen, A., Goldwater, S.: Unsupervised word segmentation and lexicon discovery using acoustic word embeddings. IEEE/ACM Trans. Audio, Speech, Lang. Process. 24(4), 669–679 (2016)

    Article  Google Scholar 

  9. Kelly, D., Mc Donald, J., Markham, C.: Weakly supervised training of a sign language recognition system using multiple instance learning density matrices. IEEE Trans. Syst. Man, Cybern. Part B (Cybernetics) 41(2), 526–541 (2011)

    Article  Google Scholar 

  10. Koller, O., Forster, J., Ney, H.: Continuous sign language recognition: towards large vocabulary statistical recognition systems handling multiple signers. Comput. Vis. Image Understanding 141, 108–125 (2015)

    Article  Google Scholar 

  11. Koller, O., Ney, H., Bowden, R.: Deep hand: how to train a CNN on 1 million hand images when your data is continuous and weakly labelled. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3793–3802 (2016)

    Google Scholar 

  12. Koller, O., Zargaran, S., Ney, H.: Re-Sign: re-aligned end-to-end sequence modelling with deep recurrent CNN-HMMs. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3416–3424 (2017)

    Google Scholar 

  13. Kristoffersen, J.H., et al.: Ordbog over Dansk Tegnsprog (2008). http://www.tegnsprog.dk

  14. Levin, K., Henry, K., Jansen, A., Livescu, K.: Fixed-dimensional acoustic embeddings of variable-length segments in low-resource settings. In: Proceeding of IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 410–415 (2013)

    Google Scholar 

  15. Ludusan, B., et al.: Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems. In: Proceeding of Language Resources and Evaluation Conference (2014)

    Google Scholar 

  16. Nayak, S., Duncan, K., Sarkar, S., Loeding, B.L.: Finding recurrent patterns from continuous sign language sentences for automated extraction of signs. J. Mach. Learn. Res. 13, 2589–2615 (2012)

    MathSciNet  Google Scholar 

  17. Ondel, L., Burget, L., Černocký, J.: Variational inference for acoustic unit discovery. Procedia Comput. Sci. 81, 80–86 (2016)

    Article  Google Scholar 

  18. Park, A.S., Glass, J.R.: Unsupervised pattern discovery in speech. IEEE Trans. Audio Speech Lang. Process. 16(1), 186–197 (2008)

    Article  Google Scholar 

  19. Pfister, T., Charles, J., Zisserman, A.: Large-scale learning of sign language by watching TV (using co-occurrences). In: Proceedings of the British Machine Vision Conference, pp. 1–11 (2013)

    Google Scholar 

  20. Polat, K., Saraçlar, M.: Unsupervised term discovery for continuous sign language. In: Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pp. 189–196. European Language Resources Association (ELRA), Marseille, France (2020)

    Google Scholar 

  21. Sutton, V.: Sign writing. Deaf Action Committee (DAC) for Sign Writing (2000)

    Google Scholar 

  22. Theodorakis, S., Pitsikalis, V., Maragos, P.: Dynamic-static unsupervised sequentiality, statistical subunits and lexicon for sign language recognition. Image Vis. Comput. 32, 533–549 (2014)

    Article  Google Scholar 

  23. Thual, A., Dancette, C., Karadayi, J., Benjumea, J., Dupoux, E.: A k-nearest neighbours approach to unsupervised spoken term discovery. In: Proceeding of 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 491–497 (2018)

    Google Scholar 

  24. Tornay, S., Magimai-Doss, M.: Subunits inference and lexicon development based on pairwise comparison of utterances and signs. Information 10, 298 (2019)

    Article  Google Scholar 

  25. Versteegh, M., et al.: The zero resource speech challenge 2015. In: Proceeding of Interspeech, pp. 3169–3173 (2015)

    Google Scholar 

  26. Yin, P., Starner, T., Hamilton, H., Essa, I., Rehg, J.M.: Learning the basic units in American sign language using discriminative segmental feature selection. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4757–4760 (2009)

    Google Scholar 

Download references

Acknowledgments

This work is supported in part by the Scientific and Technological Research Council of Turkey (TUBITAK) under Project 117E059.

Author information

Authors and Affiliations

  1. Boğaziçi University, Istanbul, Turkey

    Korhan Polat & Murat Saraçlar

Authors
  1. Korhan Polat
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Murat Saraçlar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Korhan Polat .

Editor information

Editors and Affiliations

  1. University of Clermont Auvergne, Clermont Ferrand, France

    Adrien Bartoli

  2. Università degli Studi di Udine, Udine, Italy

    Andrea Fusiello

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Polat, K., Saraçlar, M. (2020). Unsupervised Discovery of Sign Terms by K-Nearest Neighbours Approach. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12536. Springer, Cham. https://doi.org/10.1007/978-3-030-66096-3_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-66096-3_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-66095-6

  • Online ISBN: 978-3-030-66096-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation