Skip to main content
SpringerLink
Log in

KPML: A Novel Probabilistic Perspective Kernel Mahalanobis Distance Metric Learning Model for Semi-supervised Clustering

  • Conference paper
  • First Online:
Database and Expert Systems Applications (DEXA 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12392))

Included in the following conference series:

Abstract

Metric learning aims to transform features of data into another based on some given distance relationships, which may improve the performances of distance-based machine learning models. Most existing methods use the difference between the distance of similar pairs and that of dissimilar pairs as loss functions for training. This kind of loss function may lack interpretability since people can only observe the distance or the difference of the distance, a number with no bounds, but have no idea about how large or small it is. To provide more explanation of these metric learning models, in this paper, we propose the probabilistic theoretical analysis of metric learning, design a special loss function, and propose the Kernelized Probabilistic Metric Learning (KPML) approach. With all the distance values transformed into probabilities, we can, therefore, compare and explain the results of the model. Besides, to effectively make use of both the labeled and unlabeled data to enhance the performance of semi-supervised clustering, we propose a KPML-based approach that leverages metric learning and semi-supervised learning effectively in a novel way. Finally, we use our model to do experiments about kNN-based semi-supervised clustering and the results show that our model significantly outperforms baselines across various datasets.

This work was supported by the National Key R&D Program of China [2018YFB1004700]; the National Natural Science Foundation of China [61872238, 61972254]; the Tencent Joint Research Program, and the Open Project Program of Shanghai Key Laboratory of Data Science (No. 2020090600001). The authors also would like to thank Mingding Liao for his contribution on the early version of this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html.

  2. 2.

    http://vision.ucsd.edu/content/yale-face-database.

  3. 3.

    https://www.csie.ntu.edu.tw/cjlin/libsvmtools/datasets/multiclass.html.

  4. 4.

    http://www.ics.ucl.edu/mlean/MLRepository.html.

References

  1. Afzalan, M., Jazizadeh, F.: An automated spectral clustering for multi-scale data. Neurocomputing 347, 94–108 (2019). https://doi.org/10.1016/j.neucom.2019.03.008

    Article  Google Scholar 

  2. Baccour. L., Alimi, A.M, John, R.I: Intuitionistic fuzzy similarity measures and their role in classification. JIIS, 25(2), 221–237 (2016). http://www.degruyter.com/view/j/jisys.2016.25.issue-2/jisys-2015-0086/jisys-2015-0086.xml

  3. Belesiotis, A., Skoutas, D., Efstathiades, C., Kaffes, V., Pfoser, D.: Spatio-textual user matching and clustering based on set similarity joins. Very Large Data Bases J. (VLDBJ) 27(3), 297–320 (2018). https://doi.org/10.1007/s00778-018-0498-5

    Article  Google Scholar 

  4. Bhatnagar, B.L., Singh, S., Arora, C., Jawahar, C.V.: Unsupervised learning of deep feature representation for clustering egocentric actions. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 1447–1453 (2017). https://doi.org/10.24963/ijcai.2017/200

  5. Bohne, J., Ying, Y., Gentric, S., Pontil, M.: Learning local metrics from pairwise similarity data. Pattern Recognit. 75, 315–326 (2018). https://doi.org/10.1016/j.patcog.2017.04.002

    Article  Google Scholar 

  6. Domeniconi, C., Gunopulos, D., Peng, J.: Large margin nearest neighbor classifiers. IEEE Trans. Neural Netw. Learn. Syst. 16(4), 899–909 (2005). https://doi.org/10.1109/TNN.2005.849821

    Article  Google Scholar 

  7. Goldberger, J., Roweis, S.T., Hinton, G.E., Salakhutdinov, R.: Neighbourhood components analysis. In: Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 513–520 (2004). http://papers.nips.cc/paper/2566-neighbourhood-components-analysis

  8. Haponchyk, I., Uva, A., Yu, S., Uryupina, O., Moschitti, A.: Supervised clustering of questions into intents for dialog system applications. In: Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2310–2321 (2018). https://doi.org/10.18653/v1/d18-1254

  9. He, Y., Chen, W., Chen, Y., Mao, Y.: Kernel density metric learning. In: IEEE International Conference on Data Mining (ICDM), pp. 271–280 (2013). https://doi.org/10.1109/ICDM.2013.153

  10. Hsieh, C.K., Yang, L., Cui, Y., Lin, T.Y., Belongie, S., Estrin, D.: Collaborative metric learning. In: International Conference on World Wide Web (WWW), pp. 193–201 (2017). https://doi.org/10.1145/3038912.3052639

  11. Kalintha, W., Ono, S., Numao, M., Fukui, K.: Kernelized evolutionary distance metric learning for semi-supervised clustering. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 4945–4946 (2017). http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14714

  12. Kang, Z., Peng, C., Cheng, Q.: Twin learning for similarity and clustering: a unified kernel approach. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 2080–2086 (2017). http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14569

  13. Kulis, B., et al.: Metric learning: A survey. Found. Trends® Mach. Learn. 5(4), 287–364 (2013). https://doi.org/10.1561/2200000019

  14. Kwok, J.T., Tsang, I.W.: Learning with idealized kernels. In: ACM International Conference on Machine Learning (ICML), pp. 400–407 (2003). http://www.aaai.org/Library/ICML/2003/icml03-054.php

  15. Li, D., Tian, Y.: Global and local metric learning via eigenvectors. Knowl. Based Syst. 116, 152–162 (2017). https://doi.org/10.1016/j.knosys.2016.11.004

    Article  Google Scholar 

  16. Li, X., Yin, H., Zhou, K., Zhou, X.: Semi-supervised clustering with deep metric learning and graph embedding. World Wide Web (WWW) 23(2), 781–798 (2020). https://doi.org/10.1007/s11280-019-00723-8

    Article  Google Scholar 

  17. Li, Y., Tian, X., Tao, D.: Regularized large margin distance metric learning. In: IEEE International Conference on Data Mining (ICDM), pp. 1015–1022 (2016). https://doi.org/10.1109/ICDM.2016.0129

  18. Liu, G., Zheng, K., Wang, Y., Orgun, M.A., Liu, A., Zhao, L., Zhou, X.: Multi-constrained graph pattern matching in large-scale contextual social graphs. In: IEEE International Conference on Data Engineering (ICDE), pp. 351–362 (2015). https://doi.org/10.1109/ICDE.2015.7113297

  19. Nguyen, B., Morell, C., Baets, B.D.: Supervised distance metric learning through maximization of the Jeffrey divergence. Pattern Recognit. 64, 215–225 (2017). https://doi.org/10.1016/j.patcog.2016.11.010

    Article  MATH  Google Scholar 

  20. Polat, K.: Similarity-based attribute weighting methods via clustering algorithms in the classification of imbalanced medical datasets. Neural Comput. Appl. 30(3), 987–1013 (2018). https://doi.org/10.1007/s00521-018-3471-8

    Article  Google Scholar 

  21. Schölkopf, B., Smola, A.J.: Learning with Kernels: support vector machines, regularization, optimization, and beyond. In: Adaptive Computation and Machine Learning Series. MIT Press (2002). http://www.worldcat.org/oclc/48970254

  22. Schultz, M., Joachims, T.: Learning a distance metric from relative comparisons. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 41–48 (2003). http://papers.nips.cc/paper/2366-learning-a-distance-metric-from-relative-comparisons

  23. Shalev-Shwartz, S., Singer, Y., Ng, A.Y.: Online and batch learning of pseudo-metrics. In: International Conference on Machine Learning (ICML) (2004). https://doi.org/10.1145/1015330.1015376

  24. Shalev-Shwartz, S., Singer, Y., Ng, A.Y.: Online and batch learning of pseudo-metrics. In: ACM International Conference on Machine Learning (ICML), p. 94 (2004). https://doi.org/10.1145/1015330.1015376

  25. Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press (2004). https://doi.org/10.1017/CBO9780511809682. https://kernelmethods.blogs.bristol.ac.uk/

  26. Simard, P.Y., LeCun, Y., Denker, J.S.: Efficient pattern recognition using a new transformation distance. In: Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 50–58 (1992)

    Google Scholar 

  27. Smola, A.J.: Learning with Kernels. Citeseer (1998). http://d-nb.info/955631580

  28. Song, K., Nie, F., Han, J., Li, X.: Parameter free large margin nearest neighbor for distance metric learning. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 2555–2561 (2017). http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14616

  29. St Amand, J., Huan, J.: Sparse compositional local metric learning. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). pp. 1097–1104 (2017). https://doi.org/10.1145/3097983.3098153

  30. Wang, Q., Yin, H., Hu, Z., Lian, D., Wang, H., Huang, Z.: Neural memory streaming recommender networks with adversarial training. In: ACM SIGKDD International Conference on Knowledge Discovery & Data Mining(KDD), pp. 2467–2475 (2018). https://doi.org/10.1145/3219819.3220004

  31. Weinberger, K.Q., Blitzer, J., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. In: Advances in neural information processing systems (NeurIPS), pp. 1473–1480 (2005). http://papers.nips.cc/paper/2795-distance-metric-learning-for-large-margin-nearest

  32. Weinberger, K.Q., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. J. Mach. Learn. Res. 10, 207–244 (2009). https://dl.acm.org/citation.cfm?id=1577078

  33. Wu, Y., Wang, S., Huang, Q.: Online asymmetric similarity learning for cross-modal retrieval. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 3984–3993 (2017). https://doi.org/10.1109/CVPR.2017.424

  34. Xing, E.P., Jordan, M.I., Russell, S.J., Ng, A.Y.: Distance metric learning with application to clustering with side-information. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 505–512 (2002). http://papers.nips.cc/paper/2164-distance-metric-learning-with-application-to-clustering

  35. Ye, H.J., Zhan, D.C., Si, X.M., Jiang, Y.: Learning mahalanobis distance metric: Considering instance disturbance helps. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 3315–3321 (2017). https://doi.org/10.24963/ijcai.2017/463

  36. Yu, S., Chu, S.W., Wang, C., Chan, Y., Chang, T.: Two improved k-means algorithms. Appl. Soft Comput. 68, 747–755 (2018). https://doi.org/10.1016/j.asoc.2017.08.032

    Article  Google Scholar 

  37. Zhai, D., Liu, X., Chang, H., Zhen, Y., Chen, X., Guo, M., Gao, W.: Parametric local multiview hamming distance metric learning. Pattern Recognit. 75, 250–262 (2018). https://doi.org/10.1016/j.patcog.2017.06.018

  38. Zhang, J., Zhang, L.: Efficient stochastic optimization for low-rank distance metric learning. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 933–940 (2017). http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14373

  39. Zheng, K., Zheng, Y., Yuan, N.J., Shang, S., Zhou, X.: Online discovery of gathering patterns over trajectories. IEEE Trans. Knowl. Data Eng. 26(8), 1974–1988 (2014). https://doi.org/10.1109/TKDE.2013.160

    Article  Google Scholar 

  40. Zhu, H., Long, M., Wang, J., Cao, Y.: Deep hashing network for efficient similarity retrieval. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 2415–2421 (2016). http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12039

  41. Zuo, W., Wang, F., Zhang, D., Lin, L., Huang, Y., Meng, D., Zhang, L.: Distance metric learning via iterated support vector machines. IEEE Trans. Image Process. 26(10), 4937–4950 (2017). https://doi.org/10.1109/TIP.2017.2725578

Download references

Author information

Authors and Affiliations

  1. Shanghai Key Laboratory of Data Science, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Chao Wang, Yongyi Hu, Xiaofeng Gao & Guihai Chen

Authors
  1. Chao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongyi Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaofeng Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guihai Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaofeng Gao .

Editor information

Editors and Affiliations

  1. Clausthal University of Technology, Clausthal-Zellerfeld, Germany

    Sven Hartmann

  2. Johannes Kepler University of Linz, Linz, Austria

    Josef Küng

  3. Johannes Kepler University of Linz, Linz, Austria

    Gabriele Kotsis

  4. IFS, Vienna University of Technology, Vienna, Wien, Austria

    A Min Tjoa

  5. Johannes Kepler University of Linz, Linz, Austria

    Ismail Khalil

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, C., Hu, Y., Gao, X., Chen, G. (2020). KPML: A Novel Probabilistic Perspective Kernel Mahalanobis Distance Metric Learning Model for Semi-supervised Clustering. In: Hartmann, S., Küng, J., Kotsis, G., Tjoa, A.M., Khalil, I. (eds) Database and Expert Systems Applications. DEXA 2020. Lecture Notes in Computer Science(), vol 12392. Springer, Cham. https://doi.org/10.1007/978-3-030-59051-2_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-59051-2_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59050-5

  • Online ISBN: 978-3-030-59051-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation