Skip to main content
SpringerLink
Log in

Incremental Multi-Label Learning with Active Queries

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

In multi-label learning, it is rather expensive to label instances since they are simultaneously associated with multiple labels. Therefore, active learning, which reduces the labeling cost by actively querying the labels of the most valuable data, becomes particularly important for multi-label learning. A good multi-label active learning algorithm usually consists of two crucial elements: a reasonable criterion to evaluate the gain of querying the label for an instance, and an effective classification model, based on whose prediction the criterion can be accurately computed. In this paper, we first introduce an effective multi-label classification model by combining label ranking with threshold learning, which is incrementally trained to avoid retraining from scratch after every query. Based on this model, we then propose to exploit both uncertainty and diversity in the instance space as well as the label space, and actively query the instance-label pairs which can improve the classification model most. Extensive experiments on 20 datasets demonstrate the superiority of the proposed approach to state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Settles B. Active learning literature survey. Technical Report 1648, Computer Sciences Department, University of Wisconsin Madison, 2009. http://www.burrsettles.com/pub/settles.activelearning.pdf, Nov. 2019.

  2. Balcan M F, Broder A, Zhang T. Margin based active learning. In Proc. the 20th Annual Conference on Learning Theory Learning Theory, June 2007, pp.35-50.

  3. Brinker K. Incorporating diversity in active learning with support vector machines. In Proc. the 20th International Conference on Machine Learning, August 2003, pp.59-66.

  4. Zhu J B, Wang H Z, Tsou B K, Ma M. Active learning with sampling by uncertainty and density for data annotations. IEEE Transactions on Audio, Speech, and Language Processing, 2010, 18(6): 1323-1331.

    Google Scholar 

  5. Huang S J, Jin R, Zhou Z H. Active learning by querying informative and representative examples. IEEE Transactions Pattern Analysis and Machine Intelligence, 2014, 36(10): 1936-1994.

    Google Scholar 

  6. Wang Z, Ye J P. Querying discriminative and representative samples for batch mode active learning. ACM Transactions on Knowledge Discovery from Data, 2015, 9(3): Article No. 27.

    Google Scholar 

  7. Shao H. Query by diverse committee in transfer active learning. Frontiers of Computer Science, 2019, 13(2): 280-291.

    Google Scholar 

  8. Ma Y L, Cui C R, Nie X S, Yang G P, Shaheed K, Yin Y L. Pre-course student performance prediction with multi-instance multi-label learning. Science China Information Sciences, 2019, 62(2): Article No. 29101.

  9. Zhang M L, Zhou Z H. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(8): 1819-1837.

    Google Scholar 

  10. Qi G J, Hua X S, Rui Y, Tang J H, Zhang H J. Two-dimensional active learning for image classification. In Proc. the 2008 IEEE Conference on Computer Vision and Pattern Recognition, June 2008.

  11. Zhou Z H. Abductive learning: Towards bridging machine learning and logical reasoning. Science China Information Sciences, 2019, 62(7): Article No. 76101.

  12. Li X, Guo Y H. Active learning with multi-label SVM classification. In Proc. the 23rd International Joint Conference on Artificial Intelligence, August 2013, pp.1479-1485.

  13. Singh M, Brew A, Greene D, Cunningham P. Score normalization and aggregation for active learning in multi-label classification. Technical Report, University College Dublin, 2010. http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=8480CC5725147C066F85B25B0C-0C27BE?doi=10.1.1.331.9765&rep=rep1&type=pdf, Nov. 2019.

  14. Yang B S, Sun J T, Wang T J, Chen Z. Effective multi-label active learning for text classification. In Proc. the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, June 2009, pp.917-926.

  15. Weston J, Bengio S, Usunier N. WSABIE: Scaling up to large vocabulary image annotation. In Proc. the 22nd International Joint Conference on Artificial Intelligence, July 2011, pp.2764-2770.

  16. Fürnkranz J, Hüllermeier E, Mencía E L, Brinker K. Multilabel classification via calibrated label ranking. Machine Learning, 2008, 73(2): 133-153.

    Google Scholar 

  17. Hung C W, Lin H T. Multi-label active learning with auxiliary learner. In Proc. the 3rd Asian Conference on Machine Learning, 2011, pp.315-332.

  18. Bi W, Kwok J T Y. Efficient multi-label classification with many labels. In Proc. the 30th International Conference on Machine Learning, June 2013, pp.405-413.

  19. Vasisht D, Damianou A C, Varma M, Kapoor A. Active learning for sparse Bayesian multilabel classification. In Proc. the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 2014, pp.472-481.

  20. Carbonneau M A, Granger E, Gagnon G. Bag-level aggregation for multiple-instance active learning in instance classification problems. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(5): 1441-1451.

    Google Scholar 

  21. Chen X, Yu G X, Domeniconi C, Wang J, Li Z, Zhang Z L. Cost effective multi-label active learning via querying subexamples. In Proc. the 2018 IEEE International Conference on Data Mining, November 2018, pp.905-910.

  22. Li X C, Wang L, Sung E. Multi-label SVM active learning for image classification. In Proc. the 2004 International Conference on Image Processing, October 2004, pp.2207-2210.

  23. Brinker K. On active learning in multi-label classification. In Proc. the 29th Annual Conference of the Gesellschaft für Klassifikation e.V. University of Magdeburg, March 2005, pp.206-213.

  24. Singh M, Curran E, Cunningham P. Active learning for multi-label image annotation. In Proc. the 19th Irish Conference on Artificial Intelligence and Cognitive Science, August 2008, pp.173-182.

  25. Esuli A, Sebastiani F. Active learning strategies for multilabel text classification. In Proc. the 31th European Conference on Information Retrieval Research, April 2009, pp.102-113.

  26. Huang S J, Chen S C, Zhou Z H. Multi-label active learning: Query type matters. In Proc. the 24th International Joint Conference on Artificial Intelligence, July 2015, pp.946-952.

  27. Wu J, Guo A Q, Sheng V S, Zhao P P, Cui Z M, Li H. Adaptive low-rank multi-label active learning for image classification. In Proc. the 2017 ACM on Multimedia Conference, October 2017, pp.1336-1344.

  28. Li Y C, Song Y L, Luo J B. Improving pairwise ranking for multi-label image classification. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.1837-1845.

  29. Zhang X Y, Cheng J, Xu C S, Lu H Q, Ma S D. Multiview multi-label active learning for image classification. In Proc. the IEEE International Conference on Multimedia and Expo, June 2009, pp.258-261.

  30. Wang P, Zhang P, Guo L. Mining multi-label data streams using ensemble-based active learning. In Proc. the 2012 SIAM International Conference on Data Mining, April 2012, pp.1131-1140.

  31. Huang S J, Zhou Z H. Active query driven by uncertainty and diversity for incremental multi-label learning. In Proc. the 13th IEEE International Conference on Data Mining, December 2013, pp.1079-1084.

  32. Huang S J, Gao W, Zhou Z H. Fast multi-instance multi-label learning. In Proc. the 28th AAAI Conference on Artificial Intelligence, July 2014, pp.1868-1874.

  33. Huang S J, Gao N N, Chen S C. Multi-instance multi-label active learning. In Proc. the 26th International Joint Conference on Artificial Intelligence, August 2017, pp.1886-1892.

  34. Boutell M R, Luo J B, Shen X P, Brown C M. Learning multi-label scene classification. Pattern Recognition, 2004, 37(9): 1757-1771.

    Google Scholar 

  35. Ben-David S, Loker D, Srebro N, Sridharan K. Minimizing the misclassification error rate using a surrogate convex loss. In Proc. the 29th International Conference on Machine Learning, June 2012, Article No. 15.

  36. Duygulu P, Barnard K, de Freitas J F G, Forsyth D A. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In Proc. the 7th European Conference on Computer Vision, Part IV, May 2002, pp.97-112.

  37. Trohidis K, Tsoumakas G, Kalliris G, Vlahavas I. Multilabel classification of music into emotions. In Proc. the 9th International Conference of Music Information Retrieval, September 2008, pp.325-330.

  38. Klimt B, Yang Y M. Introducing the Enron corpus. In Proc. the 1st Conference on Email and Anti-Spam, July 2004, Article No. 4.

  39. Diplaris S, Tsoumakas G, Mitkas P A, Vlahavas I. Protein classification with multiple algorithms. In Proc. the 10th Panhellenic Conference on Informatics, November 2005, pp.448-456.

  40. Zhang M L, Zhou Z H. ML-KNN: A lazy learning approach to multi-label learning. Pattern Recognition, 2007, 40(7): 2038-2048.

    MATH  Google Scholar 

  41. Sebastiani F. Machine learning in automated text categorization. ACM Computing Surveys, 2002, 34(1): 1-47.

    Google Scholar 

  42. Zhou Z H, Zhang M L, Huang S J, Li Y F. Multi-instance multi-label learning. Artificial Intelligence, 2012, 176(1): 2291-2320.

    MathSciNet  MATH  Google Scholar 

  43. Elisseeff A, Weston J. A kernel method for multi-labelled classification. In Proc. the 2011 Annual Conference on Neural Information Processing Systems, December 2001, pp.681-687.

  44. Ueda N, Saito K. Parametric mixture models for multi-labeled text. In Proc. the 2002 Annual Conference on Neural Information Processing Systems, December 2002, pp.721-728.

  45. Fan R E, Chang K W, Hsieh C J, Wang X R, Lin C J. LIBLINEAR: A library for large linear classification. The Journal of Machine Learning Research, 2008, 9: 1871-1874.

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Sheng-Jun Huang, Guo-Xiang Li, Wen-Yu Huang & Shao-Yuan Li

  2. Key Laboratory of Pattern Analysis and Machine Intelligence, Ministry of Industry and Information Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Sheng-Jun Huang, Guo-Xiang Li, Wen-Yu Huang & Shao-Yuan Li

  3. Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing University, Nanjing, 210023, China

    Sheng-Jun Huang & Shao-Yuan Li

Authors
  1. Sheng-Jun Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guo-Xiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wen-Yu Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shao-Yuan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shao-Yuan Li.

Electronic supplementary material

ESM 1

(PDF 582 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, SJ., Li, GX., Huang, WY. et al. Incremental Multi-Label Learning with Active Queries. J. Comput. Sci. Technol. 35, 234–246 (2020). https://doi.org/10.1007/s11390-020-9994-3

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-020-9994-3

Keywords

Search

Navigation