Abstract
A common problem of data-driven data mining methods is that they might lack considering domain knowledge, despite possibly having high accuracy with respect to the data. As such, prior knowledge plays an important role in many data mining applications. Incorporating prior knowledge into data mining techniques is not trivial and remains a partially open issue drawing much attention. In this paper, we propose a new support vector regression (SVR) model that takes into account the prior knowledge of domain experts in the form of inequalities, which reflect the monotonic relationship between the output and some of the attributes of the input. A dual quadratic programming problem corresponding to the SVR model is derived, along with algorithms for solving it and creating constraints, respectively. The experiment results, which were conducted on two artificial and two practical datasets, show that the proposed model, which considers the monotonicity defined by domain experts, performs better than the original SVR. Moreover, the proposed method is also suitable for prior domain knowledge of piecewise monotonicity.
Similar content being viewed by others
References
Yu T, Simoff S, Jan T (2010) VQSVM: a case study for incorporating prior domain knowledge into inductive machine learning. Neurocomputing 73(13):2614–2623
Sinha AP, Zhao H (2008) Incorporating domain knowledge into data mining classifiers: an application in indirect lending. Decis Support Syst 46(1):287–299
Cao L (2010) Domain-driven data mining: challenges and prospects. IEEE Trans Knowl Data Eng 22(6):755–769
Eryarsoy E, Koehler GJ, Aytug H (2009) Using domain-specific knowledge in generalization error bounds for support vector machine learning. Decis Support Syst 46(2):481–491
Kotlowski W, Slowinski R (2013) On nonparametric ordinal classification with monotonicity constraints. IEEE Trans Knowl Data Eng 25(11):2576–2589
Hu Q, Pan W, Song Y, Yu D (2012) Large-margin feature selection for monotonic classification. Knowl Based Syst 31:8–18
Schölkopf B, Smola AJ (2002) Learning with kernels: support vector machines, regularization, optimization and beyond. The MIT Press, Cambridge
Li ST, Chen CC (2015) A regularized monotonic fuzzy support vector machine model for data mining with prior knowledge. IEEE Trans Fuzzy Syst 23(5):1713–1727
Chen CC, Li ST (2014) Credit rating with a monotonicity-constrained support vector machine model. Expert Syst Appl 41(16):7235–7247. https://doi.org/10.1016/j.eswa.2014.05.035
Xi X, Shi H, Han L, Wang T, Ding HY, Zhang G, Tang Y, Yin Y (2017) Breast tumor segmentation with prior knowledge learning. Neurocomputing 237:145–157
Duivesteijn W, Feelders A (2008) Nearest neighbour classification with monotonicity constraints. In: Daelemans W (ed) The 2008 European conference on machine learning and knowledge discovery in databases, Antwerp, Belgium, 2008. Springer, pp 301–316
Towell GG, Shavlik JW (1994) Knowledge-based artificial neural networks. Artif Intell 70(1–2):119–165
Doumpos M, Zopounidis C (2009) Monotonic support vector machines for credit risk rating. New Math Nat Comput 5(3):557–570
Grossi V, Romei A, Turini F (2017) Survey on using constraints in data mining. Data Min Knowl Disc 31(2):424–464. https://doi.org/10.1007/s10618-016-0480-z
Petković D, Shamshirband S, Saboohi H, Ang TF, Anuar NB, Pavlović ND (2014) Support vector regression methodology for prediction of input displacement of adaptive compliant robotic gripper. Appl Intell 41(3):887–896
Vapnik VN (1995) The Nature of Statistical Learning Theory. Springer-Verlag, New York
Vapnik VN (1998) Statistical learning theory. Wiley, New York
Vapnik V, Golowich SE, Smola A (1997) Support vector method for function approximation, regression estimation, and signal processing. In: Mozer MC, Jordan MI, Petsche T (eds) Proceedings of the 9th international conference on neural information processing systems, Cambridge, MA, USA, 1997. MIT Press, pp 281–287
Basak D, Pal S, Patranabis DC (2007) Support vector regression. Neural Inf Process Lett Rev 11(10):203–224
Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297. https://doi.org/10.1023/a:1022627411411
Cristianini N, Shawe-Taylor J (2000) An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press, Cambridge
Ma J, Theiler J, Perkins S (2003) Accurate on-line support vector regression. Neural Comput 15(11):2683–2703
Yeh CY, Huang CW, Lee SJ (2011) A multiple-kernel support vector regression approach for stock market price forecasting. Expert Syst Appl 38(3):2177–2186
Balasubramanian K, Yu K, Lebanon G (2016) Smooth sparse coding via marginal regression for learning sparse representations. Artif Intell 238:83–95
Lu CJ, Lee TS, Chiu CC (2009) Financial time series forecasting using independent component analysis and support vector regression. Decis Support Syst 47(2):115–125
Lauer F, Bloch G (2008) Incorporating prior knowledge in support vector machines for classification: a review. Neurocomputing 71:1578–1594
Evgeniou T, Boussios C, Zacharia G (2005) Generalized robust conjoint estimation. Mark Sci 24(3):415–429. https://doi.org/10.1287/mksc.1040.0100
Zhou W, Zhang L, Jiao L (2002) Linear programming support vector machines. Pattern Recogn 35(12):2927–2936
Tikhonov AN, Arsenin VI, John F (1977) Solutions of Ill-posed problems (Trans: Russian Tf), vol 14. V. H. Winston and Sons, Washington, D. C
Gutiérrez PA, García S (2016) Current prospects on ordinal and monotonic classification. Prog Artif Intell 5(3):171–179. https://doi.org/10.1007/s13748-016-0088-y
Rademaker M, De Baets B, De Meyer H (2012) Optimal monotone relabelling of partially non-monotone ordinal data. Optim Methods Softw 27(1):17–31. https://doi.org/10.1080/10556788.2010.507272
Potharst R, Feelders AJ (2002) Classification trees for problems with monotonicity constraints. ACM SIGKDD Explor Newsl 4(1):1–10
Qinghua H, Xunjian C, Lei Z, Zhang D, Maozu G, Yu D (2012) Rank entropy-based decision trees for monotonic classification. IEEE Trans Knowl Data Eng 24(11):2052–2064. https://doi.org/10.1109/tkde.2011.149
Qian Y, Xu H, Liang J, Liu B, Wang J (2015) Fusing monotonic decision trees. IEEE Trans Knowl Data Eng 27(10):2717–2728. https://doi.org/10.1109/TKDE.2015.2429133
Daniels HAM, Velikova MV (2006) Derivation of monotone decision models from noisy data. IEEE Trans Syst Man Cybern Part C (Appl Rev) 36(5):705–710. https://doi.org/10.1109/TSMCC.2005.855493
Pei S, Hu Q, Chen C (2016) Multivariate decision trees with monotonicity constraints. Knowl Based Syst 112:14–25. https://doi.org/10.1016/j.knosys.2016.08.023
González S, Herrera F, García S (2015) Monotonic random forest with an ensemble pruning mechanism based on the degree of monotonicity. New Gener Comput 33(4):367–388
Pelckmans K, Espinoza M, De Brabanter J, Suykens JAK, De Moor B (2005) Primal-dual monotone kernel regression. Neural Process Lett 22(2):171–182
Shah S, Sardeshmukh A, Ahmed S, Reddy S (2016) Soft monotonic constraint support vector regression. In: Paper presented at the international conference on management of data (COMAD 2016), Pune, India
Abu-Mostafa YS (1995) Hints. Neural Comput 7:639–671
Abu-Mostafa YS (1994) Learning from hints. J Complex 10:165–178
Wismer D, Chattergy R (1978) Introduction to Nonlinear Optimization. North Holland, Amsterdam
Bertsekas D, Castañon D, Eckstein J, Zenios S (1995) Chapter 5 Parallel computing in network optimization. Handb Oper Res Manag Sci 7:331–399. https://doi.org/10.1016/S0927-0507(05)80122-7
Courant R, Hilbert D (1970) Methods of mathematical physics, vol I. Wiley, New York
Ye Y, Tse E (1989) An extension of Karmarkar’s projective algorithm for convex quadratic programming. Math Program 44(1–3):157–179
Graf HP, Cosatto E, Bottou L, Dourdanovic I, Vapnik V (2005) Parallel support vector machines: the cascade svm. Adv Neural Inf Process Syst 17:521–528
Rost B (2009) Basel committee on banking supervision. In: Handbook of transnational economic governance regimes. Brill, pp 319–328
Blake CL, Merz CJ (1998) UCI repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html. Accessed 28 Nov 2018
Clarke SM, Griebsch JH, Simpson TW (2005) Analysis of support vector regression for approximation of complex engineering analyses. J Mech Des 127(6):1077–1087
Nandi S, Toliyat HA, Li X (2005) Condition monitoring and fault diagnosis of electrical motors—a review. IEEE Trans Energy Convers 20(4):719–729
Gunn SR (1998) Support vector machines for classification and regression. ISIS Tech Rep 14(1):5–16
Chou JS, Pham AD (2015) Smart artificial firefly colony algorithm-based support vector regression for enhanced forecasting in civil engineering. Comput Aided Civ Infrastruct Eng 30(9):715–732
Hestenes MR, Stiefel E (1952) Methods of conjugate gradients for solving linear systems. J Res Natl Bureau Stand 49(6):409–436
Debnath R, Muramatsu M, Takahashi H (2005) An efficient support vector machine learning method with second-order cone programming for large-scale problems. Appl Intell 23(3):219–239
Acknowledgements
This study was supported in part by the Ministry of Science and Technology, Taiwan, under contract NSC 102-2410-H-006-080-MY3, MOST 105-2410-H-006-038-MY3 and MOST 107-2410-H-143-005. The authors also thank Mr. Chi Chou and Mr. Yu-Di, Chen for their help with the experimentation.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Chuang, HC., Chen, CC. & Li, ST. Incorporating monotonic domain knowledge in support vector learning for data mining regression problems. Neural Comput & Applic 32, 11791–11805 (2020). https://doi.org/10.1007/s00521-019-04661-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-019-04661-4