Bayesian Ying-Yang Harmony Learning for Local Factor Analysis: A Comparative Investigation

Summary

For unsupervised learning by Local Factor Analysis (LFA), it is important to determine both the component number and the local hidden dimensions appropriately, which is a typical example of model selection. One conventional approach for model selection is to implement a two-phase procedure with the help of model selection criteria, such as AIC, CAIC, BIC(MDL), SRM, CV, etc.. Although all working well given large enough samples, they still suffer from two problems. First, their performances will deteriorate greatly on a small sample size. Second, two-phase procedure requires intensive computation. To tackle the second problem, one type of efforts has been made in the literature, featured by an incremental implementation, e.g. IMoFA and VBMFA. Bayesian Ying-Yang (BYY) harmony learning provides not only a BYY harmony data smoothing criterion (BYY-C) in a two-phase implementation for the first problem, but also an algorithm called automatic BYY harmony learning (BYY-A) that have automatic model selection ability during parameter learning and thus can reduce the computational expense significantly. The lack of systematic comparisons in the literature motivates this work. Comparative experiments are first conducted on synthetic data considering not only different settings including noise, dimension and sample size, but also different evaluations including model selection accuracy and three other applied performances. Thereafter, comparisons are also made on several real world classification datasets. In two-phase implementation, the observations show that BIC and CAIC generally outperform AIC, SRM and CV, while BYY-C is the best for small sample sizes. Moreover, in the cases of a sufficiently large sample size, IMoFA, VBMFA, and BYY-A produce similar performances but with much reduced computational costs, where, still, BYY-A provides better or at least comparably good performances.