Skip to main content
SpringerLink
Log in

Relative Newton and Smoothing Multiplier Optimization Methods for Blind Source Separation

  • Chapter
Blind Speech Separation

Part of the book series: Signals and Communication Technology ((SCT))

  • 2437 Accesses

We study a relative optimization framework for quasi-maximum likelihood blind source separation and relative Newton method as its particular instance. The structure of the Hessian allows its fast approximate inversion. In the second part we present Smoothing Method of Multipliers (SMOM) for minimization of sum of pairwise maxima of smooth functions, in particular sum of absolute value terms. Incorporating Lagrange multiplier into a smooth approximation of max-type function, we obtain an extended notion of nonquadratic augmented Lagrangian. Our approach does not require artificial variables, and preserves the sparse structure of Hessian. Convergence of the method is further accelerated by the Frozen Hessian strategy. We demonstrate efficiency of this approach on an example of blind separation of sparse sources. The nonlinearity in this case is based on the absolute value function, which provides superefficient source separation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. D. Pham and P. Garat, “Blind separation of a mixture of independent sources through a quasi-maximum likelihood approach,” IEEE Transactions on Signal Processing, vol. 45, no. 7, pp. 1712-1725, 1997.

    Article  MATH  Google Scholar 

  2. A. J. Bell and T. J. Sejnowski, “An information-maximization approach to blind separation and blind deconvolution,” Neural Computation, vol. 7, no. 6, pp. 1129-1159, 1995.

    Article  Google Scholar 

  3. J.-F. Cardoso, “On the performance of orthogonal source separation algo-rithms,” in EUSIPCO, Edinburgh, Sept. 1994, pp. 776-779.

    Google Scholar 

  4. .——, “Blind signal separation: statistical principles,” Proceedings of the IEEE, vol.9, no.10, pp.2009-2025, Oct.1998.[Online]. Available:ftp://sig.enst.fr/pub/jfc/Papers/ProcIEEE.us.ps.gz

  5. S. S. Chen, D. L. Donoho, and M. A. Saunders, “Atomic decomposition by basis pursuit,” SIAM J. Sci. Comput., vol. 20, no. 1, pp. 33-61, 1998.

    Article  MathSciNet  Google Scholar 

  6. B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete basis set: A strategy employed by v1?” Vision Research, vol. 37, pp. 3311-3325, 1997.

    Article  Google Scholar 

  7. M. S. Lewicki and B. A. Olshausen, “A probabilistic framework for the adapta-tion and comparison of image codes,” Journal of the Optical Society of America, vol. 16, no. 7, pp. 1587-1601, 1999.

    Article  Google Scholar 

  8. M. Zibulevsky and B. A. Pearlmutter, “Blind source separation by sparse decomposition in a signal dictionary,” Neural Computations, vol. 13, no. 4, pp. 863-882, 2001.

    Article  MATH  Google Scholar 

  9. M. Zibulevsky, B. A. Pearlmutter, P. Bofill, and P. Kisilev, “Blind source separation by sparse decomposition,” in Independent Components Analysis: Princeiples and Practice, S. J. Roberts and R. M. Everson, Eds. Cambridge University Press, 2001.

    Google Scholar 

  10. M. Zibulevsky, P. Kisilev, Y. Y. Zeevi, and B. A. Pearlmutter, “Blind source separation via multinode sparse representation,” in Advances in Neural Infor-mation Processing Systems 12. MIT Press, 2002.

    Google Scholar 

  11. A. Hyvärinen, “Fast and robust fixed-point algorithms for independent com-ponent analysis,” IEEE Transactions on Neural Networks, vol. 10, no. 3, pp. 626-634, 1999.

    Article  Google Scholar 

  12. T. Akuzawa and N. Murata, “Multiplicative nonholonomic Newton-like algo-rithm,” Chaos, Solitons and Fractals, vol. 12, p. 785, 2001.

    Article  MATH  MathSciNet  Google Scholar 

  13. T. Akuzawa, “Extended quasi-Newton method for the ICA,” Laboratory for Mathematical Neuroscience, RIKEN Brain Science Institute, Tech. Rep., 2000, http://www.mns.brain.riken.go.jp/˜akuzawa/publ.html.

  14. D. Pham, “Joint approximate diagonalization of positive definite matrices,” SIAM J. on Matrix Anal. and Appl., vol. 22, no. 4, pp. 1136-1152, 2001.

    Article  MATH  MathSciNet  Google Scholar 

  15. D. Pham and J.-F. Cardoso, “Blind separation of instantaneous mixtures of non-stationary sources,” IEEE Transactions on Signal Processing, vol. 49, no. 9, pp. 1837-1848, 2001.

    Article  MathSciNet  Google Scholar 

  16. M. Joho and K. Rahbar, “Joint diagonalization of correlation matrices by using Newton methods with application to blind signal separation,” SAM 2002, 2002, http://www.phonak.uiuc.edu/˜joho/research/publications/sam 2002 2.pdf.

  17. A. Ziehe, P. Laskov, G. Nolte, and K.-R. Mueller, “A fast algorithm for joint diagonalization with non-orthogonal transformations and its application to blind source separation,” Journal of Machine Learning Research, vol. 5, pp. 801-818, July 2004.

    Google Scholar 

  18. B. Kort and D. Bertsekas, “Multiplier methods for convex programming,” Proc 1073 IEEE Conf. Decision Control, San-Diego, Calif., pp. 428-432, 1973.

    Google Scholar 

  19. D. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods. New York: Academic Press, 1982.

    MATH  Google Scholar 

  20. R. Polyak, “Modified barrier functions: Theory and methods,” Math. Program-ming, vol. 54, pp. 177-222, 1992.

    Article  MATH  MathSciNet  Google Scholar 

  21. A. Ben-Tal, I. Yuzefovich, and M. Zibulevsky, “Penalty/barrier multiplier meth-ods for min-max and constrained smooth convex programs,” Opt. Lab., Dept. of Indust. Eng., Technion, Haifa, Israel, Tech. Rep. 9, 1992.

    Google Scholar 

  22. P. Tseng and D. Bertsekas, “Convergence of the exponential multiplier method for convex programming,” Math. Programming, vol. 60, pp. 1-19, 1993.

    Article  MathSciNet  Google Scholar 

  23. M. G. Breitfeld and D. Shanno, “Computational experience with penalty/bar-rier methods for nonlinear programming,” Annals of Operations Research, vol. 62, pp. 439-464, 1996.

    Article  MATH  MathSciNet  Google Scholar 

  24. M. Zibulevsky, “Penalty/barrier multiplier methods for large-scale nonlinear and semidefinite programming,” Ph.D. dissertation, Technion - Israel Institute of Technology, 1996, http://ie.technion.ac.il/˜mcib/.

  25. A. Ben-Tal and M. Zibulevsky, “Penalty/barrier multiplier methods for con-vex programming problems,” SIAM Journal on Optimization, vol. 7, no. 2, pp. 347-366, 1997.

    Article  MATH  MathSciNet  Google Scholar 

  26. L. Mosheyev and M. Zibulevsky, “Penalty/barrier multiplier algorithm for semi-definite programming,” Optimization Methods and Software, vol. 13, no. 4, pp. 235-261, 2000.

    Article  MATH  MathSciNet  Google Scholar 

  27. M. Kocvara and M. Stingl, “PENNON - a code for convex nonlinear and semidefinite programming,” Optimization Methods and Softwarte, vol. 18(3), pp. 317-333, 2003.

    Article  MATH  MathSciNet  Google Scholar 

  28. A. Ben-Tal and M. Teboulle, “A smoothing technique for nondifferentiable optimization problems,” Fifth French German Conference, Lecture Notes in Math. 1405, Springer-Verlag, New York, pp. 1-11, 1989.

    Google Scholar 

  29. C. Chen and O. L. Mangasarian, “A class of smoothing functions for nonlin-ear and mixed complementarity problems,” Computational Optimization and Applications, vol. 5, pp. 97-138, 1996.

    Article  MATH  MathSciNet  Google Scholar 

  30. A. Cichocki, R. Unbehauen, and E. Rummert, “Robust learning algorithm for blind separation of signals,” Electronics Letters, vol. 30, no. 17, pp. 1386-1387, 1994.

    Article  Google Scholar 

  31. S. Amari, A. Cichocki, and H. H. Yang, “A new learning algorithm for blind signal separation,” in Advances in Neural Information Processing Systems 8. MIT Press, 1996. [Online]. Available: http://www.cs.cmu.edu/Groups/NIPS/ NIPS95/Papers.html

  32. J.-F. Cardoso and B. Laheld, “Equivariant adaptive source separation,” IEEE Transactions on Signal Processing, vol. 44, no. 12, pp. 3017-3030, 1996.

    Article  Google Scholar 

  33. P. E. Gill, W. Murray, and M. H. Wright, Practical Optimization. New York: Academic Press, 1981.

    MATH  Google Scholar 

  34. R. Rockafellar, Convex Analysis. Princeton, NJ: Princeton University Press, 1970.

    MATH  Google Scholar 

  35. M. Zibulevsky,“Smoothing method of multipliers for sum-max prob-lems,” Dept. of Elec. Eng., Technion, Tech. Rep., 2003, http://ie.technion.ac.il/˜mcib/.

  36. A. Cichocki, S. Amari, and K. Siwek,“ICALAB toolbox for im- age processing- benchmarks,”2002, http://www.bsp.brain.riken.go.jp/ICALAB/ICALABImageProc/benchmarks/.

  37. J.-F. Cardoso, “High-order contrasts for independent component analysis,” Neural Computation, vol. 11, no. 1, pp. 157-192, 1999.

    Article  MathSciNet  Google Scholar 

  38. S. Makeig, “ICA toolbox for psychophysiological research,” Computational Neurobiology Laboratory, the Salk Institute for Biological Studies, 1998, http://www.cnl.salk.edu/˜ ica.html.

  39. A. Hyvärinen,“The Fast-ICA MATLAB package,” 1998, http://www.cis.hut.fi/˜aapo/.

  40. J.-F. Cardoso, “JADE for real-valued data,” 1999, http://sig.enst.fr:80/∼car-doso/guidesepsou.html.

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Technion, 32000, Haifa, Israel

    Michael Zibulevsky

Authors
  1. Michael Zibulevsky
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. NTT Corporation, 2-4 Hikaridai, 619-0237, Soraku-gun, Kyoto, Japan

    Shoji Makino  & Hiroshi Sawada  & 

  2. University of California, San Diego, 9500 Gilman Drive, 0523, 92093-0523, La Jolla, CA, USA

    Te-Won Lee

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer

About this chapter

Cite this chapter

Zibulevsky, M. (2007). Relative Newton and Smoothing Multiplier Optimization Methods for Blind Source Separation. In: Makino, S., Sawada, H., Lee, TW. (eds) Blind Speech Separation. Signals and Communication Technology. Springer, Dordrecht. https://doi.org/10.1007/978-1-4020-6479-1_7

Download citation

  • DOI: https://doi.org/10.1007/978-1-4020-6479-1_7

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-1-4020-6478-4

  • Online ISBN: 978-1-4020-6479-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation