Skip to main content
SpringerLink
Log in

On the equivalence between nonnegative tensor factorization and tensorial probabilistic latent semantic analysis

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Non-negative Matrix Factorization (NMF) and Probabilistic Latent Semantic Analysis (PLSA) are two widely used methods for non-negative data decomposition of two-way data (e.g., document-term matrices). Studies have shown that PLSA and NMF (with the Kullback-Leibler divergence objective) are different algorithms optimizing the same objective function. Recently, analyzing multi-way data (i.e., tensors), has attracted a lot of attention as multi-way data have rich intrinsic structures and naturally appear in many real-world applications. In this paper, the relationships between NMF and PLSA extensions on multi-way data, e.g., NTF (Non-negative Tensor Factorization) and T-PLSA (Tensorial Probabilistic Latent Semantic Analysis), are studied. Two types of T-PLSA models are shown to be equivalent to two well-known non-negative factorization models: PARAFAC and Tucker3 (with the KL-divergence objective). NTF and T-PLSA are also compared empirically in terms of objective functions, decomposition results, clustering quality, and computation complexity on both synthetic and real-world datasets. Finally, we show that a hybrid method by running NTF and T-PLSA alternatively can successfully jump out of each other’s local minima and thus be able to achieve better clustering performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Acar E, Yener B (2007) Unsupervised multiway data analysis: a literature survey. Technical report, Computer Science Department, Rensselaer Polytechnic Institute

  2. Smilde A, Bro R, Geladi P (2004) Multi-way analysis: applications in the chemical sciences. Wiley, New York

    Book  Google Scholar 

  3. Bader BW, Harshman RA, Kolda TG (2007) Temporal analysis of semantic graphs using asalsan. In: Proceedings of the ICDM07, pp 33–42, October 2007

  4. Chi Y, Zhu S, Gong Y, Zhang Y (2008) Probabilistic polyadic factorization and its application to personalized recommendation. In: Proceedings of the 17th ACM conference on information and knowledge management (CIKM 2008), pp 941–950. ACM Press, New York

    Chapter  Google Scholar 

  5. Ding C, He X, Simon H (2005) On the equivalence of nonnegative matrix factorization and spectral clustering. In: Proceedings of the SIAM international conference on data mining (SDM 2005)

  6. Ding C, Li T, Jordan MI (2009) Convex and semi-nonnegative matrix factorizations. IEEE Trans Pattern Anal Mach Intell 32(99):1

    MATH  Google Scholar 

  7. Ding C, Li T, Peng W (2008) On the equivalence between non-negative matrix factorization and probabilistic latent semantic indexing. Comput Statist Data Anal 52(8):3913–3927

    Article  MathSciNet  MATH  Google Scholar 

  8. Ding C, Li T, Peng W, Park H (2006) Orthogonal nonnegative matrix t-factorizations for clustering. In: Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining, pp 126–135

  9. Duda RO, Hart PE, Stork DG (2000) Pattern classification (2nd edn). Wiley-Interscience, New York

    Google Scholar 

  10. Evrim Acar MK, Camtepe SA, Yener B (2005) Modeling multiway analysis of chatroom tensors. Proc IEEE Int Conf Intell Secur Inform 3495:256–268

    Google Scholar 

  11. Gaussier E, Goutte C (2005) Relation between PLSA and NMF and implications. In: Proceeding of the annual international ACM SIGIR conference (SIGIR 2005). ACM Press, New York, pp 601–602

    Google Scholar 

  12. Milligan GW (1985) An algorithm for generating artificial test clusters. Psychometrika 50:123–127

    Article  Google Scholar 

  13. Milligan GW, Cooper MC (1986) A study of the comparability of external criteria for hierarchical cluster analysis. Multivar Behav Res 21:846–850

    Article  Google Scholar 

  14. Hofmann T (1999) Probabilistic latent semantic indexing. In: Proceedings of the annual international ACM SIGIR conference (SIGIR 1999). ACM Press, New York, pp 50–57

    Google Scholar 

  15. Hofmann T (2001) Unsupervised learning by probabilistic latent semantic analysis. Mach Learn 42(1–2):177–196

    Article  MATH  Google Scholar 

  16. Kolda T (2001) Orthogonal tensor decomposition. SIAM J Matrix Anal Appl 23:243–255

    Article  MathSciNet  MATH  Google Scholar 

  17. Kolda TG, Bader BW (2006) The tophits model for higher-order web link analysis. In: Workshop on link analysis, counterterrorism and security

  18. Lee D, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401:788–791

    Article  Google Scholar 

  19. Lee D, Seung HS (2001) Algorithms for non-negatvie matrix factorization. In: Dietterich TG, Tresp V (eds) Advances in neural information processing systems, vol 13. MIT Press, Cambridge

    Google Scholar 

  20. Lee DD, Seung HS (2000) Algorithms for non-negative matrix factorization. In: NIPS, pp 556–562

  21. Lee DD, Seung SH (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788–791

    Article  Google Scholar 

  22. Li T (2008) Clustering based on matrix approximation: a unifying view. Knowl Inform Syst J 17(1):1–15

    Article  MATH  Google Scholar 

  23. Li T, Ding C (2006) The relationships among various nonnegative matrix factorization methods for clustering. In: Proceedings of the 2006 IEEE international conference on data mining (ICDM 2006), pp 362–371

  24. Pauca VP, Shahnaz F, Berry M, Plemmons R (2004) Text mining using non-negative matrix factorization. In: Proceedings of the SIAM international conference on data mining (SDM 2004), pp 452–456

  25. Peng W (2009) Equivalence between nonnegative tensor factorization and tensorial probabilistic latent semantic analysis. In: Proceedings of the annual international ACM SIGIR conference (SIGIR 2009), pp 668–669

  26. Peng W, Li T (2008) Author-topic evolution analysis using three-way non-negative paratucker. In: Proceedings of the annual international ACM SIGIR conference (SIGIR 2008). ACM Press, New York, pp 819–820

    Google Scholar 

  27. Peng W, Li T, Shao B (2008) Clustering multi-way data via adaptive subspace iteration. In: Proceedings of the 17th ACM conference on information and knowledge management (CIKM 2008). ACM Press, New York, pp 1519–1520

    Chapter  Google Scholar 

  28. Priebe C, Conroy J, Marchette D, Park Y (2006) Enron data set. http://cis.jhu.edu/~parky/Enron/enron.html

  29. Harshman RA (1970) Foundations of the parafac procedure: models and conditions for an ‘explanatory’ multi-modal factor analysis. UCLA Work Pap Phon 16:1–84

    Google Scholar 

  30. Schwarz G (1978) Estimating the dimension of a model. Ann Stat 6(2):461–464

    Article  MATH  Google Scholar 

  31. Shashanka M, Raj B, Smaragdis P (2008) Probabilistic latent variable models as nonnegative factorizations. Comput Intell Neurosci doi:10.1155/2008/947438

    Google Scholar 

  32. Shashua A, Hazan T (2005) Non-negative tensor factorization with applications to statistics and computer vision. In: Proceedings of the 22nd international conference on machine learning (ICML 2005). ACM Press, New York, pp 792–799

    Chapter  Google Scholar 

  33. Strehl A, Ghosh J (2002) Cluster ensembles—a knowledge reuse framework for combining multiple partitions. J Mach Learn Res 3:583–617

    MathSciNet  Google Scholar 

  34. Sun J, Zeng H, Liu H, Lu Y, Chen Z (2005) Cubesvd: a novel approach to personalized web search. In: Proceedings of the 14th international conference on World Wide Web, pp 652–662

  35. Vasilescu MAO, Terzopoulos D (2002) Multilinear analysis of image ensembles: tensorfaces. In: Proceedings of the 7th European conference on computer vision—part I (ECCV’02), pp 447–460

  36. Vichi M, Rocci R, Kiers HAL (2007) Simultaneous component and clustering models for three-way data: within and between approaches. J Classif 24(1):71–98

    Article  MathSciNet  MATH  Google Scholar 

  37. Wang H, Ahuja N (2005) Rank-r approximation of tensors: using image-as-matrix representation. In: CVPR ’05, vol 2, pp 346–353

  38. Xu W, Liu X, Gong Y (2003) Document clustering based on non-negative matrix factorization. In: Proceedings of ACM conference of research and development in IR(SIRGIR), pp 267–273, Toronto, Canada

  39. Zhang T, Golub G (2001) Rank-one approximation to high order tensor. SIAM J Matrix Anal Appl 23:534–550

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Xerox Innovation Group, Xerox Corporation, Rochester, NY, 14580, USA

    Wei Peng

  2. School of Computer Science, Florida International University, Miami, FL, 33199, USA

    Tao Li

Authors
  1. Wei Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tao Li.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Peng, W., Li, T. On the equivalence between nonnegative tensor factorization and tensorial probabilistic latent semantic analysis. Appl Intell 35, 285–295 (2011). https://doi.org/10.1007/s10489-010-0220-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-010-0220-9

Keywords

Search

Navigation