Dingyuan Zhu
ABSTRACT
Network embedding, aiming to embed a network into a low dimensional vector space while preserving the inherent structural properties of the network, has attracted considerable attentions recently. Most of the existing embedding methods embed nodes as point vectors in a low-dimensional continuous space. In this way, the formation of the edge is deterministic and only determined by the positions of the nodes. However, the formation and evolution of real-world networks are full of uncertainties, which makes these methods not optimal. To address the problem, we propose a novel Deep Variational Network Embedding in Wasserstein Space (DVNE) in this paper. The proposed method learns a Gaussian distribution in the Wasserstein space as the latent representation of each node, which can simultaneously preserve the network structure and model the uncertainty of nodes. Specifically, we use 2-Wasserstein distance as the similarity measure between the distributions, which can well preserve the transitivity in the network with a linear computational cost. Moreover, our method implies the mathematical relevance of mean and variance by the deep variational model, which can well capture the position of the node by the mean vectors and the uncertainties of nodes by the variance. Additionally, our method captures both the local and global network structure by preserving the first-order and second-order proximity in the network. Our experimental results demonstrate that our method can effectively model the uncertainty of nodes in networks, and show a substantial gain on real-world applications such as link prediction and multi-label classification compared with the state-of-the-art methods.
Supplemental Material
- Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré . 2008. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media.Google Scholar
- Aleksandar Bojchevski and Stephan Günnemann . 2017. Deep gaussian embedding of attributed graphs: Unsupervised inductive learning via ranking. arXiv preprint arXiv:1707.03815 (2017).Google Scholar
- A. Bojchevski and S. Günnemann . 2017. Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking. ArXiv e-prints (July . 2017). showeprint{arxiv}stat.ML/1707.03815Google Scholar
- Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister . 2015. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision Vol. 51, 1 (2015), 22--45. Google ScholarDigital Library
- Nicolas Bonneel, Michiel Van De Panne, Sylvain Paris, and Wolfgang Heidrich . 2011. Displacement interpolation using Lagrangian mass transport ACM Transactions on Graphics (TOG), Vol. Vol. 30. ACM, 158. Google ScholarDigital Library
- Victor Bryant . 1985. Metric spaces: iteration and application. Cambridge University Press.Google Scholar
- Chen Chen and Hanghang Tong . 2015. Fast eigen-functions tracking on dynamic graphs. In Proceedings of the 2015 SIAM International Conference on Data Mining. SIAM, 559--567.Google ScholarCross Ref
- Siheng Chen, Sufeng Niu, Leman Akoglu, Jelena Kovavcević, and Christos Faloutsos . 2017. Fast, Warped Graph Embedding: Unifying Framework and One-Click Algorithm. arXiv preprint arXiv:1702.05764 (2017).Google Scholar
- Philippe Clement and Wolfgang Desch . 2008. An elementary proof of the triangle inequality for the Wasserstein metric. Proc. Amer. Math. Soc. Vol. 136, 1 (2008), 333--339.Google ScholarCross Ref
- Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter . 2015. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015).Google Scholar
- Nicolas Courty, Rémi Flamary, and Mélanie Ducoffe . 2017 a. Learning Wasserstein Embeddings. arXiv preprint arXiv:1710.07457 (2017).Google Scholar
- Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy . 2017 b. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence Vol. 39, 9 (2017), 1853--1865.Google Scholar
- Peng Cui, Xiao Wang, Jian Pei, and Wenwu Zhu . 2017. A Survey on Network Embedding. arXiv preprint arXiv:1711.08752 (2017).Google Scholar
- Marco Cuturi and Arnaud Doucet . 2014. Fast computation of Wasserstein barycenters. In International Conference on Machine Learning. 685--693. Google ScholarDigital Library
- Fernando De Goes, Katherine Breeden, Victor Ostromoukhov, and Mathieu Desbrun . 2012. Blue noise through optimal transport. ACM Transactions on Graphics (TOG) Vol. 31, 6 (2012), 171. Google ScholarDigital Library
- Carl Doersch . 2016. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908 (2016).Google Scholar
- Ludovic Dos Santos, Benjamin Piwowarski, and Patrick Gallinari . 2016. Multilabel classification on heterogeneous graphs with gaussian embeddings Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 606--622.Google Scholar
- Tom Fawcett . 2006. An introduction to ROC analysis. Pattern recognition letters Vol. 27, 8 (2006), 861--874. Google ScholarDigital Library
- Bent Fuglede and Flemming Topsoe . 2004. Jensen-Shannon divergence and Hilbert space embedding Information Theory, 2004. ISIT 2004. Proceedings. International Symposium on. IEEE, 31.Google Scholar
- Clark R Givens, Rae Michael Shortt, et almbox. . 1984. A class of Wasserstein metrics for probability distributions. The Michigan Mathematical Journal Vol. 31, 2 (1984), 231--240.Google ScholarCross Ref
- Xavier Glorot and Yoshua Bengio . 2010. Understanding the difficulty of training deep feedforward neural networks Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 249--256.Google Scholar
- Aditya Grover and Jure Leskovec . 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 855--864. Google ScholarDigital Library
- Steve R Gunn et almbox. . 1998. Support vector machines for classification and regression. ISIS technical report Vol. 14, 1 (1998), 5--16.Google Scholar
- Shizhu He, Kang Liu, Guoliang Ji, and Jun Zhao . 2015. Learning to represent knowledge graphs with gaussian embedding Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. ACM, 623--632. Google ScholarDigital Library
- Paul W Holland and Samuel Leinhardt . 1972. Holland and Leinhardt reply: some evidence on the transitivity of positive interpersonal sentiment.Google Scholar
- Zhipeng Huang and Nikos Mamoulis . 2017. Heterogeneous Information Network Embedding for Meta Path based Proximity. arXiv preprint arXiv:1701.05291 (2017).Google Scholar
- Thomas N Kipf and Max Welling . 2016. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016).Google Scholar
- Solomon Kullback and Richard A Leibler . 1951. On information and sufficiency. The annals of mathematical statistics Vol. 22, 1 (1951), 79--86.Google Scholar
- Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang . 2006. A tutorial on energy-based learning. Predicting structured data Vol. 1, 0 (2006).Google Scholar
- Jure Leskovec and Julian J Mcauley . 2012. Learning to discover social circles in ego networks Advances in neural information processing systems. 539--547. Google ScholarDigital Library
- David Liben-Nowell and Jon Kleinberg . 2007. The link-prediction problem for social networks. journal of the Association for Information Science and Technology Vol. 58, 7 (2007), 1019--1031. Google ScholarDigital Library
- Laurens van der Maaten and Geoffrey Hinton . 2008. Visualizing data using t-SNE. Journal of Machine Learning Research Vol. 9, Nov (2008), 2579--2605.Google Scholar
- Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore . 2000. Automating the construction of internet portals with machine learning. Information Retrieval Vol. 3, 2 (2000), 127--163. Google ScholarDigital Library
- Vinod Nair and Geoffrey E Hinton . 2010. Rectified linear units improve restricted boltzmann machines Proceedings of the 27th international conference on machine learning (ICML-10). 807--814. Google ScholarDigital Library
- Feiping Nie, Wei Zhu, and Xuelong Li . 2017. Unsupervised Large Graph Embedding.. In AAAI. 2422--2428.Google Scholar
- Mingdong Ou, Peng Cui, Jian Pei, Ziwei Zhang, and Wenwu Zhu . 2016. Asymmetric transitivity preserving graph embedding Proc. of ACM SIGKDD. 1105--1114. Google ScholarDigital Library
- Bryan Perozzi, Rami Al-Rfou, and Steven Skiena . 2014. Deepwalk: Online learning of social representations Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 701--710. Google ScholarDigital Library
- Zafarani Reza and Liu Huan . 2009. Social Computing Data Repository. (2009).Google Scholar
- Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei . 2015. Line: Large-scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web. ACM, 1067--1077. Google ScholarDigital Library
- Tijmen Tieleman and Geoffrey Hinton . 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning Vol. 4, 2 (2012), 26--31.Google Scholar
- Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf . 2017. Wasserstein Auto-Encoders. arXiv preprint arXiv:1711.01558 (2017).Google Scholar
- Ke Tu, Peng Cui, Xiao Wang, Fei Wang, and Wenwu Zhu . 2017. Structural Deep Embedding for Hyper-Networks. arXiv preprint arXiv:1711.10146 (2017).Google Scholar
- Luke Vilnis and Andrew McCallum . 2014. Word representations via gaussian embedding. arXiv preprint arXiv:1412.6623 (2014).Google Scholar
- Daixin Wang, Peng Cui, and Wenwu Zhu . 2016. Structural deep network embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 1225--1234. Google ScholarDigital Library
- Huahua Wang and Arindam Banerjee . 2014. Bregman alternating direction method of multipliers Advances in Neural Information Processing Systems. 2816--2824. Google ScholarDigital Library
- Xiao Wang, Peng Cui, Jing Wang, Jian Pei, Wenwu Zhu, and Shiqiang Yang . 2017. Community Preserving Network Embedding. (2017).Google Scholar
- Chengxi Zang, Peng Cui, Christos Faloutsos, and Wenwu Zhu . 2017. Long Short Memory Process: Modeling Growth Dynamics of Microscopic Social Connectivity Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 565--574. Google ScholarDigital Library
Index Terms
- Deep Variational Network Embedding in Wasserstein Space
Recommendations
Structural Deep Network Embedding
KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data MiningNetwork embedding is an important method to learn low-dimensional representations of vertexes in networks, aiming to capture and preserve the network structure. Almost all the existing network embedding methods adopt shallow models. However, since the ...
Cross-Network Embedding for Multi-Network Alignment
WWW '19: The World Wide Web ConferenceRecently, data mining through analyzing the complex structure and diverse relationships on multi-network has attracted much attention in both academia and industry. One crucial prerequisite for this kind of multi-network mining is to map the nodes ...
Co-Regularized Deep Multi-Network Embedding
WWW '18: Proceedings of the 2018 World Wide Web ConferenceNetwork embedding aims to learn a low-dimensional vector representation for each node in the social and information networks, with the constraint to preserve network structures. Most existing methods focus on single network embedding, ignoring the ...
Comments