Dustin L. Arendt
ABSTRACT
We introduce "Parallel Embeddings", a new technique that generalizes the classical Parallel Coordinates visualization technique to sequences of learned representations. This visualization technique is designed for concept-oriented "model comparison" tasks, allowing data scientists to understand qualitative differences in how models interpret input data. We compare user performance with our tool against Tensor Board Embedding Projector for understanding model accuracy and qualitative model differences. With our tool, users were more accurate and learned strategies for the tasks more quickly. Furthermore, users' analytical process in the comparison condition was positively influenced by using our tool beforehand.
- Eric Alexander and Michael Gleicher. 2015. Task-driven comparison of topic models. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2015), 320--329.Google Scholar
Digital Library
- Saleema Amershi, Max Chickering, Steven M Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. 2015. Modeltracker: Redesigning performance analysis tools for machine learning. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 337--346.Google ScholarDigital Library
- Dustin Arendt and Meg Pirrung. 2017. The "y" of it Matters, Even for Storyline Visualization. In 2017 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, 81--91.Google Scholar
- David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6541--6549.Google ScholarCross Ref
- James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, Feb (2012), 281--305.Google ScholarDigital Library
- Pierre Dragicevic. 2016. Fair statistical communication in HCI. In Modern Statistical Methods for HCI. Springer, 291--330.Google Scholar
- Tim Dwyer and Lev Nachmanson. 2009. Fast edge-routing for large graphs. In International Symposium on Graph Drawing. Springer, 147--158.Google Scholar
- Niklas Elmqvist and Jean-Daniel Fekete. 2009. Hierarchical aggregation for information visualization: Overview, techniques, and design guidelines. IEEE Transactions on Visualization and Computer Graphics 16, 3 (2009), 439--454.Google ScholarDigital Library
- Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and Robust Automated Machine Learning. In Advances in Neural Information Processing Systems 28, C. Cortes, N.D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (Eds.). Curran Associates, Inc., 2962--2970.Google ScholarDigital Library
- Kārlis Freivalds. 2001. Curved edge routing. In International Symposium on Fundamentals of Computation Theory. Springer, 126--137.Google ScholarCross Ref
- Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in Psychology. Vol. 52. Elsevier, 139--183.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 770--778.Google ScholarCross Ref
- Julian Heinrich and Daniel Weiskopf. 2013. State of the Art of Parallel Coordinates.. In Eurographics (STARs). 95--116.Google Scholar
- Fred Matthew Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE Transactions on Visualization and Computer Graphics (2018).Google ScholarDigital Library
- Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. 2018. Human-level performance in first-person multi-player games with population-based deep reinforcement learning. arXiv preprint arXiv:1807.01281 (2018).Google Scholar
- Jimmy Johansson, Patric Ljung, Mikael Jern, and Matthew Cooper. 2005. Revealing structure within clustered parallel coordinates displays. In IEEE Symposium on Information Visualization, 2005. INFOVIS 2005. IEEE, 125--132.Google ScholarCross Ref
- Minsuk Kahng, Pierre Y Andrews, Aditya Kalro, and Duen Horng Polo Chau. 2017. ActiVis: Visual exploration of industry-scale deep neural network models. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 88--97.Google ScholarCross Ref
- Minsuk Kahng, Dezhi Fang, and Duen Horng Polo Chau. 2016. Visual exploration of machine learning results using data cube analysis. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics. ACM, 1.Google ScholarDigital Library
- Wiebke Köpp and Tino Weinkauf. 2018. Temporal treemaps: Static visualization of evolving trees. IEEE transactions on visualization and computer graphics 25, 1 (2018), 534--543.Google Scholar
- Antoine Lhuillier, Christophe Hurter, and Alexandru Telea. 2017. State of the art in edge and trail bundling techniques. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 619--645.Google Scholar
- Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology (1932).Google Scholar
- Dongyu Liu, Weiwei Cui, Kai Jin, Yuxiao Guo, and Huamin Qu. 2018. Deep-tracker: Visualizing the training process of convolutional neural networks. ACM Transactions on Intelligent Systems and Technology (TIST) 10, 1 (2018), 6.Google Scholar
- Mengchen Liu, Jiaxin Shi, Zhen Li, Chongxuan Li, Jun Zhu, and Shixia Liu. 2016. Towards better analysis of deep convolutional neural networks. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2016), 91--100.Google ScholarDigital Library
- Jonas Lukasczyk, Gunther Weber, Ross Maciejewski, Christoph Garth, and Heike Leitte. 2017. Nested tracking graphs. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 12--22.Google Scholar
- Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, Nov (2008), 2579--2605.Google Scholar
- Kevin T McDonnell and Klaus Mueller. 2008. Illustrative parallel coordinates. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 1031--1038.Google Scholar
- Leland McInnes, John Healy, and James Melville. 2018. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018).Google Scholar
- H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. 2013. Ad click prediction: a view from the trenches. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data mining. ACM, 1222--1230.Google ScholarDigital Library
- Paul Mihas. 2019. Qualitative data analysis. In Oxford Research Encyclopedia of Education.Google Scholar
- Paulo E Rauber, Samuel G Fadel, Alexandre X Falcao, and Alexandru C Telea. 2016. Visualizing the hidden activity of artificial neural networks. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2016), 101--110.Google ScholarDigital Library
- Khairi Reda, Chayant Tantipathananandh, Andrew Johnson, Jason Leigh, and Tanya Berger-Wolf. 2011. Visualizing the evolution of community structures in dynamic social networks. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 1061--1070.Google Scholar
- Hans Rosling and Zhongxing Zhang. 2011. Health advocacy with Gapminder animated statistics. Journal of epidemiology and global health 1, 1 (2011), 11--14.Google ScholarCross Ref
- Martin Rosvall and Carl T Bergstrom. 2010. Mapping change in large networks. PloS one 5, 1 (2010), e8694.Google ScholarCross Ref
- Emily Saldanha, Brenda Praggastis, Todd Billow, and Dustin L. Arendt. 2019. ReLVis: Visual Analytics for Situational Awareness During Reinforcement Learning Experimentation. In EuroVis 2019 - Short Papers, Jimmy Johansson, Filip Sadlo, and G. Elisabeta Marai (Eds.). The Eurographics Association.Google Scholar
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
- Daniel Smilkov, Nikhil Thorat, Charles Nicholson, Emily Reif, Fernanda B Viégas, and Martin Wattenberg. 2016. Embedding projector: Interactive visualization and interpretation of embeddings. In Proc. Neural Inf. Process. Syst. Workshop Interpretable ML Complex Syst.Google Scholar
- Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexander M Rush. 2018. Seq-2-seq-vis: A visual debugging tool for sequence-to-sequence models. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 353--363.Google ScholarDigital Library
- C. J. Van Rijsbergen. 1979. Information Retrieval (2nd ed.). Butterworth-Heinemann.Google Scholar
- Corinna Vehlow, Fabian Beck, and Daniel Weiskopf. 2017. Visualizing group structures in graphs: A survey. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 201--225.Google Scholar
- Junpeng Wang, Liang Gou, Han-Wei Shen, and Hao Yang. 2018. Dqnviz: A visual analytics approach to understand deep q-networks. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 288--298.Google ScholarDigital Library
- John Wenskovitch, Ian Crandell, Naren Ramakrishnan, Leanna House, and Chris North. 2017. Towards a systematic combination of dimension reduction and clustering in visual analytics. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 131--141.Google ScholarCross Ref
- Wathsala Widanagamaachchi, Cameron Christensen, Valerio Pascucci, and Peer-Timo Bremer. 2012. Interactive exploration of large-scale time-varying data using dynamic tracking graphs. In IEEE Symposium on Large Data Analysis and Visualization (LDAV). IEEE, 9--17.Google ScholarCross Ref
- Moritz Wittenhagen, Christian Cherek, and Jan Borchers. 2016. Chronicler: Interactive exploration of source code history. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3522--3532.Google ScholarDigital Library
- Wei Yu, Kuiyuan Yang, Yalong Bai, Hongxun Yao, and Yong Rui. 2014. Visualizing and comparing convolutional neural networks. arXiv preprint arXiv:1412.6631 (2014).Google Scholar
- Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. 2016. Graying the black box: Understanding DQNs. (2016), 1899--1908.Google Scholar
- Haipeng Zeng, Hammad Haleem, Xavier Plantaz, Nan Cao, and Huamin Qu. 2017. CNNComparator: Comparative analytics of convolutional neural networks. arXiv preprint arXiv:1710.05285 (2017).Google Scholar
- Jiawei Zhang, Yang Wang, Piero Molino, Lezhi Li, and David S Ebert. 2018. Manifold: A model-agnostic framework for interpretation and diagnosis of machine learning models. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 364--373.Google ScholarDigital Library
Index Terms
- Parallel embeddings: a visualization technique for contrasting learned representations
Recommendations
Generic tool for visualization of model differences
IWMCP '10: Proceedings of the 1st International Workshop on Model Comparison in PracticeModel comparison includes three major concerns: presentation, calculation, and visualization of model differences. In this paper we address the concern of visualization of model differences in the context of model configuration management systems. Since ...
Polyarchy visualization: visualizing multiple intersecting hierarchies
CHI '02: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsWe describe a new information structure composed of multiple intersecting hierarchies, which we call Polyarchies. Visualizing polyarchies enables use of novel views for discovery of relationships which are very difficult using existing hierarchy ...
Animated visualization of multiple intersecting hierarchies
The authors describe a new information structure composed of multiple intersecting hierarchies, which we call a Polyarchy. Visualizing polyarchies enables use of novel views for discovery of relationships which are very difficult using existing ...
Comments