Yishai Oltchik
ABSTRACT
Network contention frequently dominates the run time of parallel algorithms and limits scaling performance. Most previous studies mitigate or eliminate contention by utilizing one of several approaches: communication-minimizing algorithms; hotspot-avoiding routing schemes; topology-aware task mapping; or improving global network properties, such as bisection bandwidth, edge-expansion, partitioning, and network diameter. In practice, parallel jobs often use only a fraction of a host system. How do processor allocation policies affect contention within a partition?
We utilize edge-isoperimetric analysis of network graphs to determine whether a network partition defined by a processor allocation has optimal internal bisection. Increasing the bisection allows a more efficient use of the network resources, decreasing or completely eliminating the link contention. We study torus networks and characterize partition geometries that maximize internal bisection bandwidth, and examine the allocation policies of Mira and JUQUEEN, the two largest publicly-accessible Blue~Gene/Q torus-based supercomputers. Our analysis shows that the bisection bandwidth of their partitions can often be improved by changing the partitions' geometries, yielding up to a x2~speedup for contention-bound workloads. Benchmark experiments validate the predictions. Our analysis applies to allocation policies of other networks.
- Rudolf Ahlswede and Sergej L Bezrukov. Edge isoperimetric theorems for integer point arrays. Applied Mathematics Letters, 8(2):75--80, 1995.Google Scholar
Cross Ref
- Jung Ho Ahn, Nathan Binkert, Al Davis, Moray McLaren, and Robert S Schreiber. HyperX: topology, routing, and packaging of efficient large-scale networks. In Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, page 41. ACM, 2009.Google Scholar
- Yuuichirou Ajima, Tomohiro Inoue, Shinya Hiramoto, and Toshiyuki Shimizu. Tofu: Interconnect for the K computer. Fujitsu Sci. Tech. J, 48(3):280--285, 2012.Google Scholar
- Bob Alverson, Tim Johnson, Joe Kopnick, Mike Higgins, and James Reinhard. Cray Cascade: a Scalable HPC System based on a Dragonfly Network. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, page 103. IEEE Computer Society Press, 2012.Google Scholar
- Argonne National Laboratory website. Official machine overview of Mira. https://www.alcf.anl.gov/user-guides/machine-overview-bgq.Google Scholar
- Grey Ballard, E Carson, J Demmel, M Hoemmen, Nicholas Knight, and Oded Schwartz. Communication lower bounds and optimal algorithms for numerical linear algebra. Acta Numerica, 23:1--155, 2014.Google ScholarCross Ref
- Grey Ballard, James Demmel, Andrew Gearhart, Benjamin Lipshitz, Yishai Oltchik, Oded Schwartz, and Sivan Toledo. Network topologies and inevitable contention. In Communication Optimizations in HPC (COMHPC), International Workshop on, pages 39--52. IEEE, 2016.Google Scholar
- Grey Ballard, James Demmel, Olga Holtz, Benjamin Lipshitz, and Oded Schwartz. Communication-optimal parallel algorithm for Strassen's matrix multiplication. In Proceedings of the twenty-fourth annual ACM symposium on Parallelism in algorithms and architectures, pages 193--204. ACM, 2012.Google ScholarDigital Library
- Maciej Besta and Torsten Hoefler. Slim fly: A cost effective low-diameter network topology. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 348--359. IEEE Press, 2014.Google ScholarDigital Library
- Abhinav Bhatele, Nikhil Jain, Katherine E Isaacs, Ronak Buch, Todd Gamblin, Steven H Langer, and Laxmikant V Kale. Optimizing the performance of parallel applications on a 5d torus via task mapping. In High Performance Computing (HiPC), 2014 21st International Conference on, pages 1--10. IEEE, 2014.Google ScholarCross Ref
- Béla Bollobás and Imre Leader. Edge-isoperimetric inequalities in the grid. Combinatorica, 11(4):299--314, 1991.Google ScholarCross Ref
- Dong Chen, Noel Eisley, Philip Heidelberger, Sameer Kumar, Amith Mamidala, Fabrizio Petrini, Robert Senger, Yutaka Sugawara, Robert Walkup, Burkhard Steinmacher-Burow, Anamitra Choudhury, Yogish Sabharwal, Swati Singhal, and Jeffrey J. Parker. Looking under the hood of the IBM Blue Gene/Q network. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, page 69. IEEE Computer Society Press, 2012.Google Scholar
- Dong Chen, Noel Eisley, Philip Heidelberger, Robert Senger, Yutaka Sugawara, Sameer Kumar, Valentina Salapura, David Satterfield, Burkhard Steinmacher-Burow, and Jeffrey Parker. The IBM Blue Gene/Q interconnection fabric. IEEE Micro, 32(1):32--43, 2012.Google ScholarDigital Library
- James Demmel, Laura Grigori, Mark Hoemmen, and Julien Langou. Communication-optimal parallel and sequential QR and LU factorizations. SIAM Journal on Scientific Computing, 34(1):A206--A239, 2012.Google ScholarDigital Library
- Jack J Dongarra, Piotr Luszczek, and Antoine Petitet. The LINPACK benchmark: past, present and future. Concurrency and Computation: practice and experience, 15(9):803--820, 2003.Google Scholar
- Jack J Dongarra, Hans W Meuer, and Erich Strohmaier. Top500 Supercomputer Sites, November 2017, 2017. www.top500.org.Google Scholar
- Michael R Garey, David S Johnson, and Larry Stockmeyer. Some simplified NP-complete problems. In Proceedings of the sixth annual ACM symposium on Theory of computing, pages 47--63. ACM, 1974.Google ScholarDigital Library
- Lawrence Hueston Harper. Optimal assignments of numbers to vertices. Journal of the Society for Industrial and Applied Mathematics, 12(1):131--135, 1964.Google ScholarCross Ref
- Nikhil Jain, Abhinav Bhatele, Xiang Ni, Todd Gamblin, and Laxmikant V Kale. Partitioning low-diameter networks to eliminate inter-job interference. In Parallel and Distributed Processing Symposium (IPDPS), 2017 IEEE International, pages 439--448. IEEE, 2017.Google ScholarCross Ref
- Sangeetha Abdu Jyothi, Ankit Singla, P Brighten Godfrey, and Alexandra Kolla. Measuring and understanding throughput of network topologies. In High Performance Computing, Networking, Storage and Analysis, SC16: International Conference for, pages 761--772. IEEE, 2016.Google Scholar
- John Kim, Wiliam J Dally, Steve Scott, and Dennis Abts. Technology-driven, highly-scalable dragonfly topology. In Computer Architecture, 2008. ISCA'08. 35th International Symposium on, pages 77--88. IEEE, 2008.Google ScholarDigital Library
- Lawrence Livermore National Laboratory website. Sequoia supercomputer transitions to classified work. https://www.llnl.gov/news/sequoia-supercomputer-transitions-classified-work.Google Scholar
- Lawrence Livermore National Laboratory website. Using the Sequoia and Vulcan BG/Q Systems. https://computing.llnl.gov/tutorials/bgq/.Google Scholar
- James R Lee, Shayan Oveis Gharan, and Luca Trevisan. Multiway spectral partitioning and higher-order cheeger inequalities. Journal of the ACM (JACM), 61(6):37, 2014.Google Scholar
- John H Lindsey. Assignment of numbers to vertices. The American Mathematical Monthly, 71(5):508--516, 1964.Google ScholarCross Ref
- Benjamin Lipshitz, Grey Ballard, James Demmel, and Oded Schwartz. Communication-avoiding parallel strassen: Implementation and performance. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, page 101. IEEE Computer Society Press, 2012.Google Scholar
- NASA Website. Official machine overview of the Pleiades supercompter. https://www.nas.nasa.gov/hecc/resources/pleiades.html.Google Scholar
- Oak Ridge National Laboratory website. Official machine overview of Titan. https://www.olcf.ornl.gov/olcf-resources/compute-systems/titan/.Google Scholar
- Yishai Oltchik and Oded Schwartz. Network partitioning and avoidable contention. arXiv, 2020.Google Scholar
- Arjun Singh. Load-balanced routing in interconnection networks. PhD thesis, Stanford University, 2005.Google Scholar
- Edgar Solomonik and James Demmel. Communication-optimal parallel 2.5D matrix multiplication and LU factorization algorithms. In European Conference on Parallel Processing, pages 90--109. Springer, 2011.Google ScholarCross Ref
- Asaf Valadarsky, Michael Dinitz, and Michael Schapira. Xpander: unveiling the secrets of high-performance datacenters. In Proceedings of the 14th ACM Workshop on Hot Topics in Networks, page 16. ACM, 2015.Google Scholar
Index Terms
- Network Partitioning and Avoidable Contention
Recommendations
A case study in using massively parallel simulation for extreme-scale torus network codesign
SIGSIM PADS '14: Proceedings of the 2nd ACM SIGSIM Conference on Principles of Advanced Discrete SimulationA high-bandwidth, low-latency interconnect will be a critical component of future exascale systems. The torus network topology, which uses multidimensional network links to improve path diversity and exploit locality between nodes, is a potential ...
The Effects of Problem Partitioning, Allocation, and Granularity on the Performance of Multiple-Processor Systems
In this paper we analyze the effects of the problem decomposition, the allocation of subproblems to processors, and the grain size of subproblems on the performance of a multiple- processor shared-memory architecture. Our results indicate that for ...
Interlacing Bypass Rings to Torus Networks for More Efficient Networks
We introduce a new technique for generating more efficient networks by systematically interlacing bypass rings to torus networks (iBT networks). The resulting network can improve the original torus network by reducing the network diameter, node-to-node ...
Comments