Bryce Cronkite-Ratcliff
Nick McKeown
ABSTRACT
New congestion control algorithms are rapidly improving datacenters by reducing latency, overcoming incast, increasing throughput and improving fairness. Ideally, the operating system in every server and virtual machine is updated to support new congestion control algorithms. However, legacy applications often cannot be upgraded to a new operating system version, which means the advances are off-limits to them. Worse, as we show, legacy applications can be squeezed out, which in the worst case prevents the entire network from adopting new algorithms.
Our goal is to make it easy to deploy new and improved congestion control algorithms into multitenant datacenters, without having to worry about TCP-friendliness with non-participating virtual machines. This paper presents a solution we call virtualized congestion control. The datacenter owner may introduce a new congestion control algorithm in the hypervisors. Internally, the hypervisors translate between the new congestion control algorithm and the old legacy congestion control, allowing legacy applications to enjoy the benefits of the new algorithm. We have implemented proof-of-concept systems for virtualized congestion control in the Linux kernel and in VMware’s ESXi hypervisor, achieving improved fairness, performance, and control over guest bandwidth allocations.
- Mohammad Alizadeh, Albert Greenberg, David A Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. Data Center TCP (DCTCP). ACM SIGCOMM, 2011. Google Scholar
Digital Library
- Mohammad Alizadeh, Adel Javanmard, and Balaji Prabhakar. Analysis of DCTCP: stability, convergence, and fairness. ACM SIGMETRICS, 2011. Google ScholarDigital Library
- Costin Raiciu, Sebastien Barre, Christopher Pluntke, Adam Greenhalgh, Damon Wischik, and Mark Handley. Improving datacenter performance and robustness with multipath TCP. ACM SIGCOMM, 2011. Google ScholarDigital Library
- Balajee Vamanan, Jahangir Hasan, and TN Vijaykumar. Deadline-aware Datacenter TCP (D2TCP). ACM SIGCOMM, 2012. Google ScholarDigital Library
- Haitao Wu, Jiabo Ju, Guohan Lu, Chuanxiong Guo, Yongqiang Xiong, and Yongguang Zhang. Tuning ECN for data center networks. ACM CoNEXT, 2012. Google ScholarDigital Library
- Brent Stephens, Alan L Cox, Anubhav Singla, Jenny Carter, Colin Dixon, and Wes Felter. Practical DCB for improved data center networks. IEEE Infocom, 2014.Google ScholarCross Ref
- Glenn Judd. Attaining the promise and avoiding the pitfalls of TCP in the datacenter. USENIX NSDI, 2015. Google ScholarDigital Library
- Radhika Mittal, Nandita Dukkipati, Emily Blem, Hassan Wassel, Monia Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, David Zats, et al. TIMELY: RTT-based congestion control for the datacenter. ACM SIGCOMM, 2015. Google ScholarDigital Library
- Changhyun Lee, Chunjong Park, Keon Jang, Sue Moon, and Dongsu Han. Accurate latency-based congestion feedback for datacenters. USENIX ATC, 2015. Google ScholarDigital Library
- Qingxi Li, Mo Dong, and Brighten Godfrey. Halfback: Running short flows quickly and safely. ACM CoNEXT, 2015.Google ScholarDigital Library
- Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. Congestion control for large-scale RDMA deployments. ACM SIGCOMM, 2015. Google ScholarDigital Library
- Prasanthi Sreekumari and Jae-il Jung. Transport protocols for data center networks: a survey of issues, solutions and challenges. Photonic Network Communications, pages 1–17, 2015. Google ScholarDigital Library
- Keith Winstein and Hari Balakrishnan. TCP ex machina: Computer-generated congestion control. ACM SIGCOMM, 2013. Google ScholarDigital Library
- Anirudh Sivaraman, Keith Winstein, Pratiksha Thaker, and Hari Balakrishnan. An experimental study of the learnability of congestion control. ACM SIGCOMM, 2014. Google ScholarDigital Library
- Mo Dong, Qingxi Li, Doron Zarchy, Brighten Godfrey, and Michael Schapira. Rethinking congestion control architecture: Performance-oriented congestion control. ACM SIGCOMM, 2014. Google ScholarDigital Library
- Mirja Kuhlewind, David P Wagner, Juan Manuel Reyes Espinosa, and Bob Briscoe. Using Data Center TCP (DCTCP) in the Internet. IEEE Globecom Workshops, 2014.Google Scholar
- Eitan Zahavi, Alexander Shpiner, Ori Rottenstreich, Avinoam Kolodny, and Isaac Keslassy. Links as a Service (LaaS): Guaranteed tenant isolation in the shared cloud. ACM/IEEE ANCS, 2016. Google ScholarDigital Library
- Stefano Vissicchio, Olivier Tilmans, Laurent Vanbever, and Jennifer Rexford. Central control over distributed routing. ACM SIGCOMM, 2015. Google ScholarDigital Library
- Michele Luglio, M Yahya Sanadidi, Mario Gerla, and James Stepanek. On-board satellite split TCP proxy. IEEE J. Select. Areas Commun., 22(2):362–370, 2004. Google ScholarDigital Library
- Xiang Chen, Hongqiang Zhai, Jianfeng Wang, and Yuguang Fang. A survey on improving TCP performance over wireless networks. Resource management in wireless networking, 2005.Google Scholar
- vCC project. http://webee.technion.ac.il/ isaac/vcc/.Google Scholar
- Lampros Kalampoukas, Anujan Varma, and KK Ramakrishnan. Explicit window adaptation: A method to enhance TCP performance. IEEE Infocom, 1998.Google ScholarCross Ref
- Shrikrishna Karandikar, Shivkumar Kalyanaraman, Prasad Bagal, and Bob Packer. TCP rate control. ACM SIGCOMM, 2000. Google ScholarDigital Library
- James Aweya, Michel Ouellette, and Delfin Montuno. A self-regulating TCP acknowledgment (ACK) pacing scheme. International Journal of Network Management, 12(3):145–163, 2002. Google ScholarDigital Library
- Huan-Yun Wei, Shih-Chiang Tsao, and Ying-Dar Lin. Assessing and improving TCP rate shaping over edge gateways. IEEE Trans. Comput., 53(3):259–275, 2004. Google ScholarDigital Library
- Haiqing Jiang, Yaogong Wang, Kyunghan Lee, and Injong Rhee. Tackling bufferbloat in 3G/4G networks. IMC, 2012. Google ScholarDigital Library
- Tal Garfinkel and Mendel Rosenblum. A virtual machine introspection based architecture for intrusion detection. NDSS, 2003.Google Scholar
- Bryan D Payne, Martim Carbone, Monirul Sharif, and Wenke Lee. Lares: An architecture for secure active monitoring using virtualization. IEEE Symposium on Security and Privacy, 2008. Google ScholarDigital Library
- VMsafe. https://www.vmware.com/company/news/releases/vmsafe_vmworld.Google Scholar
- Hari Balakrishnan, Srinivasan Seshan, and Randy H Katz. Improving reliable transport and handoff performance in cellular wireless networks. Wireless Networks, 1(4):469–481, 1995. Google ScholarDigital Library
- RFC 5681. https://tools.ietf.org/html/rfc5681.Google Scholar
- Stefan Savage, Neal Cardwell, David Wetherall, and Tom Anderson. TCP congestion control with a misbehaving receiver. ACM SIGCOMM, 1999. Google ScholarDigital Library
- RFC 6582. https://tools.ietf.org/html/rfc6582.Google Scholar
- Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, Bob Lantz, and Nick McKeown. Reproducible network experiments using container-based emulation. ACM CoNEXT, 2012. Google ScholarDigital Library
- Sally Floyd. TCP and explicit congestion notification. ACM SIGCOMM, 1994. Google ScholarDigital Library
- Mirja Kühlewind, Sebastian Neuner, and Brian Trammell. On the state of ECN and TCP options on the Internet. International Conference on Passive and Active Measurement, 2013. Google ScholarDigital Library
- Yin Zhang and Lili Qiu. Understanding the end-to-end performance impact of RED in a heterogeneous environment. Technical report, Cornell, 2000. Google ScholarDigital Library
- VMware vSphere DVFilter. https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-639ED633-A89A-470F-8056-5BB71E8C3F8F.html.Google Scholar
- Alok Kumar, Sushant Jain, Uday Naik, Nikhil Kasinadhuni, Enrique Cauich Zermeno, C. Stephen Gunn, Jing Ai, Bjorn Carlin, Mihai Amarandei-Stavila, Mathieu Robin, Aspi Siganporia, Stephen Stuart, and Amin Vahdat. BwE: Flexible, hierarchical bandwidth allocation for WAN distributed computing. ACM SIGCOMM, 2015. Google ScholarDigital Library
- Microsoft Hyper-V Extensible Switch. https://msdn.microsoft.com/en-us/library/windows/hardware/jj673961%28v=vs.85%29.aspx.Google Scholar
- Sivasankar Radhakrishnan, Yilong Geng, Vimalkumar Jeyakumar, Abdul Kabbani, George Porter, and Amin Vahdat. Senic: Scalable NIC for end-host rate limiting. USENIX NSDI, 2014. Google ScholarDigital Library
- Radhika Niranjan Mysore, George Porter, and Amin Vahdat. FasTrak: enabling express lanes in multi-tenant data centers. ACM CoNEXT, 2013. Google ScholarDigital Library
- Jeffrey C Mogul, Jayaram Mudigonda, Jose Renato Santos, and Yoshio Turner. The NIC is the hypervisor: bare-metal guests in IaaS clouds. 2013.Google Scholar
- Sangjin Han, Keon Jang, Aurojit Panda, Shoumik Palkar, Dongsu Han, and Sylvia Ratnasamy. SoftNIC: A software NIC to augment hardware. Technical Report UCB/EECS-2015-155, UC Berkeley, 2015.Google Scholar
- Stephen Kent and Karen Seo. Security Architecture for the Internet Protocol. RFC 4301, RFC Editor, December 2005.Google Scholar
- Andrea Bittau, Michael Hamburg, Mark Handley, David Mazières, and Dan Boneh. The case for ubiquitous transport-level encryption. USENIX Security, 2010. Google ScholarDigital Library
- Keqiang He, Eric Rozner, Agarwal Kanak, Yu Gu, Wes Felter, John Carter, and Aditya Akella. AC/DC TCP: Virtual congestion control enforcement for datacenter networks. ACM SIGCOMM, 2016. Google ScholarDigital Library
- F5 Networks. Optimize WAN and LAN application performance with TCP Express. 2007.Google Scholar
- Hitesh Ballani, Paolo Costa, Thomas Karagiannis, and Ant Rowstron. Towards predictable datacenter networks. ACM SIGCOMM, 2011. Google ScholarDigital Library
- Chuanxiong Guo, Guohan Lu, Helen J Wang, Shuang Yang, Chao Kong, Peng Sun, Wenfei Wu, and Yongguang Zhang. Secondnet: a data center network virtualization architecture with bandwidth guarantees. ACM CoNEXT, 2010. Google ScholarDigital Library
- Alan Shieh, Srikanth Kandula, Albert G Greenberg, Changhoon Kim, and Bikas Saha. Sharing the data center network. USENIX NSDI, 2011. Google ScholarDigital Library
- Vimalkumar Jeyakumar, Mohammad Alizadeh, David Changhoon Kim, and Albert Greenberg. EyeQ: Practical network performance isolation at the edge. USENIX NSDI, 2013. Google ScholarDigital Library
- Lucian Popa, Praveen Yalagandula, Sujata Banerjee, Jeffrey C. Mogul, Yoshio Turner, and Jose Renato Santos. Elasticswitch: Practical work-conserving bandwidth guarantees for cloud computing. ACM SIGCOMM, 2013. Google ScholarDigital Library
- Gautam Kumar, Srikanth Kandula, Peter Bodik, and Ishai Menache. Virtualizing traffic shapers for practical resource allocation. USENIX HotCloud, 2013.Google Scholar
Index Terms
- Virtualized Congestion Control
Recommendations
Server consolidation with migration control for virtualized data centers
Virtualization has become a key technology for simplifying service management and reducing energy costs in data centers. One of the challenges faced by data centers is to decide when, how, and which virtual machines (VMs) have to be consolidated into a ...
Transparently bridging semantic gap in CPU management for virtualized environments
Consolidated environments are progressively accommodating diverse and unpredictable workloads in conjunction with virtual desktop infrastructure and cloud computing. Unpredictable workloads, however, aggravate the semantic gap between the virtual ...
Host Congestion Control
ACM SIGCOMM '23: Proceedings of the ACM SIGCOMM 2023 ConferenceThe conventional wisdom in systems and networking communities is that congestion happens primarily within the network fabric. However, adoption of high-bandwidth access links and relatively stagnant technology trends for resources within hosts have led ...
Comments