ABSTRACT
RDMA has been widely adopted as a promising solution for high performance networks, but is still unavailable for a large number of socket-based applications running in public clouds due to the following reasons. There is no available virtualization technique of RDMA that can meet the cloud's requirements. Moreover, it is cost prohibitive to rewrite the socket-based applications with the Verbs API. To address the above problems, we present vSocket, a software-based RDMA virtualization framework for socket-based applications in public clouds. vSocket takes into account the demands of clouds such as security rules and network isolation, so it can be deployed in the current public clouds. Furthermore, vSocket provides native socket API so that socket-based applications can use it without any modifications. Finally, to validate the performance gains, we implemented a prototype and compared it with current virtual network solutions against 1) basic network benchmarks and 2) the Redis, a typical I/O intensive application. Experimental results show that the latency of basic benchmarks can be reduced by 88% and the throughput of Redis is improved by 4 times.
- 2018. Cloudius Systems. Seastar. http://seastar.io/ [Online; accessed 8-December-2018].Google Scholar
- 2018. Mellanox ConnectX-4 adaptor. http://www.mellanox.com/page/products_dyn?product_family=201&mtag=connectx_4_vpi_card. [Online; accessed 19-September-2018].Google Scholar
- 2018. Netperf benchmark. https://hewle_packard.github.io/netperf/ [Online; accessed 19-September-2018].Google Scholar
- 2018. Network ACLs. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html. [Online; accessed 26-February-2019].Google Scholar
- 2018. The official benchmark of Redis. https://redis.io/topics/benchmarks [Online; accessed 19-September-2018].Google Scholar
- 2018. Perf Wiki. https://perf.wiki.kernel.org/index.php/Main_Page. [Online; accessed 19-September-2018].Google Scholar
- 2018. The RDMA Verbs Specification. http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf. [Online; accessed 19-September-2018].Google Scholar
- 2018. Redis, an open source (BSD licensed), in-memory data structure store. https://redis.io/ [Online; accessed 19-September-2018].Google Scholar
- 2018. Security Groups for Your VPC. https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html. [Online; accessed 8-December-2018].Google Scholar
- 2018. Single root I/O virtualization. http://pcisig.com/specifications/iov/single_root/ [Online; accessed 8-December-2018].Google Scholar
- Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Savannah, GA, 265-283. Google ScholarDigital Library
- Inho Cho, Keon Jang, and Dongsu Han. 2017. Credit-Scheduled Delay-Bounded Congestion Control for Datacenters. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). ACM, New York, NY, USA, 239-252. Google ScholarDigital Library
- Aleksandar Dragojevic, Dushyanth Narayanan, Orion Hodson, and Miguel Castro. 2014. FaRM: Fast Remote Memory. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI'14). USENIX Association, Berkeley, CA, USA, 401-414. http://dl.acm.org/citation.cfm?id=2616448.2616486 Google ScholarDigital Library
- Adam Dunkels. 2001. Design and Implementation of the lwIP TCP/IP Stack. In Swedish Institute of Computer Science.Google Scholar
- M. Ferdman, P. Lotfi-Kamran, K. Balet, and B. Falsafi. 2011. Cuckoo directory: A scalable directory for many-core systems. In 2011 IEEE 17th International Symposium on High Performance Computer Architecture. 169-180. Google ScholarDigital Library
- Daniel Firestone, Andrew Putnam, Sambhrama Mundkur, Derek Chiou, Alireza Dabagh, Mike Andrewartha, Hari Angepat, Vivek Bhanu, Adrian Caulfield, Eric Chung, et al. 2018. Azure Accelerated Networking: SmartNICs in the Public Cloud. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18), Renton, WA. Google ScholarDigital Library
- D. Goldenberg, M. Kagan, R. Ravid, and M. S. Tsirkin. 2005. Zero copy sockets direct protocol over infiniband-preliminary implementation and performance analysis. In 13th Symposium on High Performance Interconnects (HOTI'05). 128-137. Google ScholarDigital Library
- Stefan Hajnoczi. 2011. QEMU Internals: vhost architecture. http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html. [Online].Google Scholar
- Mark Handley, Costin Raiciu, Alexandru Agache, Andrei Voinescu, Andrew W. Moore, Gianni Antichi, and Marcin Wojcik. 2017. Rearchitecting Datacenter Networks and Stacks for Low Latency and High Performance. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). ACM, New York, NY, USA, 29-42. Google ScholarDigital Library
- SeanHefty. 2012. Rsockets. In 2012 OpenFabris International Workshop, Monterey, CA, USA.Google Scholar
- Jinho Hwang, K K Ramakrishnan, and Timothy Wood. 2015. NetVM: high performance and fiexible networking using virtualization on commodity platforms. IEEE Transactions on Network and Service Management 12, 1 (2015), 34-47.Google ScholarCross Ref
- DPDK Intel. 2018. Data plane development kit.Google Scholar
- Manish Jain, Ravi S Prasad, and Constantinos Dovrolis. 2003. The TCP bandwidth-delay product revisited: network buffering, cross traffic, and socket buffer auto-sizing. Technical Report. Georgia Institute of Technology.Google Scholar
- EunYoung Jeong, Shinae Woo, Muhammad Asim Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. 2014. mTCP: a Highly Scalable User-level TCP Stack for Multicore Systems. In NSDI, Vol. 14. 489-502. Google ScholarDigital Library
- Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2014. Using RDMA Efficiently for Key-value Services. In Proceedings of the 2014 ACM Conference on SIGCOMM (SIGCOMM '14). ACM, New York, NY, USA, 295-306. Google ScholarDigital Library
- Patrick Kelsey. [n. d.]. Libuinet. https://github.com/pkelsey/libuinet [Online].Google Scholar
- M. Mahalingam, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar, M. Bursell, and C. Wright. 2018. Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks. https://www.rfc-editor.org/info/rfc7348 [Online].Google Scholar
- Mellanox. [n. d.]. Mellanox Messaging Accelerator. http://www.mellanox.com/page/so_ware_vma [Online].Google Scholar
- Ben Pfaff, Justin Pettit, Teemu Koponen, Ethan J Jackson, Andy Zhou, Jarno Rajahalme, Jesse Gross, Alex Wang, Joe Stringer, Pravin Shelar, et al. 2015. The Design and Implementation of Open v Switch. In NSDI, Vol. 15. 117-130. Google ScholarDigital Library
- Jonas Pfefferle et al. 2015. A Hybrid I/O Virtualization Framework for RDMA-capable Network Interfaces. In Proceedings of the VEE (VEE '15). 17-30. Google ScholarDigital Library
- Adit Ranadive et al. 2012. Toward a Paravirtual v RDMA Device for VMware ESXi Guests. VMware Technical Journal. Georgia Institute of Technology, VMWare Inc.Google Scholar
- Rusty Russell. 2008. virtio: towards a de-facto standard for virtual I/O devices. ACM SIGOPS Operating Systems Review 42, 5 (2008), 95-103. Google ScholarDigital Library
- Michael Scharf and Sebastian Kiesel. 2006. Head-of-line Blocking in TCP and SCTP: Analysis and Measurements. In GLOBECOM, Vol. 6. 1-5.Google ScholarCross Ref
- P. Shivam, P. Wyckoff, and D. Panda. 2001. EMP: Zero-Copy OS-Bypass NIC-Driven Gigabit Ethernet Message Passing. In SC '01: Proceedings of the 2001 ACM/IEEE Conference on Supercomputing. 49-49. Google ScholarDigital Library
- Jia Song and Jim Alves-Foss. 2012. Performance review of zero copy techniques. International Journal of Computer Science and Security (IJCSS) 6, 4 (2012), 256.Google Scholar
- Wikipedia. 2018. Token bucket. https://en.wikipedia.org/wiki/Token_bucket [Online; accessed 8-December-2018].Google Scholar
- Wikipedia. 2018. Traffic shaping. https://en.wikipedia.org/wiki/Traffic_shaping [Online; accessed 8-December-2018].Google Scholar
- Bairen Yi, Jiacheng Xia, Li Chen, and Kai Chen. 2017. Towards Zero Copy Dataflows Using RDMA. In Proceedings of the SIGCOMM Posters and Demos (SIGCOMM Posters and Demos '17). ACM, New York, NY, USA, 28-30. Google ScholarDigital Library
- Tianlong Yu, Shadi Abdollahian Noghabi, Shachar Raindel, Hongqiang Liu, Jitu Padhye, and Vyas Sekar. 2016. FreeFlow: High Performance Container Networking. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks (HotNets '16). ACM, New York, NY, USA, 43-49. Google ScholarDigital Library
- Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. 2015. Congestion Control for Large-Scale RDMA Deployments. SIGCOMM Comput. Commun. Rev. 45, 4 (Aug. 2015), 523-536. Google ScholarDigital Library
Index Terms
- vSocket: virtual socket interface for RDMA in public clouds
Recommendations
A Hybrid I/O Virtualization Framework for RDMA-capable Network Interfaces
VEE '15: Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution EnvironmentsDMA-capable interconnects, providing ultra-low latency and high bandwidth, are increasingly being used in the context of distributed storage and data processing systems. However, the deployment of such systems in virtualized data centers is currently ...
A Hybrid I/O Virtualization Framework for RDMA-capable Network Interfaces
VEE '15DMA-capable interconnects, providing ultra-low latency and high bandwidth, are increasingly being used in the context of distributed storage and data processing systems. However, the deployment of such systems in virtualized data centers is currently ...
RDMA Based Replication of Multiprocessor Virtual Machines over High-Performance Interconnects
CLUSTER '11: Proceedings of the 2011 IEEE International Conference on Cluster ComputingWith the growing prevalence of cloud computing and the increasing number of CPU cores in modern processors, symmetric multiprocessing (SMP) Virtual Machines (VM), i.e. virtual machines with multiple virtual CPUs, are gaining significance. However, ...
Comments