Sameer G. Kulkarni
Mayutan Arumaithurai
Xiaoming Fu
ABSTRACT
Managing Network Function (NF) service chains requires careful system resource management. We propose NFVnice, a user space NF scheduling and service chain management framework to provide fair, efficient and dynamic resource scheduling capabilities on Network Function Virtualization (NFV) platforms. The NFVnice framework monitors load on a service chain at high frequency (1000Hz) and employs backpressure to shed load early in the service chain, thereby preventing wasted work. Borrowing concepts such as rate proportional scheduling from hardware packet schedulers, CPU shares are computed by accounting for heterogeneous packet processing costs of NFs, I/O, and traffic arrival characteristics. By leveraging cgroups, a user space process scheduling abstraction exposed by the operating system, NFVnice is capable of controlling when network functions should be scheduled. NFVnice improves NF performance by complementing the capabilities of the OS scheduler but without requiring changes to the OS's scheduling mechanisms. Our controlled experiments show that NFVnice provides the appropriate rate-cost proportional fair share of CPU to NFs and significantly improves NF performance (throughput and loss) by reducing wasted work across an NF chain, compared to using the default OS scheduler. NFVnice achieves this even for heterogeneous NFs with vastly different computational costs and for heterogeneous workloads.
Supplemental Material
- 2014. Data plane development kit. http://dpdk.org/. (2014). [online].Google Scholar
- 2014. DPDK L-Thread subsystem. http://dpdk.org/doc/guides/sample_app_ug/performance_thread.html. (2014). [online].Google Scholar
- 2016. Performance measurements with RDTSC. https://www.strchr.com/performance_measurements_with_rdtsc. (June 2016). [online].Google Scholar
- 2016. VPP. https://fd.io/. (2016). [online].Google Scholar
- 2017. cgroups-Linux control groups. http://man7.org/linux/man-pages/man7/cgroups.7.html. (2017). [online].Google Scholar
- 2017. Fibers. https://msdn.microsoft.com/en-us/library/ms682661.aspx. (2017). [online].Google Scholar
- Mohammad Alizadeh, Albert Greenberg, David A Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. 2010. Data center tcp (dctcp). In ACM SIGCOMM computer communication review, Vol. 40. ACM, 63--74. Google ScholarDigital Library
- Wei Bai, Li Chen, Kai Chen, and Haitao Wu. 2016. Enabling ECN in Multi-Service Multi-Queue Data Centers. In 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI 16). USENIX Association, Santa Clara, CA, 537--549.Google Scholar
- Rudolf Bayer. 1972. Symmetric binary B-trees: Data structure and maintenance algorithms. Acta informatica 1, 4 (1972), 290--306. Google ScholarDigital Library
- Alan Demers, Srinivasan Keshav, and Scott Shenker. 1989. Analysis and simulation of a fair queueing algorithm. In ACM SIGCOMM Computer Communication Review, Vol. 19. ACM, 1--12. Google ScholarDigital Library
- Jon Dugan, Seth Elliott, Bruce A. Mah, Jeff Poskanzer, and Kaustubh Prabhu. 2014. iPerf - The ultimate speed test tool for TCP, UDP and SCTP. (2014).Google Scholar
- Paul Emmerich, Sebastian Gallenmüller, Daniel Raumer, Florian Wohlfart, and Georg Carle. 2015. MoonGen: a scriptable high-speed packet generator. In Proceedings of the 2015 ACM Conference on Internet Measurement Conference. ACM, 275--287. Google ScholarDigital Library
- ETSI-GS-NFV-002. 2013. Network Functions Virtualization (NFV): Architectural Framework. http://www.etsi.org/deliver/etsi_gs/nfv/001_099/002/01.01.01_60/gs_nfv002v010101p.pdf. (2013). [online].Google Scholar
- Wu-chang Feng, Dilip Kandlur, Debanjan Saha, and Kang Shin. 1999. BLUE: A new class of active queue management algorithms. Ann Arbor 1001 (1999), 48105.Google Scholar
- Sally Floyd and Van Jacobson. 1993. Random Early Detection Gateways for Congestion Avoidance. IEEE/ACM Trans. Netw. 1, 4 (Aug. 1993), 397--413.Google ScholarDigital Library
- Aaron Gember, Anand Krishnamurthy, Saul St John, Robert Grandl, Xiaoyang Gao, Ashok Anand, Theophilus Benson, Aditya Akella, and Vyas Sekar. 2013. Stratos: A network-aware orchestration layer for middleboxes in the cloud. Technical Report.Google Scholar
- Ali Ghodsi, Vyas Sekar, Matei Zaharia, and Ion Stoica. 2012. Multi-resource Fair Queueing for Packet Processing. SIGCOMM Comput. Commun. Rev. 42, 4 (Aug. 2012), 1--12. https://doi.org/10.1145/2377677.2377679 Google ScholarDigital Library
- Pawan Goyal, Harrick M Vin, and Haichen Chen. 1996. Start-time fair queueing: A scheduling algorithm for integrated services packet switching networks. In ACM SIGCOMM Computer Communication Review, Vol. 26. ACM, 157--168.Google ScholarDigital Library
- Leo J Guibas and Robert Sedgewick. 1978. A dichromatic framework for balanced trees. In Foundations of Computer Science, 1978., 19th Annual Symposium on. IEEE, 8--21.Google ScholarDigital Library
- J Halpern and C Pignataro. 2015. RFC 7665: Service Function Chaining (SFC) Architecture. https://tools.ietf.org/html/rfc7665. (2015). [online].Google Scholar
- Sangjin Han, Keon Jang, Aurojit Panda, Shoumik Palkar, Dongsu Han, and Sylvia Ratnasamy. 2015. SoftNIC: A Software NIC to Augment Hardware. Technical Report UCB/EECS-2015-155. EECS Department, University of California, Berkeley. http://www.eecs.berkeley.edu/Pubs/TechRpts/2015/EECS-2015-155.htmlGoogle Scholar
- Keqiang He, Eric Rozner, Kanak Agarwal, Yu Jason Gu, Wes Felter, John Carter, and Aditya Akella. 2016. AC/DC TCP: Virtual congestion control enforcement for datacenter networks. In Proceedings of the 2016 conference on ACM SIGCOMM 2016 Conference. ACM, 244--257. Google ScholarDigital Library
- J. Hwang, K. K. Ramakrishnan, and T. Wood. 2015. NetVM: High Performance and Flexible Networking Using Virtualization on Commodity Platforms. IEEE Transactions on Network and Service Management 12, 1 (March 2015), 34--47. https://doi.org/10.1109/TNSM.2015.2401568 Google ScholarCross Ref
- Raj Jain, Dah-Ming Chiu, and William R Hawe. 1984. A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer System. Vol. 38. Eastern Research Laboratory, Digital Equipment Corporation Hudson, MA.Google Scholar
- Dilip A Joseph, Arsalan Tavakoli, and Ion Stoica. 2008. A policy-aware switching layer for data centers. In ACM SIGCOMM Computer Communication Review, Vol. 38. ACM, 51--62.Google ScholarDigital Library
- Glenn Judd. 2015. Attaining the Promise and Avoiding the Pitfalls of TCP in the Datacenter.. In NSDI. 145--157.Google Scholar
- Tom Kelly, Sally Floyd, and Scott Shenker. 2003. Patterns of congestion collapse. International Computer Science Institute, and University of Cambridge (2003).Google Scholar
- D. Lapsley and S. Low. 1999. Random early marking: an optimisation approach to Internet congestion control. In Networks, 1999. (ICON '99) Proceedings. IEEE International Conference on. 67--74. https://doi.org/10.1109/ICON.1999.796161Google Scholar
- Changhyun Lee, Chunjong Park, Keon Jang, Sue B Moon, and Dongsu Han. 2015. Accurate Latency-based Congestion Feedback for Datacenters.. In USENIX Annual Technical Conference. 403--415.Google Scholar
- Yang Li, Linh Thi Xuan Phan, and Boon Thau Loo. 2016. Network functions virtualization with soft real-time guarantees. In Computer Communications, IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on. IEEE, 1--9. Google ScholarCross Ref
- Jonathan Mace, Peter Bodik, Madanlal Musuvathi, Rodrigo Fonseca, and Krishnan Varadarajan. 2016. 2DFQ: Two-Dimensional Fair Queuing for Multi-Tenant Cloud Services. In Proceedings of the 2016 ACM SIGCOMM Conference (SIGCOMM '16). ACM, New York, NY, USA, 144--159. https://doi.org/10.1145/2934872.2934878 Google ScholarDigital Library
- Joao Martins, Mohamed Ahmed, Costin Raiciu, Vladimir Olteanu, Michio Honda, Roberto Bifulco, and Felipe Huici. 2014. ClickOS and the Art of Network Function Virtualization. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). USENIX Association, Seattle, WA, 459--473. https://www.usenix.org/conference/nsdi14/technical-sessions/presentation/martinsGoogle ScholarDigital Library
- Paul Menage. 2017. Linux Kernel Documentation: CGROUPS. https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt. (2017). [online].Google Scholar
- Dirk Merkel. 2014. Docker: Lightweight Linux Containers for Consistent Development and Deployment. Linux J. 2014, 239, Article 2 (March 2014).Google ScholarDigital Library
- Radhika Mittal, Nandita Dukkipati, Emily Blem, Hassan Wassel, Monia Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, David Zats, et al. 2015. TIMELY: RTT-based Congestion Control for the Datacenter. In ACM SIGCOMM Computer Communication Review, Vol. 45. ACM, 537--550.Google ScholarDigital Library
- Jeffrey C Mogul and KK Ramakrishnan. 1997. Eliminating receive livelock in an interrupt-driven kernel. ACM Transactions on Computer Systems 15, 3 (1997), 217--252.Google ScholarDigital Library
- Ingo Molnar. 2017. Linux Kernel Documentation: CFS Scheduler Design. https://www.kernel.org/doc/Documentation/scheduler/sched-design-CFS.txt. (2017).Google Scholar
- Robert Olsson. 2005. Pktgen the linux packet generator. In Proceedings of the Linux Symposium, Ottawa, Canada, Vol. 2. 11--24.Google Scholar
- Shoumik Palkar, Chang Lan, Sangjin Han, Keon Jang, Aurojit Panda, Sylvia Ratnasamy, Luigi Rizzo, and Scott Shenker. 2015. E2: A Framework for NFV Applications. In Proceedings of the 25th Symposium on Operating Systems Principles (SOSP '15). ACM, New York, NY, USA, 121--136. https://doi.org/10.1145/2815400.2815423 Google ScholarDigital Library
- Abhay K Parekh and Robert G Gallagher. 1994. A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks: The Multiple Node Case. IEEE/ACM Transactions on Networking (ToN) 2, 2 (1994), 137--150.Google ScholarDigital Library
- Shriram Rajagopalan, Dan Williams, Hani Jamjoom, and Andrew Warfield. Split/Merge: System Support for Elastic Execution in Virtual Middleboxes. In Presented as part of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13). USENIX, 227--240.Google ScholarDigital Library
- K.K. Ramakrishnan, S. Floyd, and D. Black. 2001. RFC 3168: The Addition of Explicit Congestion Notification (ECN) to IP. https://tools.ietf.org/html/rfc3168. (2001). [online].Google Scholar
- Luigi Rizzo. 2012. netmap: A Novel Framework for Fast Packet I/O. In USENIX Annual Technical Conference. USENIX, Berkeley, CA, 101--112. https://www.usenix.org/conference/usenixfederatedconferencesweek/netmap-novel-framework-fast-packet-ioGoogle Scholar
- Luigi Rizzo, Stefano Garzarella, Giuseppe Lettieri, and Vincenzo Maffione. 2016. A Study of Speed Mismatches Between Communicating Virtual Machines. In Proceedings of the 2016 Symposium on Architectures for Networking and Communications Systems (ANCS '16). ACM, New York, NY, USA, 61--67. https://doi.org/10.1145/2881025.2881037 Google ScholarDigital Library
- Luigi Rizzo, Paolo Valente, Giuseppe Lettieri, and Vincenzo Maffione. 2016. PSPAT: software packet scheduling at hardware speed. http://info.iet.unipi.it/~luigi/papers/20160921-pspat.pdf. (2016). [online].Google Scholar
- Vyas Sekar, Norbert Egi, Sylvia Ratnasamy, Michael K. Reiter, and Guangyu Shi. 2012. Design and Implementation of a Consolidated Middlebox Architecture. In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation (NSDI'12). USENIX Association, Berkeley, CA, USA, 24--24.Google ScholarDigital Library
- Madhavapeddi Shreedhar and George Varghese. 1996. Efficient fair queuing using deficit round-robin. IEEE/ACM Transactions on networking 4, 3 (1996), 375--385. Google ScholarDigital Library
- Anirudh Sivaraman, Suvinay Subramanian, Mohammad Alizadeh, Sharad Chole, Shang-Tse Chuang, Anurag Agrawal, Hari Balakrishnan, Tom Edsall, Sachin Katti, and Nick McKeown. 2016. Programmable Packet Scheduling at Line Rate. In Proceedings of the 2016 conference on ACM SIGCOMM 2016 Conference. ACM, 44--57. Google ScholarDigital Library
- Dimitrios Stiliadis and Anujan Varma. 1998. Efficient fair queueing algorithms for packet-switched networks. IEEE/ACM Transactions on Networking (ToN) 6, 2 (1998), 175--185.Google ScholarDigital Library
- Dimitrios Stiliadis and Anujan Varma. 1998. Rate-proportional servers: a design methodology for fair queueing algorithms. IEEE/ACM Transactions on networking 6, 2 (1998), 164--174. Google ScholarDigital Library
- Ion Stoica, Scott Shenker, and Hui Zhang. 2003. Core-stateless Fair Queueing: A Scalable Architecture to Approximate Fair Bandwidth Allocations in High-speed Networks. IEEE/ACM Trans. Netw. 11, 1 (Feb. 2003), 33--46. https://doi.org/10.1109/TNET.2002.808414 Google ScholarDigital Library
- Rahul Upadhyaya, CB Anantha Padmanabhan, Meenakshi Sundaram Lakshmanan, and Satya Routray. 2016. Optimising NFV Service Chains on OpenStack Using Docker. https://www.openstack.org/videos/video/optimising-nfv-service-chains-on-openstack-using-docker. (April 2016).Google Scholar
- Lixia Zhang. 1991. VirtualClock: a new traffic control algorithm for packet-switched networks. ACM Transactions on Computer Systems (TOCS) 9, 2 (1991), 101--124. Google ScholarDigital Library
- Wei Zhang, Guyue Liu, Wenhui Zhang, Neel Shah, Phillip Lopreiato, Gregoire Todeschi, K.K. Ramakrishnan, and Timothy Wood. 2016. OpenNetVM: A Platform for High Performance Network Service Chains. In Proceedings of the 2016 Workshop on Hot Topics in Middleboxes and Network Function Virtualization (Hot-MIddlebox '16). ACM, New York, NY, USA, 26--31. https://doi.org/2940147.2940155Google ScholarDigital Library
Index Terms
- NFVnice: Dynamic Backpressure and Scheduling for NFV Service Chains
Recommendations
Improving Resource Isolation of Critical Tasks in a Workload
Job Scheduling Strategies for Parallel ProcessingAbstractTypical cluster schedulers co-locate critical tasks and background batch tasks to improve the utilization of resources in the cluster. However, this leads to resource contention and interference between the diverse co-located tasks. To ensure ...
A Novel NFV Schedule Optimization Approach with Sensitivity to Packets Dropping Positions
TOPIC '18: Proceedings of the 2018 Workshop on Theory and Practice for Integrated Cloud, Fog and Edge Computing ParadigmsNetwork Function Virtualization (NFV) is a promising approach for providing efficient and flexible network functions deployment, which also reduces the cost in the network operations compared to traditional method using dedicated hardware. Multiple ...
NFVnice: Dynamic Backpressure and Scheduling for NFV Service Chains
Managing Network Function (NF) service chains requires careful system resource management. We propose <italic>NFVnice</italic>, a user space NF scheduling and service chain management framework to provide fair, efficient and dynamic resource scheduling ...
Comments