Zaoxing Liu
ABSTRACT
Network management relies on an up-to-date and accurate view of many traffic metrics for tasks such as traffic engineering (e.g., heavy hitters), anomaly detection (e.g., entropy of source addresses), and security (e.g., DDoS detection). Obtaining an accurate estimate of these metrics while using little router CPU and memory is challenging. This in turn has inspired a large body of work in data streaming devoted to developing optimized algorithms for individual monitoring tasks, as well as recent approaches to make it simpler to implement these algorithms (e.g., OpenSketch). While this body of work has been seminal, we argue that this trajectory of crafting special purpose algorithms is untenable in the long term. We make a case for a "RISC" approach for flow monitoring analogous to a reduced instruction set in computer architecture---a simple and generic monitoring primitive from which a range of metrics can be computed with high accuracy. Building on recent theoretical advances in universal streaming, we show that this "holy grail" for flow monitoring might be well within our reach.
Supplemental Material
- The caida ucsd anonymized internet traces 2015 - sanjose dira. http://www.caida.org/data/passive/passive_2015_dataset.xml.Google Scholar
- Opensketch simulation library. https://github.com/USC-NSL/opensketch.Google Scholar
- N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments. In Proceedings of the Twenty-eighth Annual ACM Symposium on Theory of Computing, STOC '96, pages 20--29, New York, NY, USA, 1996. ACM. Google ScholarDigital Library
- N. Bandi, A. Metwally, D. Agrawal, and A. El Abbadi. Fast data stream algorithms using associative memories. In Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data, SIGMOD '07, pages 247--256, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
- T. Benson, A. Anand, A. Akella, and M. Zhang. Microte: Fine grained traffic engineering for data centers. In Proceedings of the Seventh COnference on Emerging Networking EXperiments and Technologies, CoNEXT '11, pages 8:1--8:12, New York, NY, USA, 2011. ACM. Google ScholarDigital Library
- D. Brauckhoff, B. Tellenbach, A. Wagner, M. May, and A. Lakhina. Impact of packet sampling on anomaly detection metrics. In Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement, IMC '06, pages 159--164, New York, NY, USA, 2006. ACM. Google ScholarDigital Library
- V. Braverman and S. R. Chestnut. Universal Sketches for the Frequency Negative Moments and Other Decreasing Streaming Sums. In N. Garg, K. Jansen, A. Rao, and J. D. P. Rolim, editors, Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM 2015), volume 40 of Leibniz International Proceedings in Informatics (LIPIcs), pages 591--605, Dagstuhl, Germany, 2015. Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik.Google Scholar
- V. Braverman, J. Katzman, C. Seidell, and G. Vorsanger. An optimal algorithm for large frequency moments using o(n(1-2/k)) bits. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, APPROX/RANDOM 2014, September 4--6, 2014, Barcelona, Spain, pages 531--544, 2014.Google Scholar
- V. Braverman and R. Ostrovsky. Zero-one frequency laws. In Proceedings of the Forty-second ACM Symposium on Theory of Computing, STOC '10, pages 281--290, New York, NY, USA, 2010. ACM. Google ScholarDigital Library
- V. Braverman and R. Ostrovsky. Approximating large frequency moments with pick-and-drop sampling. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques - 16th International Workshop, APPROX 2013, and 17th International Workshop, RANDOM 2013, Berkeley, CA, USA, August 21-23, 2013. Proceedings, pages 42--57, 2013.Google Scholar
- V. Braverman and R. Ostrovsky. Generalizing the layering method of indyk and woodruff: Recursive sketches for frequency-based vectors on streams. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, pages 58--70. Springer, 2013.Google Scholar
- V. Braverman, R. Ostrovsky, and A. Roytman. Zero-one laws for sliding windows and universal sketches. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, APPROX/RANDOM 2015, August 24--26, 2015, Princeton, NJ, USA, pages 573--590, 2015.Google Scholar
- A. Chakrabarti, S. Khot, and X. Sun. Near-optimal lower bounds on the multi-party communication complexity of set disjointness. In IEEE Conference on Computational Complexity, pages 107--117. IEEE Computer Society, 2003.Google ScholarCross Ref
- M. Charikar, K. Chen, and M. Farach-Colton. Finding frequent items in data streams. In Automata, Languages and Programming, pages 693--703. Springer, 2002. Google ScholarDigital Library
- B. Claise. Cisco systems netflow services export version 9. RFC 3954.Google Scholar
- G. Cormode, F. Korn, S. Muthukrishnan, and D. Srivastava. Finding hierarchical heavy hitters in data streams. In Proceedings of the 29th international conference on Very large data bases-Volume 29, pages 464--475. VLDB Endowment, 2003. Google ScholarDigital Library
- G. Cormode and S. Muthukrishnan. An improved data stream summary: The count-min sketch and its applications. J. Algorithms, 55(1):58--75, Apr. 2005. Google ScholarDigital Library
- S. Dasgupta and A. Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Struct. Algorithms, 22(1):60--65, Jan. 2003. Google ScholarDigital Library
- M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. SIAM J. Comput., 31(6):1794--1813, June 2002. Google ScholarDigital Library
- R. Dementiev, T. Willhalm, O. Bruggeman, P. Fay, P. Ungerer, A. Ott, P. Lu, J. Harris, P. Kerly, P. Konsor, A. Semin, M. Kanaly, R. Brazones, and R. Shah. Intel performance counter monitor - a better way to measure cpu utilization. https://software.intel.com/en-us/articles/intel-performance-counter-monitor.Google Scholar
- N. Duffield, C. Lund, and M. Thorup. Estimating flow distributions from sampled flow statistics. In Proceedings of the 2003 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, SIGCOMM '03, pages 325--336, New York, NY, USA, 2003. ACM. Google ScholarDigital Library
- C. Estan and G. Varghese. New directions in traffic measurement and accounting. In Proceedings of the 2002 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, SIGCOMM '02, pages 323--336, New York, NY, USA, 2002. ACM. Google ScholarDigital Library
- C. Estan and G. Varghese. New directions in traffic measurement and accounting, volume 32. ACM, 2002.Google Scholar
- C. Estan and G. Varghese. New directions in traffic measurement and accounting. In Proceedings of the 2002 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, SIGCOMM '02, pages 323--336, New York, NY, USA, 2002. ACM. Google ScholarDigital Library
- A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, and F. True. Deriving traffic demands for operational ip networks: Methodology and experience. IEEE/ACM Trans. Netw., 9(3):265--280, June 2001. Google ScholarDigital Library
- P. Indyk, A. McGregor, I. Newman, and K. Onak. Open problems in data streams, property testing, and related topics, 2011. Available at: people. cs. umass. edu/ mcgregor/papers/11-openproblems. pdf, 2011.Google Scholar
- B. Krishnamurthy, S. Sen, Y. Zhang, and Y. Chen. Sketch-based change detection: methods, evaluation, and applications. In Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement, pages 234--247. ACM, 2003. Google ScholarDigital Library
- A. Kumar, M. Sung, J. J. Xu, and J. Wang. Data streaming algorithms for efficient and accurate estimation of flow size distribution. In Proceedings of the Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS '04/Performance '04, pages 177--188, New York, NY, USA, 2004. ACM. Google ScholarDigital Library
- A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang. Data streaming algorithms for estimating entropy of network traffic. In Proceedings of the Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS '06/Performance '06, pages 145--156, New York, NY, USA, 2006. ACM. Google ScholarDigital Library
- A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang. Data streaming algorithms for estimating entropy of network traffic. In ACM SIGMETRICS Performance Evaluation Review, volume 34, pages 145--156. ACM, 2006. Google ScholarDigital Library
- N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner. Openflow: Enabling innovation in campus networks. SIGCOMM Comput. Commun. Rev., 38(2):69--74, Mar. 2008. Google ScholarDigital Library
- A. Ramachandran, S. Seetharaman, N. Feamster, and V. Vazirani. Fast monitoring of traffic subpopulations. In Proceedings of the 8th ACM SIGCOMM Conference on Internet Measurement, IMC '08, pages 257--270, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- R. Schweller, Z. Li, Y. Chen, Y. Gao, A. Gupta, Y. Zhang, P. A. Dinda, M.-Y. Kao, and G. Memik. Reversible sketches: enabling monitoring and analysis over high-speed data streams. IEEE/ACM Transactions on Networking (ToN), 15(5):1059--1072, 2007. Google ScholarDigital Library
- V. Sekar, M. K. Reiter, W. Willinger, H. Zhang, R. R. Kompella, and D. G. Andersen. csamp: A system for network-wide flow monitoring. In NSDI, volume 8, pages 233--246, 2008. Google ScholarDigital Library
- V. Sekar, M. K. Reiter, and H. Zhang. Revisiting the case for a minimalist approach for network flow monitoring. In Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, IMC '10, pages 328--341, New York, NY, USA, 2010. ACM. Google ScholarDigital Library
- Y. Xie, V. Sekar, D. A. Maltz, M. K. Reiter, and H. Zhang. Worm origin identification using random moonwalks. In 2005 IEEE Symposium on Security and Privacy (S&P 2005), 8-11 May 2005, Oakland, CA, USA, pages 242--256. IEEE Computer Society, 2005. Google ScholarDigital Library
- M. Yu, L. Jose, and R. Miao. Software defined traffic measurement with opensketch. In Proceedings of the 10th USENIX Conference on Networked Systems Design and Implementation, nsdi'13, pages 29--42, Berkeley, CA, USA, 2013. USENIX Association. Google ScholarDigital Library
- L. Yuan, C.-N. Chuah, and P. Mohapatra. Progme: towards programmable network measurement. IEEE/ACM Transactions on Networking (TON), 19(1):115--128, 2011. Google ScholarDigital Library
- Y. Zhang. An adaptive flow counting method for anomaly detection in sdn. In Proceedings of the Ninth ACM Conference on Emerging Networking Experiments and Technologies, CoNEXT '13, pages 25--30, New York, NY, USA, 2013. ACM. Google ScholarDigital Library
- Y. Zhang, S. Singh, S. Sen, N. Duffield, and C. Lund. Online identification of hierarchical heavy hitters: algorithms, evaluation, and applications. In Proceedings of the 4th ACM SIGCOMM conference on Internet measurement, pages 101--114. ACM, 2004. Google ScholarDigital Library
- H. C. Zhao, A. Lall, M. Ogihara, O. Spatscheck, J. Wang, and J. Xu. A data streaming algorithm for estimating entropies of od flows. In Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement, IMC '07, pages 279--290, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
Index Terms
- Enabling a "RISC" Approach for Software-Defined Monitoring using Universal Streaming
Recommendations
A microcoded RISC
A new, microcoded, RISC-type system is proposed and presented. The microcode is stored in a 256 x 64 PROM Nanomemory in the CPU. The 8-bit opcode of each instruction is a direct address to the Nanomemory. Each Nanomemory 64-bit word (horizontal ...
A microcoded RISC
A new, microcoded, RISC-type system is proposed and presented. The microcode is stored in a 256 x 64 PROM Nanomemory in the CPU. The 8-bit opcode of each instruction is a direct address to the Nanomemory. Each Nanomemory 64-bit word (horizontal ...
A microcoded RISC
MICRO 19: Proceedings of the 19th annual workshop on MicroprogrammingA new, microcoded, RISC-type system is proposed and presented. The microcode is stored in a 256 x 64 PROM Nanomemory in the CPU. The 8-bit opcode of each instruction is a direct address to the Nanomemory. Each Nanomemory 64-bit word (horizontal ...
Comments