Thomas Heinze
Zbigniew Jerzak
ABSTRACT
Typical use cases like financial trading or monitoring of manufacturing equipment pose huge challenges regarding end to end latency as well as throughput towards existing data stream processing systems. Established solutions like Apache S4 or Storm need to scale out to a large set of hosts to meet these challenges. An ideal system can react to workload changes by on demand acquisition or release of hosts. Thereby, it can handle unexpected peak loads as well as improve the average utilization of the system. This property is called elasticity.
The major challenge for an elastic scaling system is to find the right point in time to scale in or out. To determine this right point is difficult, because it depends on constantly changing system and workload characteristics. In this demonstration, we apply three alternative auto-scaling techniques known from other domains on top of an existing elastic data stream processing system. A user of the demonstration can experience the influence of the chosen auto-scaling technique on the latency and the system utilization using a real-world use case based on different workloads from the Frankfurt stock exchange.
- E. G. Coffman Jr, M. R. Garey, and D. S. Johnson. Approximation algorithms for bin packing: A survey. In Approximation algorithms for NP-hard problems, pages 46--93, 1996. Google Scholar
Digital Library
- R. Das, G. Tesauro, and W. E. Walsh. Model-based and model-free approaches to autonomic resource allocation. IBM Research Report, 2005.Google Scholar
- R. C. Fernandez, M. Migliavacca, E. Kalyvianaki, and P. Pietzuch. Integrating scale out and fault tolerance in stream processing using operator state management. In SIGMOD, pages 725--736, 2013. Google ScholarDigital Library
- V. Gulisano, R. Jimenez-Peris, M. Patino-Martinez, C. Soriente, and P. Valduriez. Streamcloud: An elastic and scalable data streaming system. IEEE TPDS, pages 2351--2365, 2012. Google ScholarDigital Library
- T. Heinze, V. Pappalardo, Z. Jerzak, and C. Fetzer. Auto-scaling techniques for elastic data stream processing. In ICDEW, 2014.Google Scholar
- T. Lorido-Botrán, J. Miguel-Alonso, and J. A. Lozano. Auto-scaling techniques for elastic applications in cloud environments. Department of Computer Architecture and Technology, University of Basque Country, Tech. Rep. EHU-KAT-IK-09-12, 2012.Google Scholar
Index Terms
- Auto-scaling techniques for elastic data stream processing
Recommendations
Latency-aware elastic scaling for distributed data stream processing systems
DEBS '14: Proceedings of the 8th ACM International Conference on Distributed Event-Based SystemsElastic scaling allows a data stream processing system to react to a dynamically changing query or event workload by automatically scaling in or out. Thereby, both unpredictable load peaks as well as underload situations can be handled. However, each ...
Cloud-based data stream processing
DEBS '14: Proceedings of the 8th ACM International Conference on Distributed Event-Based SystemsIn this tutorial we present the results of recent research about the cloud enablement of data streaming systems. We illustrate, based on both industrial as well as academic prototypes, new emerging uses cases and research trends. Specifically, we focus ...
QEScalor: Quantitative Elastic Scaling Framework in Distributed Streaming Processing
Computational Science – ICCS 2020AbstractRecently, researchers usually use the elastic scaling techniques as a powerful means of the distributed stream processing systems to deal with the high-speed data stream which arrives continuously and fluctuates constantly. The existing methods ...
Comments