Advanced search
1 file | 1.54 MB Add to list

Influencing factors in the scalability of distributed stream processing jobs

(2021) IEEE ACCESS. 9. p.109413-109431
Author
Organization
Abstract
More and more use cases require fast, accurate, and reliable processing of large volumes of data. To do this, a distributed stream processing framework is needed which can distribute the load over several machines. In this work, we study and benchmark the scalability of stream processing jobs in four popular frameworks: Flink, Kafka Streams, Spark Streaming, and Structured Streaming. Besides that, we determine the factors that influence the performance and efficiency of scaling processing jobs with distinct characteristics. We evaluate horizontal, as well as vertical scalability. Our results show how the scaling efficiency is impacted by many factors including the initial cluster layout and direction of scaling, the pipeline design, the framework design, resource allocation, and data characteristics. Finally, we give some recommendations on how practitioners should undertake to scale their clusters.
Keywords
BIG DATA, MANAGEMENT, MODEL, Scalability, Throughput, Pipelines, Benchmark testing, Sparks, Measurement, Storms, Apache spark, structured streaming, apache flink, apache kafka, kafka streams, distributed computing, stream processing, frameworks, scalability, benchmarking, big data

Downloads

  • published article.pdf
    • full text (Published version)
    • |
    • open access
    • |
    • PDF
    • |
    • 1.54 MB

Citation

Please use this url to cite or link to this publication:

MLA
van Dongen, Giselle, and Dirk Van den Poel. “Influencing Factors in the Scalability of Distributed Stream Processing Jobs.” IEEE ACCESS, vol. 9, 2021, pp. 109413–31, doi:10.1109/ACCESS.2021.3102645.
APA
van Dongen, G., & Van den Poel, D. (2021). Influencing factors in the scalability of distributed stream processing jobs. IEEE ACCESS, 9, 109413–109431. https://doi.org/10.1109/ACCESS.2021.3102645
Chicago author-date
Dongen, Giselle van, and Dirk Van den Poel. 2021. “Influencing Factors in the Scalability of Distributed Stream Processing Jobs.” IEEE ACCESS 9: 109413–31. https://doi.org/10.1109/ACCESS.2021.3102645.
Chicago author-date (all authors)
van Dongen, Giselle, and Dirk Van den Poel. 2021. “Influencing Factors in the Scalability of Distributed Stream Processing Jobs.” IEEE ACCESS 9: 109413–109431. doi:10.1109/ACCESS.2021.3102645.
Vancouver
1.
van Dongen G, Van den Poel D. Influencing factors in the scalability of distributed stream processing jobs. IEEE ACCESS. 2021;9:109413–31.
IEEE
[1]
G. van Dongen and D. Van den Poel, “Influencing factors in the scalability of distributed stream processing jobs,” IEEE ACCESS, vol. 9, pp. 109413–109431, 2021.
@article{8724511,
  abstract     = {{More and more use cases require fast, accurate, and reliable processing of large volumes of data. To do this, a distributed stream processing framework is needed which can distribute the load over several machines. In this work, we study and benchmark the scalability of stream processing jobs in four popular frameworks: Flink, Kafka Streams, Spark Streaming, and Structured Streaming. Besides that, we determine the factors that influence the performance and efficiency of scaling processing jobs with distinct characteristics. We evaluate horizontal, as well as vertical scalability. Our results show how the scaling efficiency is impacted by many factors including the initial cluster layout and direction of scaling, the pipeline design, the framework design, resource allocation, and data characteristics. Finally, we give some recommendations on how practitioners should undertake to scale their clusters.}},
  author       = {{van Dongen, Giselle and Van den Poel, Dirk}},
  issn         = {{2169-3536}},
  journal      = {{IEEE ACCESS}},
  keywords     = {{BIG DATA,MANAGEMENT,MODEL,Scalability,Throughput,Pipelines,Benchmark testing,Sparks,Measurement,Storms,Apache spark,structured streaming,apache flink,apache kafka,kafka streams,distributed computing,stream processing,frameworks,scalability,benchmarking,big data}},
  language     = {{eng}},
  pages        = {{109413--109431}},
  title        = {{Influencing factors in the scalability of distributed stream processing jobs}},
  url          = {{http://doi.org/10.1109/ACCESS.2021.3102645}},
  volume       = {{9}},
  year         = {{2021}},
}

Altmetric
View in Altmetric
Web of Science
Times cited: