Integrative dynamic reconfiguration in a parallel stream processing engine

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

Documents

  • pdf

    Submitted manuscript, 347 KB, PDF document

Load balancing, operator instance collocations and horizontal scaling are critical issues in Parallel Stream Processing Engines to achieve low data processing latency, optimized cluster utilization and minimized communication cost respectively. In previous work, these issues are typically tackled separately and independently. We argue that these problems are tightly coupled in the sense that they all need to determine the allocations of workloads and migrate computational states at runtime. Optimizing them independently would result in suboptimal solutions. Therefore, in this paper, we investigate how these three issues can be modeled as one integrated optimization problem. In particular, we first consider jobs where workload allocations have little effect on the communication cost, and model the problem of load balance as a Mixed-Integer Linear Program. Afterwards, we present an extended solution called ALBIC, which support general jobs. We implement the proposed techniques on top of Apache Storm, an open-source Parallel Stream Processing Engine. The extensive experimental results over both synthetic and real datasets show that our techniques clearly outperform existing approaches.
Original languageEnglish
Title of host publicationProceedings of the 33rd IEEE International Conference on Data Engineering (ICDE)
Number of pages4
PublisherIEEE Press
Publication date2017
Pages227-230
ISBN (Print)978-1-5090-6544-8
ISBN (Electronic)978-1-5090-6543-1
DOIs
Publication statusPublished - 2017
Externally publishedYes
Event33rd IEEE International Conference on Data Engineering - San Diego, United States
Duration: 19 Apr 201722 Apr 2017
Conference number: 33

Conference

Conference33rd IEEE International Conference on Data Engineering
Nummer33
LandUnited States
BySan Diego
Periode19/04/201722/04/2017

Links

Number of downloads are based on statistics from Google Scholar and www.ku.dk


No data available

ID: 179278061