■ On the fly load balancing to address hot topics in topic-based pub/sub systems

Abstract.

Distributed topic-based publish/subscribe systems like Apache Kafka provide a scalable and decentralized approach to achieve data dissemination. However, despite their wide adoption they can suffer from performance degradation due to the uneven load distribution between the nodes that receive and forward the messages (i.e., brokers). This problem occurs due to the lack of effective load balancing mechanisms that consider the impact of (i) the amount of topics that are handled by a specific broker and (ii) changes in the input rate during the course of the system execution. Furthermore, while there have been some previous works that examine the problem, most of them focus on content-based pub/sub systems or require a centralized coordinator for determining the appropriate assignments. In this work we propose a novel decentralized load balancing technique for topic-based publish/subscribe systems. More specifically, we exploit the fact that brokers in systems like Kafka can communicate using inner topics to exchange their load-related information and propose a novel decentralized algorithm that executes on each individual broker to determine the topics' partitions that should be migrated in order to avoid overloaded conditions. Our detailed experimental evaluation on our local cluster, using different applications that process various data forms from different topics, illustrate the benefits of our approach and show that we can efficiently balance the load between the brokers without the need of a centralized coordination mechanism.

Bibtex Entry.

@inproceedings{dedousis2018fly,
  title={On the fly load balancing to address hot topics in topic-based pub/sub systems},
  author={Dedousis, Dimitris and Zacheilas, Nikos and Kalogeraki, Vana},
  booktitle={2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)},
  pages={76--86},
  year={2018},
  organization={IEEE}
}