Karasu: A Collaborative Approach to Efficient Cluster Configuration for Big Data Analytics

no code implementations • 22 Aug 2023 • Dominik Scheinert, Philipp Wiesner, Thorsten Wittkopp, Lauritz Thamsen, Jonathan Will, Odej Kao

However, big data analytics jobs across users can share many common properties: they often operate on similar infrastructure, using similar algorithms implemented in similar frameworks.

Perona: Robust Infrastructure Fingerprinting for Resource-Efficient Big Data Analytics

no code implementations • 15 Nov 2022 • Dominik Scheinert, Soeren Becker, Jonathan Bader, Lauritz Thamsen, Jonathan Will, Odej Kao

Choosing a good resource configuration for big data analytics applications can be challenging, especially in cloud environments.

Benchmarking

On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds

no code implementations • 16 Nov 2021 • Dominik Scheinert, Alireza Alamgiralem, Jonathan Bader, Jonathan Will, Thorsten Wittkopp, Lauritz Thamsen

With the growing amount of data, data processing workloads and the management of their resource usage becomes increasingly important.

Management

Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation

1 code implementation • 27 Aug 2021 • Dominik Scheinert, Houkun Zhu, Lauritz Thamsen, Morgan K. Geldenhuys, Jonathan Will, Alexander Acker, Odej Kao

Distributed dataflow systems like Spark and Flink enable the use of clusters for scalable data analytics.

Descriptive

Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts

1 code implementation • 29 Jul 2021 • Dominik Scheinert, Lauritz Thamsen, Houkun Zhu, Jonathan Will, Alexander Acker, Thorsten Wittkopp, Odej Kao

First, a general model is trained on all the available data for a specific scalable analytics algorithm, hereby incorporating data from different contexts.

Descriptive