At Xriba we need to analyze and transform a lot of accountability data, to improve our machine learning models and offer our customers a fast and reliable overview of their company KPIs.

Spark is one of the best technologies for high volume data processing that optimizes time and resource utilization, but managing a spark cluster is not an easy task.

We will describe our journey from Dataproc, the Google Spark offering, to a self-managed deployment on Kubernetes that helped us to keep costs under control and deployment strategies on par with the rest of our company software.