Preface
Many fast growing organizations and autonomous teams choose Polars for their data workflows. One key reason is that Polars easily scales with them without having to refactor or face a steep increase in compute costs.
Joulen had the opportunity to start from scratch and in this case study we can read how they set up their scalable data infrastructure with Polars. In addition, the case study shows how organizations can leverage Polars to enable everyone on the team to work on both small- and large-scale analysis interchangeably.
Driving the change to renewable energy
In an era where renewable energy and sustainable practices are becoming vital, and as businesses across the globe strive to achieve their net-zero goals, the importance of effective energy management cannot be overstated. In the rapidly evolving energy industry, startups like Joulen play a crucial role in facilitating the transition to renewable sources by providing sophisticated tools for energy management and trading.
The team at Joulen has developed the PARIS platform to solve this problem. PARIS is an AI-driven software solution designed to optimize renewable energy assets. It combines advanced forecasting, intelligent optimization, automated trading, and intuitive control interfaces, to maximize the value and efficiency of renewable technologies. The platform empowers businesses to make data-driven decisions and accelerate their transition to sustainable energy practices.
Building scalable data workflows
Energy management and trading is a complicated business. The key advantage of Polars is that it allows the data science team at Joulen to focus on optimizing our clients’ energy flows as Polars takes care of optimizing our data flows.
Liam Brannigan, Data Team Lead @ Joulen
The development team at Joulen requires a data analytics stack that is cost-effective and highly scalable. The three pillars of this approach at Joulen are: data storage in Parquet on S3, serverless computation with elastic containers, and data analysis with Polars. This approach is cost effective, not only from low cost storage on S3, but also because Polars uses query optimisations to limit the amount of data transferred. The approach is highly scalable as Polars uses all of the available computational power of the serverless containers. As the company grows, the team scales the pipelines by simply adjusting the computing resources of these containers.
A further advantage of Polars is that it allows the Joulen team to organize around a single data analytics technology. On a given day the team may need to do detailed analysis for a single client as well as run large-scale pipelines across all clients. As Polars works just as well on a laptop as on a large cloud instance the team can use the same code base for both tasks. Crucially, this also means that all members of the development team have the skills that allow them to work on both small- and large-scale analysis interchangeably.
The Joulen team uses Polars to extract analytics data from Postgres and KDB databases. This analytics data is stored in a lakehouse on S3 with the delta-rs library. The delta-rs library is an open source implementation of the DeltaLake protocol. The team chose delta-rs for its speed and tight integrations with Polars that allows the Polars query optimizer to optimize queries against the lakehouse on S3. The high performance of Polars and the lakehouse gives the team a real-time view of performance across the assets.
Analyzing streams of data from energy assets
The core use case of Polars for Joulen is time series analysis. The team analyzes streams of data from solar panels, wind turbines, and batteries, to prepare the inputs for time series forecasting models and the battery management system. The team chose Polars for its fundamental speed and full set of features for time series analysis. Using machine learning libraries, such as XGBoost, Nixtla and Polars_OLS, the training and inference pipelines are fully compatible with Polars. In fact, the team even uses Polars plugins that allow linear models to be fit within a Polars expression!
Visualization is a key part of the analytics process at Joulen. The team built interactive Streamlit apps to generate insights into the performance of energy generation and storage assets. These apps are powered by Polars uder the hood, as its processing speed gives a much better low latency experience for end users.