Processing Millions of Smart Meter Readings Daily: How ANWB Energy Built a Scalable Data Pipeline with dbt and Snowflake

Smart meters provide detailed household energy data, which is particularly useful when combined with a dynamic pricing model such as that provided by ANWB Energy. Lower prices at night when demand is low and higher prices during peak evening hours encourage efficient energy use and grid stability. While energy data is straightforward for an individual household, imagine the complexities of 15 minute intervals from 200,000 customers. For energy suppliers, having accurate and up-to-date data is crucial. They must purchase energy from generators in advance, and any mismatch between predicted and actual usage requires expensive spot market transactions. To handle this massive data challenge, we implemented a modern data stack using dbt, GitLab CI/CD pipelines, and Snowflake. Dbt models and transforms the data using software engineering best practises such as automated testing. Incremental models process millions of smart meter readings daily by only updating changed or new data rather than recomputing entire datasets. This approach dramatically reduces computational resources and costs while ensuring that forecasting models have access to accurate and up-to-date consumption patterns.