Supercharged Data Platform Development with Lakeflow Declarative Pipelines

An e-mobility company wanted hourly insights for its operations team and its customers, live within two months. The starting point was a set of empty Databricks workspaces and event ingestion into the cloud environment. In Databricks, events arrive as a stream and are enriched with financial data and asset information from API’s. The result feeds operational dashboards for internal use, customer-facing dashboards and an SFTP export flow used for billing. This talk is about building that end to end with Lakeflow Declarative Pipelines: streaming the events in, enriching them with the API data, and keeping the dashboards fresh every hour. Most of the time goes to what we learned along the way: - When a streaming table beats a materialized view, and when it does not. - Combining streaming and batch ingestion in one pipeline with AUTO CDC flows  - Robust ingestion from fragile, poorly documented APIs: VARIANT, conditional pipeline tasks and custom data sources. - One pipeline with many tables versus many small pipelines, and where the driver starts to hurt. - How to keep business logic in the hands of business users, so changing a query does not need an engineer.