Using AI/LLMs to enhance your dbt documentation workflow

It's still very much WIP, but combining a couple of open source python packages we can automate documentation generation on dbt including automated ERD generation: Keeping documentation up to date is a common challenge for many teams, especially when it comes to ERDs. While dbt makes documentation more accessible by centralising it in the codebase, both the documentation and ERD creation processes still require manual effort. This is where tools like dbt-osmosis come in. It can automatically generate missing YAML files or populate columns that are missing from existing ones. It also has built-in support for generating documentation content using OpenAI’s API. By combining this with another open-source package for ERD diagram generation, I managed to automate both processes — documentation and ERD creation, and integrate them directly into the CI pipeline. This not only reduces manual work but also ensures that documentation stays accurate and up to date with the latest changes in the dbt project. - ERD generation - documentation using LLMs - dbt