From One Contract to Five Hundred: Keeping Data Contracts Consistent at Scale

Most data teams have lived through the same failure: a producer renames a column or changes a definition, and a dashboard, model or downstream pipeline silently breaks days later. Data contracts are the attempt to fix that. A data contract is a versioned, machine-readable document, owned by the producer, that describes a logical data model explicitly: its schema and types, the meaning of each field, quality expectations such as freshness and uniqueness, service levels, ownership and terms of use. But being valid against a standard like the Data Contract Specification (DCS) or the Open Data Contract Standard (ODCS) only means a contract is well-formed, it says nothing about whether it follows your organisation's rules: naming conventions, mandatory descriptions, approved storage platforms, a semver policy. As the number of contracts grows from a handful to hundreds, those conventions drift tend to drift and reviewing every contract by hand is not scalable. This talk introduces what data contracts are, our experience with scaling data contracts and the challenges that come with it. We will discuss our solution, contract-checker to some of these challenges. contract-checker is an open source Python CLI developed by us that lets you codify those conventions once, in a YAML file, and enforce them across every contract in your repository. We will also look at how the contract-checker supports DCS and ODCS out of the box, how to introduce it incrementally without breaking existing contracts, and how to extend it with your own check types.