How to Ensure that Your ETL Pipelines Deliver High Quality Data | DataEngConf BCN ’18
Complex ETL demands accuracy when data is mission critical. But data pipelines are changing often. Regressions are possible from minor changes to DAGs and tasks. In this talk, we’ll cover examples of tests that can be run against your tables and data models. You will learn about the different classes of tests, how to set them up, and the important metrics to monitor. You’ll be the enabler of accurate business decisions, with confidence in your data quality and no more guesswork or surprises about data quality.