Databricks — Delta Live Tables, Job Workflows & Orchestration Patterns

Topics we’ll cover: Delta Live Table Pipelines, Auto Loader, DQ checks, CDCs (SCD type 1 & 2); Job Workflows and various data orchestration patterns.

Let’s start today’s discussion with three different scenarios:

• Scenario 1: Self-Service Analytics — Data analysts & BI teams need various KPIs for reporting purposes. Though they are aware of the data product requirements and the logic to implement, they need to wait for data engineers to be onboarded, to create the derived data products and data pipelines.
• Scenario 2: Solving Data Quality Issues — Data teams bring data from various external and internal sources. Often they struggle to find a data quality framework which can be integrated with data pipelines easily.
• Scenario 3: Faster Time-to-Market (TTM) — In a fast evolving business area — data teams often struggle to respond to business by creating new data products and enhancing existing, because of longer development and deployment cycle.

In this blog, we’ll see how we can use Delta Live Table (DLT) to address the above situations.