Azure DevOps Pipeline Setup for Azure Data Factory (v2)

Azure Data Factory (v2) is a very popular Azure managed service and being used heavily from simple to complex ETL (extract-transform-load), ELT (extract-load-transform) & data integration scenarios.

On the other hand, Azure DevOps has become a robust tool-set for collaboration & building CI-CD pipelines.

In this blog, we’ll see how we can implement a DevOps pipeline with ADFv2.

Data factories vs Data factory (v2)

Data factories

• Contains a set of Data factory (v1 and/or v2) instances.

Data factory (v2)

• Contains a set of pipelines, datasets, linked services, triggers, integration runtimes.
• Each instance is associated to a resource group. Multiple instances can be associated to a single resource group.
• Each instance is associated with a single location (e.g. East US, UK South, Central India etc.)
• Instance level access control; users having access to a Data factory instance, can access all of the pipelines inside it.
• Associated with a single Git/Azure Repo.