This blog post provides an in-depth analysis of GCP’s data processing pipeline services, including Cloud Dataflow, Cloud Data Fusion, and Cloud Composer. Through sample architectures and feature comparisons, we’ll explore the optimal use cases for each service.

Prosenjit Chakraborty

·

Follow

5 min read

·

Mar 4, 2023

--

Share

Cloud Dataflow

Properties

• Server-less Batch & Stream processing.
• Distributed processing backend of Apache Beam.
• Based on Dataflow templates to package a Dataflow pipeline for deployment.
• Google provided open source pre-built Dataflow templates or custom templates.
• Google provided templates — 3 categories: Streaming templates, Batch templates and Utility templates.
• Streaming analytics — ensures exactly-once processing of events.
• Dataflow components are — Jobs, Pipelines, Workbench, Snapshots, SQL workspace.

1. Pipelines: Encapsulates the entire series of computations involved. This is created based on existing Google Dataflow parameterized templates (i.e. pre-built pipelines) or custom…