Workflow: Workflow involves sequencing and dependency management of processes. Processing: There are two data ingestion models: batch processing, in which source data is collected periodically and sent to the destination system, and stream processing, in which data is sourced, manipulated, and loaded as soon as it’s created The ultimate goal is to make it possible to analyze the data.
Transformation: Transformation refers to operations that change data, which may include data standardization, sorting, deduplication, validation, and verification. Also, the data may be synchronized in real time or at scheduled intervals.ĭestination: A destination may be a data store - such as an on-premises or cloud-based data warehouse, a data lake, or a data mart - or it may be a BI or analytics application. Most pipelines ingest raw data from multiple sources via a push mechanism, an API call, a replication engine that pulls data at regular intervals, or a webhook. Source: Data sources may include relational databases and data from SaaS applications. A data pipeline may be a simple process of data extraction and loading, or, it may be designed to handle data in a more advanced manner, such as training datasets for machine learning.
What happens to the data along the way depends upon the business use case and the destination itself. To understand how a data pipeline works, think of any pipe that receives something from a source and carries it to a destination. A pipeline also may include filtering and features that provide resiliency against failure.
It seems as if every business these days is seeking ways to integrate data from multiple sources to gain business insights for competitive advantage.Ī data pipeline is a set of actions that ingest raw data from disparate sources and move the data to a destination for storage and analysis. What is a Data Pipeline? Process and Examples