Data pipelines: the complete overview

Every organisation already has data pipelines. A clerk exporting a spreadsheet from the core system each morning, cleaning it by hand and emailing it to head office is a data pipeline. So is the overnight job that nobody dares to touch because the person who wrote it has left. The question is not whether you have pipelines, but whether they are reliable, understood and fast enough for what the business now asks of them.

A data pipeline is the automated path that moves data from where it is created to where it is useful, transforming it on the way. "Useful" used to mean a report. Today it also means a fraud model scoring transactions as they happen, a chatbot answering from policy documents, or a regulator's return that must reconcile to the cent.

This overview explains the ecosystem: what a pipeline is made of, the two ways data moves, where it lives, the tools we use for each part, and the use cases we build. The rest of the series goes hands-on, one component at a time, on our own OpenShift lab cluster.

What a data pipeline is, and is not

A pipeline has three jobs: move data between systems, transform it into a shape people and programs can use, and guarantee something about the result: that it arrived, that it is complete, that it matches the source.

Three terms get mixed up, so here is how we use them:

• ETL (extract, transform, load) transforms the data before loading it into the target, usually because the target was an expensive warehouse with limited capacity.
• ELT (extract, load, transform) loads the raw data first and transforms it inside the platform. This is the modern default: storage is cheap, the raw copy is kept, and a transformation can be rerun without going back to the source.
• A data platform is the whole environment the pipelines run in: storage, compute, catalog, security and the tools. A pipeline is one flow through it. Our data platforms service covers the platform; these labs cover the flows.

A pipeline is not a single product. Vendors sell "data integration tools", but every real pipeline is a chain of components, and the design decisions sit between them: how often data moves, where the raw copy lives, who checks it, and what happens when a step fails at 3 a.m.

The stages every pipeline has

Stage	What happens	Typical choices
Sources	Where data is born: transactional databases, applications, files from partners, logs, sensors, documents	Oracle, PostgreSQL, SQL Server, SAP, core banking, SWIFT files, CCTV, IoT
Ingest	Copying data out of the source without harming it	Scheduled extracts, change data capture (CDC) from the database log, event streams, file drops
Store	Keeping a raw, unaltered copy, then the refined versions	Object storage with open table formats (the lakehouse), or a warehouse
Transform	Cleaning, typing, deduplicating, joining, aggregating, masking	SQL, Spark, dbt, stream processors
Serve	Putting the result where consumers can reach it quickly	Query engines, warehouse tables, Postgres, Redis or Data Grid caches, APIs
Consume	The reason the pipeline exists	Dashboards, regulatory reports, machine-learning models, AI assistants, other systems

Four things sit around every stage, and they are the difference between a pipeline and a script:

• Orchestration runs steps in the right order, at the right time, retries failures and knows what depends on what.
• Data quality checks every load: row counts match the source, no nulls in key columns, totals reconcile, values are in range.
• Catalog and lineage record what datasets exist, what they mean, who owns them and which pipeline produced them from what.
• Observability tells you when data is late, a job failed or a job suddenly costs three times as much.

The feedback arrow in the diagram matters: when a quality check fails, the fix is almost always upstream, and the affected loads must be rerun. A pipeline that cannot be rerun safely (because it would double-count) is the most common design flaw we find in assessments.

Batch and streaming

Data moves in two rhythms, and most platforms need both.

	Batch	Streaming
Moves data	On a schedule, in chunks	Continuously, event by event
Latency	Minutes to hours	Milliseconds to seconds
Strengths	Complete, reproducible, cheap per record, easy to reconcile	Fast reaction, decoupled systems, every event kept in order
Weaknesses	Stale between runs; a late source delays everything	Harder to reason about, exactly-once needs care, out-of-order events
Suits	Reporting, regulatory returns, model training, month-end	Fraud, alerts, operational dashboards, feeding caches and search
Core tools	Spark, SQL, dbt, Airflow	Kafka, Kafka Connect, Flink, Kafka Streams

Batch is the workhorse. A nightly or hourly job extracts what changed, transforms it, and writes a new version of the tables. It is complete by construction: the job sees the whole day's data and can check the totals. When something goes wrong, you fix it and rerun the day.

Streaming handles what cannot wait. A transaction lands in a Kafka topic the moment it is committed; a stream processor enriches and scores it; a fraud rule fires within a second. The stream is also a clean way to decouple systems: the core system writes once, and any number of consumers read at their own pace.

Two patterns combine them. The lambda architecture runs both a streaming path for speed and a batch path for accuracy, and reconciles them (twice the code). The kappa architecture treats the stream as the one source of truth and replays it for batch work, which is simpler but demands a Kafka cluster that keeps long history. In practice we build a streaming path for the few things that need it and batch for everything else, with both landing in the same storage.

Our Kafka labs cover the streaming side in depth, on the Streams for Apache Kafka cluster in our lab.

Where the data lives: warehouse, lake and lakehouse

	Data warehouse	Data lake	Lakehouse
Storage	The database's own, proprietary	Object storage, files	Object storage, files with an open table format
Data	Structured, modelled up front	Anything, often unmanaged	Anything, with tables where structure is known
Transactions and schema	Yes	No	Yes, from the table format
Engines	One	Any, but each reads raw files	Any, all seeing the same tables
Typical problem	Expensive to scale, slow to change	Becomes a "data swamp" nobody trusts	Newer, needs a catalog and some discipline

The lakehouse is the design most new platforms use, and the one these labs are built around. The data sits in cheap object storage (S3-compatible: Ceph, MinIO, NetApp ONTAP S3, Dell or HPE arrays, or OpenShift Data Foundation). An open table format, most often Apache Iceberg, adds what files alone lack: tables with schemas, ACID transactions, time travel to earlier versions, and the ability to evolve a schema without rewriting everything. Delta Lake and Apache Hudi play the same role; Iceberg has become the common ground that most engines and vendors support.

Inside the lakehouse, data is refined in layers, often called medallion layers:

1. Bronze holds data exactly as it arrived, with the time it arrived. Nothing is ever changed here, so any later mistake can be corrected by reprocessing from Bronze.
2. Silver holds the same data cleaned: correct types, duplicates removed, keys resolved, personal data masked where policy requires.
3. Gold holds business tables: customer, account, transaction, daily balances, shaped for the people and models that use them.

A catalog (an Iceberg REST catalog such as Apache Polaris or Project Nessie, or the one built into the platform) is what makes "any engine" true: Spark, Trino, Flink and Python notebooks all ask the catalog where a table's current version is, and they all see the same answer.

The tools we use

Every part of the ecosystem has a mature open-source project, and most have a supported Red Hat or partner product on OpenShift. This is the stack these labs use.

Need	Open source	On OpenShift and in our lab
Event streaming	Apache Kafka, Kafka Connect	Streams for Apache Kafka (Kafka 4.2 in KRaft mode, the Strimzi operator)
Change data capture	Debezium	Debezium connectors on Kafka Connect
Batch and file ingestion	Airbyte, Apache NiFi, plain COPY and S3 uploads	NiFi or custom containers, run by the orchestrator
Batch processing	Apache Spark	Spark on Kubernetes (the Kubeflow Spark operator), or Spark inside OpenShift AI workbenches
Stream processing	Apache Flink, Kafka Streams	Flink operator; Kafka Streams in Quarkus services
SQL transformations	dbt, Spark SQL	dbt against Trino or Spark
Object storage	MinIO, Ceph	OpenShift Data Foundation (Ceph) or the array's own S3 (NetApp ONTAP, Dell, HPE)
Table format and catalog	Apache Iceberg; Apache Polaris, Project Nessie	Iceberg with a REST catalog; vendor catalogs such as Dell's on their platform
Query engine	Trino, Spark SQL, DuckDB	Trino (the project formerly called PrestoSQL) on OpenShift
Serving databases	PostgreSQL, Redis, ClickHouse	Crunchy Postgres for Kubernetes, Red Hat Data Grid, Redis
Orchestration	Apache Airflow, Dagster, Argo Workflows	Airflow on OpenShift; OpenShift Pipelines (Tekton) and Argo Workflows for container-native jobs
Data quality	Great Expectations, Soda Core	Run as a pipeline step; results into the catalog
Catalog, lineage, governance	OpenMetadata, DataHub, OpenLineage	OpenMetadata on OpenShift, lineage from Airflow and Spark
Dashboards	Apache Superset, Grafana	Grafana with the OpenShift monitoring stack; Superset for business users
AI and machine learning	Kubeflow Pipelines, Feast, Milvus, pgvector	OpenShift AI (data science pipelines, model serving), Milvus or pgvector for vectors
Observability	Prometheus, OpenTelemetry, Tempo	Built into OpenShift; OpenTelemetry and Tempo operators in our lab

Products in bold run on our lab cluster today. Everything else is upstream open source, which you can run yourself or buy supported through the vendor platforms below.

A few practical notes from the lab:

• Kafka with KRaft (no ZooKeeper) is the only mode in Kafka 4. Streams for Apache Kafka 3.x on OpenShift manages brokers, topics, users and connectors as Kubernetes resources, which makes the whole streaming layer part of GitOps.
• Spark on Kubernetes replaces the old Hadoop cluster. Each job gets its own driver and executor pods, scaled for that job, and nothing idles between runs.
• Postgres is the universal serving store. Crunchy's operator gives us high availability, backups to S3 and read replicas in a few lines of YAML, and with pgvector it also serves as the vector store for smaller AI workloads.
• Keep the raw copy in open formats. Parquet files in Iceberg tables on S3 can be read by every tool on this list, now and in ten years. That is the single most important decision for a platform that must outlive its vendors.

Vendor platforms

The same components are sold packaged and supported. Red Hat supplies the platform layer (OpenShift, Streams for Apache Kafka, OpenShift AI, Data Foundation). Dell and HPE bundle storage, compute and a catalog or data layer for AI pipelines; our articles on the Dell AI Data Platform and the HPE unified data layer explain what each adds. Oracle, Snowflake, Databricks and the hyperscalers offer managed lakehouses; for in-country and air-gapped deployments the open-source stack on OpenShift is usually the only option, and it is the one this series builds.

Use cases

The pipeline patterns above are the same everywhere. The sources and the consumers differ by sector.

Banks and financial institutions

• Regulatory reporting: nightly batch from core banking and treasury systems into reconciled Gold tables, from which returns to the central bank are generated and signed off, with lineage back to the source for every figure.
• Fraud and anti-money-laundering: card and transfer events streamed through Kafka, enriched with customer history from a Data Grid cache, scored by a model in under a second, and written back as alerts for the fraud team.
• Customer 360 and credit decisions: CDC from the core, cards, loans and mobile banking systems into one customer table, feeding both dashboards and the risk models.
• Open-banking APIs: a streaming path that keeps an API-serving Postgres in sync with the core, so partner apps never query the core directly.

Government and public sector

• Citizen services: records from registries, tax, land and utilities joined into a single view, with masking and access policies enforced in the Silver layer.
• Revenue analytics: batch pipelines over filings and payments, with quality checks that catch duplicate and missing returns before analysts see them.
• Operational dashboards: streamed status from field systems (vehicles, meters, service counters) into live Grafana or Superset dashboards.

Defence and intelligence

• Sensor and log fusion: radar, camera, network and telemetry feeds streamed into one platform, time-aligned and stored for both live watch floors and later analysis.
• Document pipelines for AI assistants: scanned orders, manuals and reports parsed, chunked and embedded into a vector store, so a doctrine or staff assistant answers from the unit's own documents. Our RAG lab builds this path.
• Maintenance and logistics: vehicle, aircraft and vessel telemetry into predictive-maintenance models and spares planning. The defence use cases area covers these in more depth.

Telecoms, energy and manufacturing

• Call and usage records: billions of small records a day, batched hourly into Iceberg, with streaming for network alarms.
• Metering and IoT: sensor readings into Kafka, aggregated by Flink, stored for years for forecasting.
• Plant data: machine events for quality control and digital twins.

Feeding AI

AI does not replace the data pipeline; it depends on it, and adds three new flows:

1. Document pipelines that turn PDFs, scans, emails and images into text, chunks and embeddings for retrieval.
2. Feature pipelines that compute the inputs models score on (a customer's transactions in the last hour, a vehicle's vibration trend), the same way online and in training.
3. Training-data pipelines that assemble, label, version and audit the datasets used to fine-tune models, so a model's answers can be traced to the data it learned from.

All three read from the same lakehouse the reports use. Our article on the enterprise AI stack places these flows in the wider picture, and our AI data services build them.

What good looks like

From our assessments, the pipelines that stay healthy share a few habits:

• Idempotent loads. Running the same day twice gives the same result, not double the rows. Iceberg's atomic overwrites and merge statements make this easy; append-only scripts make it impossible.
• Raw data kept, untouched. Every reprocessing starts from Bronze, never from the source system again.
• Quality checks as code, next to the pipeline, failing the run rather than emailing a warning.
• Everything in Git: pipeline definitions, SQL, schemas, Kafka topic and connector definitions, deployed through the same GitOps flow as applications.
• Small, named datasets with an owner, rather than one giant job that does everything.
• Security built in: encryption at rest and in transit, masking of personal data in Silver, access by role on the catalog, and an audit trail of who read what. For banks and government this is not optional, and it is much cheaper to build in from the start than to retrofit.

What's next

The rest of this series goes hands-on on our OpenShift lab cluster, one component at a time:

1. Ingestion: batch extracts, file drops and change data capture with Debezium into Kafka.
2. Apache Spark: running Spark jobs on Kubernetes and writing Iceberg tables.
3. Orchestration: Airflow and Argo Workflows, scheduling, retries and dependencies.
4. Data lakehouse: object storage, Iceberg, catalogs and Trino.
5. Data quality and governance: checks, catalog, lineage and masking.
6. Pipelines for AI: document, feature and training-data pipelines on OpenShift AI.

The Kafka labs and Crunchy Postgres labs cover the streaming and serving ends in their own series. If you are deciding whether to build any of this, our data platform assessment is where we usually start with a client.