Do classic data pipelines still matter in the AI era?

Every AI pitch arrives with a new vocabulary: vector databases, embeddings, agents, retrieval. It is reasonable to ask whether the old vocabulary, the nightly batch, the extract, the reconciliation job, the regulatory return, still matters. It does, for three reasons, and the AI programmes that succeed are the ones that take it seriously.

1. They still run the business

At a bank, data flows through dozens of pipelines that have nothing to do with AI: the core banking end-of-day batch, returns to Bangladesh Bank, reconciliation with card and payment networks, finance and risk reporting, and the dashboards management reads in the morning. A ministry has the same pattern with files, payments and case records; a force with logistics, maintenance and personnel. None of this goes away because a chatbot arrived. If anything, the chatbot adds consumers to the same data.

2. AI depends on them

A model is only as good as the data it is fed. Gartner predicts that through 2026, organisations will abandon 60% of AI projects that are not supported by AI-ready data, and its survey found that 63% of organisations either lack the right data management practices for AI or are not sure whether they have them.

"AI-ready" turns out to mean the classic disciplines: data that is clean, deduplicated, current, labelled with who may see it, and traceable to its source. A retrieval system built on stale circulars gives confident wrong answers. A fraud model trained on a table nobody reconciles learns the reconciliation errors. The engineering that makes a regulatory return correct is the engineering that makes an AI answer correct.

3. The market is paying for them

The most telling data deals of the last eighteen months were for classic pipeline companies, bought by vendors whose pitch is AI:

Deal	What changed hands	Status
Salesforce and Informatica	Data integration, quality, catalogue and master data management	Completed November 2025, about $8 billion
IBM and Confluent	The commercial company behind Apache Kafka, for streaming data	Completed March 2026, about $11 billion
Fivetran and dbt Labs	Data loading and SQL transformation	Merger completed June 2026
Databricks and Neon; Snowflake and Crunchy Data	Managed PostgreSQL	Announced May and June 2025

IBM's own words at the Confluent close describe the combination as a "smart data platform" for AI models and agents; it is a streaming company. The buyers know where the value is.

One practical note on the last row: Snowflake's purchase of Crunchy Data has not slowed the open-source Crunchy Postgres Operator we run on OpenShift. It is still developed under the Apache 2.0 licence, with version 6.0.2 released in June 2026 and commits continuing into October.

What has changed: the pipeline's shape, not its necessity

Four shifts are real, and they change how a data platform is designed:

• Documents join the pipeline. PDFs, scans, images and audio are now parsed, chunked, embedded and indexed alongside the tables, and kept fresh the same way.
• Real time matters more. Agents and fraud models need current data, so change data capture and streaming, typically Kafka with Debezium, replace the nightly batch for the feeds where a decision cannot wait. Batch stays where it is enough.
• The lakehouse is winning. Open table formats such as Apache Iceberg let one copy of the data serve SQL reports, Spark jobs and AI, instead of a warehouse and a lake kept in step by hand.
• AI helps build pipelines. It writes SQL, maps fields, documents lineage and flags anomalies. Engineers still design and own the result.

And one shift in responsibility: governance has to reach the answer. Access rules and lineage must follow the data into the vector index, or an assistant will quote a document its user was never allowed to open.

One pipeline, two outputs

The design we use treats reports and AI as two outputs of the same pipeline:

1. Sources: core systems, databases, logs and documents, read where they are.
2. Ingestion: streaming and change data capture for what must be fresh; scheduled jobs for the rest; parsing and embedding for documents.
3. Store: one copy of the data in a lakehouse, plus the vector and keyword index, with quality checks, lineage and access control shared by both.
4. Use: regulatory returns, dashboards and reconciliation on one side; model inputs, retrieval and agents on the other.

Most of the effort, and most of the risk, sits in steps 1 to 3. That is why, for a bank that wants AI, the first paid project is usually "AI-ready data": classic pipeline work with a new customer.

What to do about it

• Audit the pipelines before buying GPUs: which feeds are late, duplicated or unowned.
• Move the few feeds that need it to streaming; leave the rest in batch.
• Put the data in open formats, so the copy you make today is the copy the AI reads tomorrow.
• Carry access rules into the index from the start; retrofitting them is where pilots die.

Our streaming and lakehouse services cover the build, and the Kafka and data pipeline labs show the components running on our own cluster.