Access full report
Oops! Something went wrong while submitting the form.
🤍
Facilitated by The Modern Data Company in collaboration with the Modern Data 101 Community
Latest reads...
.png)
.png)
TABLE OF CONTENT
TL;DR
TL;DR
- The Shift in Consumers: The primary consumer of data pipelines has shifted from human analysts opening weekly dashboards to non-deterministic AI models, RAG applications, and autonomous agents making split-second decisions.
- The Unstructured Explosion: Over 80% of enterprise data is unstructured. Modern data engineers must move beyond relational tables to engineer embedding pipelines, vector databases, and semantic indexing layers.
- Batch to Real-Time Streaming: AI agents operate at machine speed; stale data degrades model accuracy. Real-time context streaming is replacing traditional overnight batch processing for operational AI workloads.
- Governance as a Precondition: Data pipeline failures cause over 27% of production AI agent crashes. Data governance and quality contracts must be embedded directly into pipeline architectures as reusable data products rather than checked retrospectively.
How Is AI Changing Data Engineering?
Imagine a restaurant that has been running the same way for twenty years. Every morning, trucks arrive with neatly labelled boxes. The kitchen staff knows what is coming, where everything belongs, and how much needs to be prepared. Orders come in, chefs cook the food, and customers sit down to eat.
That is roughly how traditional data engineering worked.
Data arrived in predictable formats. Pipelines moved it from one system to another. Warehouses stored it. Dashboards consumed it. Data engineers made sure everything moved reliably from one stage to the next until AI changed the nature of work.
The days when data engineers dealt only with structured tables and predictable schemas are history already. Today they are increasingly working with documents, images, audio, video, application logs, and other forms of unstructured data. They are building systems that need to support AI models, real-time applications, recommendation engines, and autonomous agents alongside traditional analytics workloads. The role is expanding because the expectations placed on data infrastructure are expanding.
According to MIT Technology Review Insights research, data engineers spent 37% of their time on AI-related projects in the year 2025, compared with just 19% in 2023, which is a significant shift in just two years. With the advent of AI tools and the ease they extend, AI-related work is swiftly becoming a much larger part of data engineering. The change is not just about using AI tools to write code faster; the bigger change is in what data engineers are now expected to build.
For years, the typical consumer of a data pipeline was a person opening a dashboard or an analyst running a query. But now, that consumer could be an AI model, an application, or an AI agent that uses data to make decisions and take action.
That changes everything from the type of data engineers need to manage to the speed at which it needs to be available and the level of governance required to keep it trustworthy. AI is not replacing data engineering. It is changing what data engineering needs to be.
[playbook]
This is how a traditional ETL Flow looks: [Structured Inputs] ➔ [Batch Pipelines] ➔ [Data Warehouse] ➔ [Human Dashboards / BI]
This is how a Modern AI Data Architecture looks: [Unstructured / Real-Time Data] ➔ [Streaming & Embeddings] ➔ [Vector DBs / Lakehouse] ➔ [AI Models & Agents]
Traditional ETL vs. AI-Native Data Engineering
The shift outlined above is real, but it isn’t friction-free. A recent survey covered by VentureBeat found that 77% of data engineering teams report heavier workloads despite widespread AI tool adoption. About 83% of organisations have already deployed AI-based data engineering tools, yet 45% cite integration complexity as a top challenge and 38% point to tool sprawl and fragmentation. In other words, moving to the right-hand column of this table is less a matter of installing new tools and more a matter of consolidating and re-architecting how those tools work together.
How Is AI Changing the Role of Data Engineers?
For a long time, data engineering was largely about making data available. Data came from operational systems. Engineers built pipelines to move it. Transformations cleaned and structured it. Warehouses and data lakes stored it. Analysts and business users then used that information to create reports, dashboards, and models. The process was complex, but the basic flow was relatively predictable. AI is adding several new layers to this process.
The data itself is becoming more complex. Engineers now need to work with structured and unstructured information across a much wider range of systems. The speed at which data needs to move is changing too. A daily refresh may be perfectly acceptable for a monthly business report, but it is not enough for an AI application that needs to respond to an event that happened a few seconds ago.
The consumer of the data is changing as well. A human analyst can look at a dashboard, notice something unusual, and question whether the number makes sense. An AI system may take that same number and use it as part of another decision. An autonomous agent may even act on it without a person reviewing every step.
That raises the bar for data engineering. The pipeline cannot simply be operational. The data needs to be accurate, current, discoverable, governed, and trustworthy enough for machines to use it. This is why AI is changing the data engineering role at almost every level. Engineers are increasingly responsible for building the infrastructure that allows AI systems to access enterprise information and use it reliably. The job is moving from simply moving data to creating the foundation on which intelligent systems can operate.
[related-1]
How Are Data Engineers Working With Unstructured Data?
For decades, data engineering was built around structured information. The data had a defined shape, and engineers knew what that shape was. But enterprise information has never been that tidy.
A significant amount of organisational knowledge is stored outside traditional databases. Industry estimates from IDC and Gartner put unstructured data at roughly 80 to 90 per cent of all enterprise data. The same research suggests technology spend is inverted relative to where the data actually sits, with a disproportionate share of budget still going toward the smaller structured minority. That creates a very different challenge.
Imagine you walked into a rich library where half the books have no titles, no author names, and no catalog entries. The information is available technically, but the problem is finding the right information when you actually need it. That is the challenge of unstructured enterprise data. AI is making it easier to work with this information.
Embedding models can represent different types of content in ways that allow systems to identify relationships based on meaning. Vector search can then retrieve information based on semantic similarity rather than relying only on exact keyword matches. That’s where vector databases have become increasingly important beyond their earlier use in specialised machine learning workloads. They now play a role in semantic search, recommendation systems, retrieval-augmented generation, and other AI applications that need to work with information that does not fit neatly into rows and columns.
For data engineers, this means the job is expanding beyond traditional ETL and ELT workflows. They increasingly need to build systems that can ingest, process, index, retrieve, and serve information across different data formats. The challenge is no longer simply moving data from one structured system to another. It is making the entire information landscape of an organisation useful to both humans and machines.
What Are AI-Powered Data Pipelines?
For years, building and maintaining a data pipeline involved a significant amount of repetitive engineering work. Someone had to write transformations, configure workflows, monitor jobs, investigate failures, update schemas, and fix issues when an upstream system changed. AI is beginning to take some of that work off the engineer’s plate.
AI-assisted development tools can generate parts of transformation code, explain pipeline failures, suggest optimisations, identify unusual behaviour, and help engineers work through problems faster. More advanced systems are also moving towards AI agents that can perform increasingly complex tasks across the data lifecycle.
Specialised AI systems are already being explored for tasks such as database schema optimisation, ETL pipeline generation and maintenance, and predictive resource scaling across cloud environments. The value here is not that AI eliminates engineering, but it removes some of the repetitive work surrounding engineering.
A data engineer who previously spent hours investigating a pipeline failure may now be able to use AI to understand the likely cause in minutes. An engineer writing a transformation may be able to generate the initial code with AI and then spend more time reviewing the logic and handling the edge cases. That changes where engineering effort goes.
Instead of spending as much time on repetitive implementation and maintenance, engineers can focus more on architecture, reliability, governance, performance, and the business requirements the platform needs to support.
There is, however, an important difference between using AI as an assistant and building AI into the data engineering workflow itself. Many data professionals already use AI every day. They use it to generate SQL, explain errors, write code, or document systems. But that does not necessarily mean their organisation has adopted AI-native data engineering practices. The tools are becoming widely available while the systematic adoption is still developing.
[related-2]
How Are AI Agents Changing Data Pipeline Management?
The more interesting shift is happening when AI moves from assisting engineers to performing parts of the work itself. AI agents are increasingly capable of handling tasks that once required manual intervention. They can monitor systems, investigate problems, recommend changes, and in some cases execute actions based on predefined permissions. This could eventually change how data pipelines are operated.
Instead of waiting for an engineer to discover that a pipeline has failed, an AI agent could identify the problem, investigate the upstream cause, determine whether a change is safe, and either recommend or execute a fix.
The potential is significant and so are the risks. Recent analysis of AI agent production deployments found that 27% of agent failures trace directly back to data quality and freshness issues rather than problems with the underlying model or agent design.
A data pipeline rarely exists in isolation. It feeds dashboards, analytical models, applications, and increasingly, other AI systems. This means AI agents working with data infrastructure need more than technical capability; they need context.
They need to understand what they are allowed to access. They need to know which systems they can modify. They need clear boundaries around what actions they can take. And organisations need a way to monitor those actions. The more autonomous these systems become, the more important governance and observability become.
This is also why the role of the data engineer is unlikely to disappear. The engineer’s work may move further away from manually fixing every individual problem and towards designing the systems in which automated tools and AI agents can operate safely. The focus shifts from performing every task to designing how those tasks should be performed. That is a meaningful change in the role.
[state-of-data-products]
Why Is Real-Time Data Engineering Becoming More Important?
Traditional data warehouses were largely designed around batch processing. Data could be collected throughout the day, processed overnight, and made available the next morning. For many business use cases, that model still works. But AI applications often operate on a very different timeline.
A fraud detection system may need to respond while a transaction is taking place. A recommendation engine may need to react to a customer’s latest behaviour. An AI agent may need current information before deciding what to do. An operational monitoring system may need to identify an issue within seconds rather than waiting for the next scheduled pipeline run. In these situations, stale data quickly loses its value.
This is why real-time data engineering is becoming increasingly important for AI workloads. Forrester has made a similar case: unlike human decision-makers, AI agents act at machine speed, so the freshness of the context they’re given directly shapes the quality and safety of what they do. Streaming architectures and real-time analytical systems allow organisations to process information continuously rather than waiting for a scheduled batch process.
The shift does not mean batch processing is disappearing. There are still many workloads where processing data every hour or every day is perfectly reasonable. The important change is that data engineers now need to think about freshness as part of the architecture.
The question is no longer simply where the data should be stored. It is also about how quickly the information needs to become useful. That decision increasingly determines whether an organisation needs traditional batch pipelines, real-time processing, or a combination of both. As more applications become responsive and AI systems become more operational, the demand for timely data will only increase.
Why Does AI Data Governance Need to Be Built Into Data Pipelines?
AI systems are only as reliable as the data they depend on. That principle has always been true, but it becomes much more important when AI systems are making decisions or taking actions.
Imagine a food production line. You would not wait until a product reaches the customer before checking whether the ingredients were safe. Quality controls need to exist throughout the process, from sourcing to production to delivery. Data governance is moving in a similar direction.
Traditional governance mainly relied on policies, documentation, periodic reviews, and manual approval processes. These approaches can work, but they become difficult to maintain when data is constantly changing, and AI systems are consuming information across multiple environments. It needs to know what information it can use and what it should not access. As a result, governance is increasingly being embedded directly into data platforms and pipelines.
This isn’t a hypothetical problem. A second MIT Technology Review Insights study surveying 800 senior data and technology executives found that organisations are, by their own self-assessment, doing no better today at delivering on data strategy than they were before generative AI took hold. Only 12% rated themselves as data “high achievers” in 2025, compared with 13% in 2021. That stagnation is a large part of why governance built into the pipeline, rather than bolted on afterward, is becoming necessary rather than optional.
Instead of checking everything periodically, organisations are moving towards continuous monitoring and automated controls. Instead of documenting lineage after the fact, lineage can become part of the data product itself. Instead of discovering sensitive information during the next scheduled audit, automated systems can identify potential risks as data changes.
The data product approach is central to this design. It provides a governed, reusable, versioned asset that brings together the data itself with its transformation logic, quality contracts, access policies, and lineage. The important shift is that governance is not treated as something that comes after the data has already been created. It is part of the data product from the beginning. This becomes especially important when organisations are dealing with sensitive information.
How Are Data Engineers Testing AI and Data Pipelines?
There is another practice gaining attention as data and AI systems become more complex: Evaluation-Driven Development. The idea is not entirely new.
Software engineering has spent years building testing into the development process. Teams use version control, automated tests, CI/CD, and monitoring because discovering a problem after deployment is usually much more expensive than catching it before release.
Data and AI systems are beginning to follow the same path for a simple reason: the consequences of a data problem are becoming larger. If a dashboard receives incorrect data, someone may eventually notice the problem. If a machine learning model receives incorrect information, its predictions may become unreliable. If an AI agent receives incorrect information and is allowed to act on it, the consequences can move much faster.
The important change is that teams are no longer asking only whether a pipeline ran successfully. They are asking whether the output was actually correct. That distinction matters.
As data pipelines become the foundation for AI applications, testing needs to go beyond whether the system is operational. Teams increasingly need to evaluate the quality and reliability of the results being produced.
Are Data Infrastructure and AI Infrastructure Converging?
Underneath all these trends is a larger structural shift. The boundaries between data infrastructure and AI infrastructure are fading out increasingly.
Historically, organisations often operated separate environments for analytics, business intelligence, machine learning, and AI. These systems were connected, but they were generally treated as different parts of the technology stack.
Gartner has predicted that fragmented data management markets will increasingly converge into a unified ecosystem by 2028. It has also projected that 80% of GenAI business applications will be built directly on organisations’ existing data management platforms.
The reasoning is fairly straightforward. If an organisation already has a data platform that understands its customers, products, transactions, operations, and business processes, it makes little sense to build an entirely separate foundation for every new AI application.
The challenge is that many existing data platforms were not originally designed for this new world. The modern data platform is therefore becoming much more than a place to store information. It is becoming the foundation for enterprise intelligence.
[related-3]
What Is the Future of Data Engineering in an AI-First World?
Every trend points towards the same conclusion: the role of the data engineer is expanding.
The job is no longer simply to build pipelines that move data from one system to another. Data engineers are increasingly building the infrastructure that allows AI systems to find, understand, trust, and act on information.
The consumer of that infrastructure could be an analyst, a machine learning model, a generative AI application, or an autonomous agent making decisions on behalf of the organisation. That changes the expectations placed on the data platform.
Data needs to be available when it is needed. It needs to be accurate. It needs context. It needs governance. It needs to be observable. And when machines begin acting on that information, organisations need confidence that the systems behind it are behaving as expected.
This is why the future of data engineering is not simply about adding AI to existing pipelines. It is about rethinking what those pipelines are designed to support. The strongest data engineering teams in 2026 will not necessarily be the ones using the most AI tools. They will be the ones that understand where AI can remove repetitive work and where human judgement remains essential.
The data engineer’s job is changing. But perhaps the more important point is that the role is becoming even more central to the organisation. The models may be the part everyone talks about. The AI agents may be the part everyone sees. But underneath all of it is a data foundation. And someone still has to build it.
FAQs
Q1. Is AI replacing data engineers?
AI is not replacing data engineers. It is changing how they spend their time. AI-assisted tools can automate repetitive work such as code generation, pipeline troubleshooting, documentation, monitoring, and optimisation. This allows data engineers to spend more time on architecture, reliability, governance, and system design. As AI becomes more capable, the role is likely to move further towards designing and managing the infrastructure that AI systems depend on.
Q2. How is AI changing data pipelines?
AI is making data pipelines more automated and capable of handling more complex data. AI-assisted tools can help engineers generate transformations, troubleshoot failures, and optimise workflows. At the same time, AI applications are creating demand for pipelines that can handle unstructured data and deliver information in real time rather than relying entirely on traditional batch processing.
Q3. Why is unstructured data important for AI data engineering?
A large amount of enterprise knowledge exists in documents, images, videos, audio recordings, emails, and logs rather than traditional databases. AI applications increasingly need access to this information. Data engineers therefore need to build systems that can process, index, search, and retrieve unstructured data. Embeddings and vector databases are becoming important components of this architecture.
Q4. Do all AI applications need real-time data pipelines?
No. Batch processing remains useful for many analytical and reporting workloads. Real-time data pipelines become important when an application needs to respond to rapidly changing information. Fraud detection, personalised recommendations, operational monitoring, and autonomous AI agents are examples where stale data can reduce the usefulness or reliability of the system.
Your Copy of the Modern Data Survey Report
See what sets high-performing data teams apart.
Better decisions start with shared insight.
Pass it along to your team →
https://www.moderndata101.com/survey
Your Copy of the Modern Data Survey Report
See what sets high-performing data teams apart.
Better decisions start with shared insight.
Pass it along to your team →
https://www.moderndata101.com/survey
Author Connect 🖋️
Find more community resources
About Modern Data 101
Modern Data 101 is a movement redefining how the world thinks about data. A community built by the same team behind the world’s first data operating system, Modern Data 101 sits at the intersection of data, product thinking, and AI. Spread across 150+ countries, the community brings together a global network of practitioners, architects, and leaders who are actively building the next generation of data systems.
At its core, Modern Data 101 exists to simplify the journey from raw data to tangible and observable impact. It advocates high-potential data systems and next-gen architectures to unify and activate insights and automation across analytics, applications, and operational workflows at the edge.
In a world shifting from data stacks to AI ecosystems, Modern Data 101 helps teams not just navigate the change but lead it.
Access full report
Oops! Something went wrong while submitting the form.
