Scaling Data Platforms With AI

Start with the basic promise of a platform: more data shouldn't mean proportionally more cost. In practice, most enterprises don't see that curve bend the right way, because they're still measuring cost at the warehouse or cluster level: one number that blends a hundred different jobs together. You can see total spend rise, but you can't see which query, which team, or which dashboard actually drove it. This is a definite visibility problem instead of pricing.

• Real economies of scale start once you push attribution down to the execution level: tagging spend by query, job, and owner, not just by warehouse. So cost becomes something engineers can act on instead of something finance reports on after the fact.
• Layer AI on top, and the unit of spend changes again: tokens are becoming what compute used to be, and a poorly designed agent that rebuilds its own context on every call will multiply your bill.‍
• What works for us is enabling a shared context layer once and letting every agent and model draw from it, rather than letting each one re-derive the same answer at its own expense.

AI is genuinely good at the mechanical layer of this work. Cleaning data, writing the first draft of a query, generating boilerplate pipeline code. Analysts have historically spent the large majority of their time on exactly that kind of prep work, so automating it isn't a threat but a relief.

The judgment layer, deciding what question actually matters to the business, and what an anomaly means in context, is still squarely human.

• For engineers specifically, the shift is more structural. A lot of orchestration logic exists today only because nobody has gone back to delete it. AI is making that cleanup, and a good chunk of net-new pipeline work, increasingly automatic.

What doesn't automate is the responsibility for whether an autonomous system can be trusted. Lineage, observability, and guardrails that catch a bad decision before an agent acts on it. An approach proving wise for data and AI leaders is to become someone who designs the trust boundaries that let AI operate inside them safely.

Most budget conversations get framed as a tug-of-war: dollars for new things versus dollars for keeping things safe. That is a recurrent problem. Governance gets bucketed as defence, which makes it the first line cut when growth needs funding.

And that's backwards, because ungoverned data is exactly what slows growth down. Nobody trusts a number they can't trace, so they re-verify it manually, and that re-verification is itself a hidden tax on speed.

The more durable approach is to stop treating governance as a separate line item competing with offence, and build it into the data platform itself. Policy enforcement, lineage, and access control as defaults that ship with the platform rather than a parallel program.

At real scale, this isn't optional: AI governance and data governance have become the same problem viewed from two angles, because an AI system making a decision you can't trace is simultaneously an audit problem, a compliance problem, and a velocity problem. Fund it like infrastructure.

Shadow data is fed because someone needed an answer faster than the official path could deliver it, so they pulled a CSV, stood up their own database, or copied a table somewhere convenient.

Multiply that decision across a few hundred teams over a few years, and you get a second, invisible data estate that nobody governs and almost everybody depends on.

The instinct is to fight this with more enforcement, but that usually just pushes the behaviour further underground. The actual fix is making the governed path the fastest path: self-service provisioning with guardrails already attached, so the easiest way to get data is also the compliant way to get it.

If your data platform is slower than a workaround, the workaround wins every time, no matter how many policies you write. This is the same lesson data mesh learned the hard way: you don't solve decentralisation by recentralising harder. We solve it by making the central platform genuinely good enough that nobody feels the need to disassociate from it.

The most common pattern is buying into a vendor's roadmap instead of your own. The era of cheap capital encouraged a lot of companies to stack up specialised tools. One for ingestion, one for transformation, one for orchestration, one for quality, and so on.

• ‍Each justified in isolation but never evaluated as a system. The bill that shows up later isn't really the sum of those subscriptions. It's the integration tax. The hours engineers spend glueing tools together and cross-checking values instead of shipping anything.

Consolidation is a winner. Fewer, more deeply integrated systems instead of nine specialised point solutions. The impact gap almost always traces back to the same root cause: a stack too fragmented for a business user to touch without a technical translator standing between them and the data. Data Platforms only pay off when they're simple enough that the people who actually need the insight can get to it themselves.

Data architectures today still carry one stale assumption: that a human will look at a dashboard once a day. Once AI agents become the primary consumer of your data, that assumption doesn’t hold at all. Agents query continuously and act on whatever they find in the moment.

• The risk is, in fact, quite fatal. Retrofitting a batch-first architecture for real-time use later is one of the most expensive and disruptive migrations a data team can take on, far more costly than designing for it from the start.

The Data Leaders who are likely resisting that aspect will have AI projects on their hands that most often don’t match up to the ROI promises. AI built on such foundations would be unreliable, and the instinct will be to blame the model. When the culprit is the data infrastructure underneath it, which is unable to deliver fresh, trustworthy context fast enough to support AI operations.