10 Ways to Make LLMs Cheaper and Faster Using Data Products

Originally published on

, the following is a revised edition.

In the world of AI, Large Language Models (LLMs) are like having a super-smart, incredibly resourceful intern. They can figure out code, draft emails for operational communications, summarise documents, and even brainstorm wild ideas. The catch? Like any high-performing talent, they can get expensive and, if not managed well, can slow things down.

If your LLM-powered applications are draining your budget fast or taking their sweet time to respond, it's not an anomaly and you're not alone. The secret to unlocking their true potential, both in terms of cost and speed, lies not just in the models themselves, but in how you feed them. Eventually, environment matters more than capability.

Think of it like this: you wouldn't give a Michelin-star chef half-rotten ingredients and expect a gourmet meal on time, right? In fact, a star chef would tell you the magic IS IN THE INGREDIENT! To let them naturally flourish through the right cooking (aka the right modelling). You'd provide them with prepped, high-quality components. The same goes for LLMs.

Good AI starts with good data. Treating Data as first-class citizens transforms their quality | (Image source)

‍

Data isn't just a foundation to kickstart any AI project; it's high-quality data as LLM fuel: the precision fuel that dictates how efficiently and effectively your LLMs run. Without smart data delivery, you’re essentially paying top dollar for an LLM to do basic data prep work.

This is where Data Products become relevant. By treating your data as if they were key ingredients, and how they're delivered to LLMs or AI models, you can drastically cut costs and ensure inference latency reduction. Here are 10 ways to do just that.

Top 10 Ways to Make LLMs Cheaper and Faster Using Data Products

Here are 10 ways to achieve data‑product‑driven LLM optimisation, making them cheaper, faster, and smarter.

1. Use Data Products to Feed Clean, Context-Rich Inputs

This is fundamental. Consider explaining a complex project to someone by handing them every single document, email, and sticky note ever created. All context they'd ever need right? No. They'd drown in information, not know where to start, and end up in confusion instead of clarity.

LLMs face a similar challenge when fed raw, unfiltered data.

By using **well-defined, versioned data products (**what we call versioned context delivery) ****to build curated context pipelines, you ensure your LLM consumes only what it truly needs from a dedicated, curated output port. This isn't just about tidiness; it's about reducing preprocessing time, minimising the risk of the LLM "hallucinating" due to irrelevant noise, and critically slashing your token usage. Every unnecessary word costs tokens, and tokens cost money. Less garbage in, less garbage out, less money spent on garbage. LLM cost containment success.

Diagram showing raw data sources flowing into a versioned, cleaned, and chunked data product pipeline, which then delivers curated context directly to an LLM. — High-Precision Context-Matching for High LLM Success, with ample context around data, abolishing the debt of Garbage-In-Garbage-Out (GIGO) | Image Source: Animesh Kumar

‍

2. Structure Data as First-Class Citizens in the AI Ecosystem, Not Unstructured Dumps

We often think of LLMs as masters of unstructured text, but that doesn't mean you should just dump raw documents on them and hope for the best. AI is unfortunately NOT a magic wand. While they can handle it, they process structured formats like JSON, Markdown blocks, or pre-computed embeddings far more efficiently. Schema-aligned prompting shows how structure really drives consistency.

When your data products deliver model context as structured input formats (observe that context is structured, data doesn't have to be), the LLM spends less effort parsing and understanding the input, and more time generating a quality response. Structured inputs mean LLMs process less junk, respond faster, and ultimately become cheaper to run.

Diagram showing raw data sources (like documents or logs) being processed into structured formats (such as JSON or pre‑computed embeds), which are then passed into an LLM retriever or prompt as clean, schema‑driven context, reducing parsing time, token usage, and ambiguity. — **From Unstructured to Structured Context.** Raw documents and logs are processed into structured formats, such as JSON chunks or validated data schemas, then provided as clean, predictable model context. This avoids dumping raw text, simplifies parsing for the LLM, and cuts down on token overhead while improving reliability. | Image Source: Nick Hagar

‍

3. Isolate and Curate Context with Output Ports

One of the quickest ways to inflate LLM costs and dilute accuracy is to overload prompts with too much irrelevant context. It’s tempting to provide "all" the data, but it's redundant. Consider asking for directions to the nearest coffee shop and getting handed a map of the country.

By using data product output ports to serve precise, isolated slices of context (e.g., just the latest customer service notes, specific product specifications, or a subset of FAQs), the prompt length can dramatically come down. The sharpness and precision in the translated query directly translate to fewer tokens consumed, preserving accuracy by ensuring the LLM focuses only on the most relevant information. This keeps your LLM concise, accurate, and your bill significantly lower, enabling token savings at scale.

Diagram showing a refined retrieval pipeline where a filtered, focused query feeds only the most relevant context into an LLM, resulting in fewer tokens, better answers, and lower cost. — **Precision in = Efficiency out.** By narrowing the query to just the relevant context, you reduce token usage, improve accuracy, and minimise cost. All without compromising response quality. | Image Source: Damien Benveniste

‍

4. Standardise Prompt Templates Based on Data Product Schemas

If every user or application sends prompts to your LLM in a slightly different format, you're essentially asking the LLM to relearn your query structure every single time. It's inefficient and error-prone.

By designing prompt templates that align directly with the structure (schemas) of your data product’s outputs, you create a consistent, predictable interface. This standardisation reduces variability, dramatically improves the reliability of responses, and critically, enables caching for common queries (more on caching soon!). Think of it as having a standardised order form for your LLM. It makes processing faster, more predictable, and cuts down on frustrating "misunderstandings."

ALT Text: Diagram showing a structured data product schema feeding into a prompt template. The template enforces consistent placeholders aligned with schema fields, then passes standardised prompt output to an LLM, yielding predictable, cacheable responses and minimising prompt variability. — **Schema-Aligned Prompting: Consistent interfaces, reliable responses.** Prompt templates defined according to your data product schema provide a stable, standardised structure, reducing variability, boosting predictability, and enabling caching. | Image Source: Jeffrey Ip

‍

5. Leverage the Self-Service Platform for Input Selection

Data Engineers often become bottlenecks, manually prepping and integrating data for new LLM experiments. This is slow and expensive, tying up valuable engineering time. By providing a **self-service platform that allows users (data scientists, analysts, business users) to intuitively choose which data product and which specific output port to pull context from**, you democratise data access for LLMs

Diagram showing a multi-layered self-service data developer platform: a central Control Plane for governance/orchestration, a Development Plane for data product specifications and workload management, and a Data Activation Plane for deployment and execution, enabling domain teams to self-serve and compose data products. — A hierarchical, multi-plane architecture (Control Plane, Development Plane, Data Activation Plane) enabling a unified self-service data developer platform. Domain teams can autonomously build and consume data products using modular building blocks, mirroring the long-standing principles of datadeveloperplatform.org.

‍

No engineering required for every new experiment! This empowers domain experts to quickly iterate and test ideas, making the experimentation phase significantly cheaper and faster, and ultimately accelerating time-to-value for new AI applications.

A platform UI/architecture diagram where users (domain teams) can browse and select from multiple data products and their output ports. Once selected, those outputs feed into downstream AI workflows, enabling self-service context selection without centralized engineering intervention. — Domain teams (analysts, data scientists, business users) browse a catalogue of available data products via a unified platform and select specific output ports for LLM input without needing engineering support. This empowers experimentation, speeds iteration, and avoids bottlenecks | Image Source: Google’s Cloud Architecture Centre

‍

6. Bundle Preprocessing + LLM Calls as Reusable Pipelines

If different teams are each writing their own scripts to chunk documents, embed text, then call an LLM API, you're looking at a lot of duplicated effort and operational overhead. You’ll hit AI performance bottlenecks as you scale. This is where the magic of a platform-driven approach truly shines.

Use your data platform to create shareable, reusable preprocessing pipelines. In other words, callable AI pipelines that encapsulate the full LLM workflow: input port → necessary data transformation/chunking → LLM prompt construction → LLM API call → response processing. These callable AI pipelines help mitigate AI performance bottlenecks, avoid repetitive work, standardise best practices, and drastically reduce operational overhead. It’s like having pre-built, optimised workflows for common LLM tasks, letting teams focus on unique business logic rather than re-inventing the wheel every time.

Diagram illustrating a reusable AI pipeline: raw documents flow into preprocessing stages (cleaning, chunking, embedding), then through prompt templating and an LLM API call, with output processing, all packaged as a shared, callable pipeline accessible by various users or teams. — **Modular, reusable pipelines for end-to-end AI workflows.** This diagram visualises a shareable pipeline that bundles document ingestion, chunking, embedding, prompt construction, LLM API calls, and response handling into a single, callable workflow that multiple teams can leverage, minimising duplicated effort and operational overhead. | Image Source: Modern Data 101 Archives

‍

7. Cache Token-Heavy Inputs and Partial Results

Some parts of working with LLMs are inherently expensive, especially if they involve large context windows or complex reasoning steps. Think of it like paying a premium for a very specific, rare ingredient every time you cook, even if you only need a tiny bit of it.

By integrating semantic result caching at the platform level, especially for expensive LLM sub-tasks like document chunking, embedding generation, or even common query responses, you can slash recurring costs. If the LLM has already processed and embedded a specific document, cache that embedding! If a common query has a stable answer, cache the response! This significantly reduces redundant LLM calls and token usage, leading to massive cost savings over time.

Diagram illustrating a caching workflow: user query is embedded and checked against a cache first; on a cache hit, the cached result is returned immediately; on a cache miss, the query is sent to the LLM, and the response plus embeddings are stored in the cache for future reuse. — **Reduce token costs with semantic caching.** Platform-level caching captures embedding results and query responses so that repeated or semantically similar inputs bypass expensive LLM calls, cutting latency and saving on tokens. | Image Source: Arun Shankar

‍

8. Split Reasoning Tasks Across Tools, Not Just the LLM

LLMs are incredible at natural language understanding and generation, but they're not always the cheapest or fastest tool for every job. Asking an LLM to do simple arithmetic or perform complex database joins is like hiring a rocket scientist to sort your mail. Obviously they can do it, but it’s overkill and overpriced.

Offload simpler logic to other, more efficient components within your data platform. Use SQL for precise factual retrieval, heuristics for filtering, or traditional code for rule-based logic. Reserve LLMs for where their true power lies: complex reasoning, nuanced natural language processing, and creative generation. This, not using a sledgehammer to crack a nut approach ensures you are using LLMs only when their unique capabilities are truly needed. This saves significant costs improving overall system performance.

Diagram separating simple, structured logic (e.g. SQL queries or heuristics) from LLM-driven reasoning: user input is evaluated, some tasks (retrieval, filtering) are handled by efficient tools, then the LLM is used only when needed for nuanced language processing or decision making. — **Hybrid reasoning architecture: Offloading basic logic to efficient systems.** SQL and rule-based components handle routine tasks like filtering and data retrieval, while the LLM is invoked only for complex reasoning and generation, preserving speed, reducing token usage, and cutting costs | Image Source: Arun Shankar

‍

9. Use Feedback Data to Tune the Output Port or Prompt

Your LLM applications don't live in a vacuum. User interactions provide a goldmine of information about what's working and what's not. Ignoring this is like building a product without ever talking to your customers.

Instrument feedback loops directly into your data products and LLM integration, enabling feedback-driven schema evolution that continually refines output quality. This feedback, be it the user upvotes/downvotes, explicit corrections, or implicit behavioural signals, can be used to fine-tune how the data product outputs are structured. The continuous feedback loop improves both the speed and the trustworthiness of your LLM responses while also ensuring that your AI is getting smarter and more efficient.

Diagram showing a circular feedback loop where user feedback (ratings, corrections, signals) is captured, processed through a metrics/quality layer, and then used to adjust data product schemas or prompt templates for future LLM calls. — **Closing the Loop with Feedback.** User interactions (upvotes, corrections, triggers) feed into the metric store or data quality layer, which then informs updates to prompt templates and output schemas, delivering smarter, more reliable LLM responses over time. | Image Source: Animesh Kumar

10. Continuously Optimise Through Platform Governance

Finally, getting LLMs cheap and fast isn't a one-and-done deal, it's an ongoing journey. Without visibility, costs can quietly swell and performance can degrade.

Building platform governance dashboards that track LLM usage by data product, team, and specific use case gives you the transparency needed to manage your LLM budget bloat. This also enables AI governance automation, allowing platform teams to identify expensive usage patterns, guide users toward faster, cheaper configurations, and enforce policies for optimal LLM consumption.

Architecture diagram of a data governance layer: illustrates a centralized control plane that collects audit logs and usage metrics for all LLM-backed data products, enabling policy enforcement, cost attribution, anomaly detection, and governance across teams and use cases. — **Platform governance for sustainable LLM scale.** This architecture diagram shows a centralized governance and control plane overseeing data product usage across domains, continuously monitoring usage, enforcing policies, and optimising cost by tracking LLM consumption by team, use case, and product. | Image Source: Ritwika Chowdhury

Final Note

The temptation to adopt an LLM is undeniable, and it does go with the trend, but its real-world impact hinges on managing the operational costs and speed. This shift to AI-native applications is an evolution in our data strategy too. By leveraging the principles of Data Products, organisations can fundamentally transform how they interact with LLMs.

From serving cleaner inputs and structuring data intelligently, to enabling self-service, bundling workflows, and implementing smart caching and governance, each of these 10 strategies contributes to a leaner, faster, and more effective LLM deployment. It's not just about getting answers, it's about advancing to smart answers with speed and affordability.

FAQs

❓ How can data products reduce the cost of running LLMs?

Data Products can route queries to the most cost-effective LLM based on task complexity, automatically compressing and refining the prompts to minimise token consumption, implementing caching to reuse past responses for common queries, leverage RAG to provide precise, relevant context which reduces the amount of data the LLM needs to process, and enable the fine-tuning of smaller, specialised models that are cheaper to operate, Data Products can significantly reduce the cost of running LLMs.

❓ What is the role of a self-service platform in optimising LLM performance?

A self-service platform provides intuitive interfaces for non-technical users to access, configure, and manage LLM resources without relying on data scientists or engineers. This includes features like simplified prompt engineering interfaces to easily test and iterate on prompts, access to various pre-trained or fine-tuned models for specific tasks, and tools for monitoring model performance and cost in real-time. By democratising access and control, a self-service platform accelerates experimentation, fosters quicker iteration on prompts and models, and allows teams to rapidly deploy and scale LLM-powered applications.

❓ Why is feeding LLMs with curated data better than using raw inputs?

Curated data provides precise context, reduces the noise, hence minimising the chances of the LLM going off-topic, allowing the model to focus its processing power on generating accurate and concise responses. It is specifically prepared to be clean, relevant, and structured, directly addressing the LLM's needs. Raw inputs often contain irrelevant information, or unstructured formats that can confuse the model, leading to hallucination, inaccurate outputs, and increased processing time and cost as the LLM struggles to sift through unnecessary data.

Join the Global Community of 10K+ Data Product Leaders, Practitioners, and Customers!

Connect with a global community of data experts to share and learn about data products, data platforms, and all things modern data! Subscribe to moderndata101.com for a host of other resources on Data Product management and more!

‍