JLT

Tag: Agentic AI

  • AI Infrastructure Cost Management: The Hidden Costs Nobody Talks About

    Every team I talk to that runs AI in production says the same thing once the initial excitement fades: the costs are higher than they expected. Not because of bad planning. Because the actual cost structure of production AI systems contains line items that nobody puts in the original budget.

    Compute Costs Are Just the Starting Point

    The obvious expense is inference compute. GPU time, API calls, token consumption. Teams budget for this and generally get it right within a reasonable range. The problem comes from everything else.

    Cold start latency forces many teams to keep models loaded even during low-traffic periods. A model sitting in memory on an idle GPU cluster still costs money. The math changes when you look at 24/7 operation versus actual usage patterns.

    Data Pipeline Maintenance

    Production AI systems are only as good as their data pipelines. When those pipelines break, models serve stale information or fail entirely. Maintaining these pipelines requires:

    • Continuous data validation and quality checks
    • Pipeline monitoring that catches drift before it impacts outputs
    • Engineering time to fix pipeline failures at 2 AM
    • Version control for training datasets and preprocessing logic

    Most organizations treat data pipeline costs as operational overhead rather than AI costs. They are the same thing when your AI system depends on fresh, accurate data.

    Evaluation and Testing Overhead

    Deploying a new model version requires validation. This means running test sets, comparing outputs against baselines, and running shadow deployments before cutting over traffic. Each step consumes compute and human time.

    A conservative estimate for thorough model evaluation is 40-80 engineering hours per significant model update. For teams releasing updates monthly, this adds up to a substantial recurring cost that rarely appears in AI project budgets.

    Monitoring and Incident Response

    Production AI systems require monitoring that traditional software does not. You need to track not just uptime and latency, but output quality metrics, drift indicators, and user feedback signals. When a model starts degrading, you need visibility before users report the problem.

    Incident response for AI systems also differs from traditional software. A buggy API service gets patched. A model that developed a subtle bias problem requires investigation, retraining, and validation before the fix deploys.

    The Practical Question

    Most AI cost analyses focus on the visible expenses: compute, storage, API fees. The real question is whether your organization has accounting for the invisible costs that come with running AI systems reliably at scale. What happens when you add them up for a full year of production operation?

  • AI Evaluation: Why Your Benchmarks Do not Match Production

    The AI industry runs on benchmarks. MMLU, HumanEval, GPQA — each promises to measure something real about model capability. Engineering teams use these numbers to decide which model to deploy. Product managers use them to set expectations. Investors use them to compare startups.

    The problem: benchmark performance does not reliably predict production performance.

    What Benchmarks Actually Measure

    Benchmarks test a model ability on curated datasets under specific conditions. HumanEval measures code completion on LeetCode-style problems. MMLU tests knowledge retrieval across 57 subjects. Each benchmark defines narrow success criteria and holds the test conditions constant.

    Production environments do not hold anything constant. Users submit malformed inputs. Edge cases arrive in unpredictable sequences. The same question gets asked thirty different ways. A model that scores 90% on a benchmark might drop to 60% when the input distribution shifts even slightly.

    The Benchmark Gaming Problem

    When incentives are misaligned, benchmarks get gamed. Labs optimize specifically for benchmark datasets. This works — until the benchmark leakage becomes obvious and the scores lose credibility. We have seen this play out repeatedly: models that ranked high on coding benchmarks produced unusable code in production.

    The deeper issue is that benchmarks measure what gets measured. Creativity, edge case handling, and real-world judgment do not translate cleanly into standardized tests.

    What Production Teams Actually Need

    Teams deploying AI in production care about three things: latency, accuracy, and failure behavior. Latency affects user experience directly. Accuracy determines whether the output gets used. Failure behavior decides how the system degrades under stress.

    Benchmarks rarely address all three simultaneously. A model that fast might sacrifice accuracy. A model that accurate might fail in ways that are hard to detect. The trade-off space is complex, and single-number benchmarks cannot capture it.

    Building Better Evaluation Locally

    The practical alternative: evaluate on your own data, under your own conditions. Sample real queries from production. Test against the specific task you need the model to perform. Measure latency, error rates, and user satisfaction.

    This approach requires more effort than citing a benchmark. It also produces more useful results. Teams that do this consistently make better deployment decisions than teams that rely on published benchmarks alone.

    The Honest Framework

    If you are evaluating AI systems for production use, treat benchmark scores as one data point among many. Run your own evaluation. Test for your specific use case. Measure what actually matters to your users.

    The question is not whether a model is good — it is whether it solves your problem at acceptable cost and risk. Benchmarks cannot answer that. Only your own evaluation can.

    How are you evaluating AI systems for your specific use case? Are benchmarks giving you false confidence?

  • The Prompt Engineering Trap: Why More Tokens Don’t Mean Better Results

    The prompt engineering discourse has gone sideways. Somewhere between the viral Twitter threads and the $500/hour consultants, we lost the plot. The conversation shifted from “How do I get better outputs?” to “How do I craft the perfect prompt architecture?” These are not the same problem.

    I’ve watched teams spend weeks perfecting prompt templates while ignoring the actual bottleneck: they were asking the wrong questions.

    The Optimization Trap

    The assumption behind elaborate prompt engineering is that better prompts produce better results. This is true but incomplete. Better prompts produce better rephrasings of your implicit assumptions. If your assumptions are wrong, better prompting just produces wrong answers with better formatting.

    Consider the typical workflow: stakeholder describes a feature requirement, engineer prompts an AI to generate a spec, prompt gets refined to produce more detailed specs, iterations continue until the output looks polished. The spec is clean, well-structured, and completely disconnected from what users actually need.

    The optimization target drifted from “solve the problem” to “produce good-sounding output.”

    This is the trap. Prompt engineering optimizes for the artifact, not the outcome. Teams get very good at producing polished nonsense.

    What Actually Matters

    After watching this pattern repeat across dozens of projects, three factors consistently determine whether AI assistance produces useful results:

    Question quality is upstream of prompt quality. The best prompts I’ve seen aren’t syntactically sophisticated. They’re precise about what problem needs solving, what constraints exist, and what success looks like. This precision comes from the human’s understanding, not the prompt’s structure. When I see prompts with elaborate role definitions, chain-of-thought sequences, and output format specifications, I usually see a team trying to compensate for unclear thinking with prompt complexity.

    Iteration cadence beats iteration depth. The teams getting real value from AI aren’t the ones crafting perfect single-shot prompts. They’re running rapid cycles: prompt, evaluate, adjust, prompt again. A mediocre prompt run five times with feedback beats a perfect prompt run once. The learning compounds. Prompt engineering as a discipline treats prompts as finished artifacts to optimize. Effective usage treats prompts as hypotheses to test.

    Context quality beats context quantity. The race to fill context windows with documents, code, and specifications often backfires. More context means more noise. It means the AI spends tokens on relevance ranking instead of reasoning. I’ve consistently seen better results from carefully selected, highly relevant context than from comprehensive dumps. Three pages of exactly the right information outperform fifty pages of everything.

    The Meta-Problem

    Here’s what nobody talks about: prompt engineering as a practice assumes the human knows what they want. The elaborate frameworks—CoT, ReAct, Tree of Thoughts—assume you can specify the reasoning path. When the problem is figuring out what you actually need, these frameworks add structure without adding clarity.

    The teams that struggle most with AI tools aren’t the ones using bad prompts. They’re the ones who haven’t done the work to understand their own problems. AI makes it easier to produce answers. It doesn’t make it easier to ask the right questions.

    This isn’t a limitation of current AI. It’s a fundamental constraint. AI can help you explore solution spaces. It cannot help you define the problem space unless you’ve already done that work yourself.

    Practical Implications

    If you’re trying to improve how your team uses AI tools, the sequence matters:

    1. Clarify before you prompt. Spend time writing out what you actually know, what you don’t know, and what constraints exist. This work belongs to humans.
    2. Test prompts against real cases. Run your “optimized” prompt against five actual problems. Measure whether the outputs solve the problem, not whether they look polished.
    3. Favor specificity over sophistication. “Explain this error in plain English, focusing on root cause and fix” outperforms elaborate role-play scenarios and output format specifications.
    4. Build feedback loops. Track which prompts work and which don’t. The patterns matter more than any individual prompt.
    5. Know when to stop prompting. If you’ve iterated three times and the output still doesn’t solve the problem, the problem isn’t the prompt. The problem is either the question or the tool selection.

    The Honest Assessment

    Prompt engineering has value. For well-defined problems with clear constraints, thoughtful prompting improves results. The issue is that most teams use sophisticated prompting techniques on poorly-defined problems, then blame the technique when it fails.

    The people getting the most value from AI tools aren’t the best prompt engineers. They’re the ones who know when prompting is the right tool and when they need to step back and think through the problem themselves.

    The skill that matters isn’t knowing how to prompt. It’s knowing when to stop prompting and start reasoning.

    What’s your experience been like? Are you spending more time on prompt structure or problem definition?

  • AI Coding Assistants: Six Months In the Trenches

    I spent the last six months working with AI coding assistants daily. Not as a demo, but as my primary workflow. Here’s what actually changed.

    The shift isn’t about AI writing your code. It’s about how you think about problems.

    The Real Productivity Gain

    Most discussions focus on autocomplete speed. That’s the visible part. The real gain is harder to measure: reduced friction between thinking and implementing.

    When I have an idea, I can test it immediately. Describe the function in plain language, review what the AI generates, iterate. The bottleneck shifts from typing to reasoning.

    Three things surprised me:

    • Debugging time dropped: AI reads error messages differently than humans. It correlates the error with your specific codebase, not just the general pattern. Half my debugging sessions now end in minutes instead of hours.
    • Code review quality improved: When AI suggests changes, it explains the reasoning. I find myself understanding other people’s code faster because the AI can summarize unfamiliar sections.
    • Documentation got actually written: Instead of dreading the docstring, I let AI draft it and then review. This sounds minor until you realize how much institutional knowledge disappears when nobody documents the tricky parts.

    Where It Breaks Down

    AI coding assistants fail in specific ways. Understanding these failure modes matters more than the capabilities.

    Context windows are real constraints. Feed an AI a 50-file codebase and ask about architectural decisions made three years ago, you’ll get confident nonsense. The model works best with focused, recent changes.

    Security edge cases get missed. AI will suggest code that works for the happy path. It doesn’t naturally think about adversarial inputs, race conditions, or compliance requirements unless you explicitly ask.

    The biggest risk is subtle: learned helplessness. If you rely on AI to generate everything, you stop building the mental models that let you catch mistakes. The tool makes you faster until you forget how to verify the output.

    What I’d Tell My Past Self

    Use AI for the mechanical work. Let it handle boilerplate, refactoring, test generation, and initial drafts. Your job is to define what good looks like and verify the result.

    The developers who thrive won’t be the ones who use AI most. They’ll be the ones who know when to trust it and when to dig in manually.

    The question isn’t whether to use AI coding assistants. It’s whether you’re using them to augment your thinking or to replace it.

    What’s your experience been? Are you seeing real productivity gains, or is the tooling still too immature for your workflow?

  • RAG vs Fine-tuning: What Nobody Tells You

    I’ve been watching the RAG vs Fine-tuning debate unfold for months now. Every week there’s a new benchmark, a new paper, another startup claiming their approach is superior. But talking to engineering teams on the ground, the picture gets messier.

    The choice between these two approaches isn’t just technical — it shapes how your product evolves, how fast you can iterate, and what your team looks like.

    What These Approaches Actually Do

    Retrieval-Augmented Generation pulls information at query time. When a user asks something, the system finds relevant documents and feeds them into the model alongside the question. The model then generates an answer using that context.

    Fine-tuning takes a different path. Instead of retrieving information at query time, you train the model on your specific data upfront. After training, the model “knows” your domain without needing external documents.

    Both paths solve the same problem — getting a model to answer questions about your specific business — but the operational characteristics differ significantly.

    • When to Reach for RAG: If your data changes frequently, if you need to cite sources, or if audit trails matter. RAG lets you swap out documents without retraining. Legal firms and healthcare providers often prefer this because every answer can point to the exact document that informed it.
    • When to Fine-tune: If latency is critical, if you’re building specialized terminology that confuses base models, or if your data is stable but large. A fine-tuned model responds faster because nothing needs to be retrieved at inference time.

    The Hidden Cost Nobody Talks About

    The benchmarks you see in vendor marketing tell a partial story. They measure accuracy on test sets — curated questions with known answers. Real deployments are messier.

    Users ask things you didn’t anticipate. They phrase questions in ways that don’t match your document structure. They expect answers that combine information from multiple sources.

    With RAG, you can debug this by looking at what documents got retrieved. You can see if the retrieval step failed. With fine-tuning, the knowledge is baked into model weights — harder to inspect, harder to correct when the model confidently says something wrong.

    On the other hand, fine-tuned models don’t suffer from the “garbage in, garbage out” problem that plagues RAG systems. If your document retrieval is flaky, your answers will be too.

    What Teams Actually Choose

    Talking to ML engineers and product managers, I see a pattern emerging. Early-stage products tend to start with RAG because it’s faster to ship. You can connect your existing document store and have something working in days.

    As products mature, some teams migrate to fine-tuning. This usually happens when they hit latency ceilings or when they need consistent sub-second responses in user-facing applications.

    A smaller group does both — fine-tuning the model to understand domain language, then using RAG to provide up-to-date context. This is more expensive and complex, but it captures benefits of both approaches.

    The honest answer is that there’s no universally correct choice. The right approach depends on your data characteristics, your latency requirements, and how much your domain knowledge differs from what the base model was trained on.

    Which approach are you using today, and what drove that decision? I’d be curious to hear if the reality matches what the benchmarks promised.

  • AI Agent Governance: Managing Risk in Autonomous Systems

    The rapid adoption of AI agents in enterprise environments has created a new challenge: governance. As organizations deploy increasingly autonomous systems, the question is no longer just about what these agents can do, but how to ensure they operate within acceptable boundaries.

    This isn’t a theoretical concern. Companies are already facing real-world incidents where AI agents have made decisions that, while technically correct, violated business policies or ethical standards.

    The Governance Gap

    The traditional model of software governance — where humans review every line of code and every decision — breaks down when dealing with autonomous agents. These systems can make thousands of decisions per minute, each one potentially impacting business operations.

    The governance challenge has three core dimensions:

    • Decision Transparency: Unlike traditional software, AI agents often make decisions based on complex reasoning that’s difficult to trace. When an agent denies a loan application or prioritizes one customer over another, stakeholders need to understand why.
    • Policy Enforcement: Business policies that were designed for human decision-making need to be translated into constraints that AI agents can understand and follow. This requires a new layer of policy engineering.
    • Accountability Framework: When an autonomous agent makes a mistake, who is responsible? The developer who trained it? The business owner who deployed it? The compliance team who approved it?

    Building Effective Governance

    Organizations that are successfully managing AI agent risk have adopted a three-pronged approach:

    • Guardrail Architecture: Instead of trying to control every decision, they create hard boundaries that agents cannot cross. This includes data access limits, decision thresholds, and explicit “forbidden actions.”
    • Continuous Monitoring: Real-time monitoring systems track agent decisions and flag anomalies. This isn’t just about catching mistakes — it’s about identifying patterns that might indicate systemic issues.
    • Human-in-the-Loop: Critical decisions still involve human review. The key is determining which decisions require human oversight and which can be safely automated.

    The Business Case for Governance

    Investing in AI agent governance isn’t just about risk mitigation — it’s about enabling innovation. Organizations that lack proper governance frameworks often find themselves unable to deploy AI agents in high-stakes scenarios due to regulatory uncertainty or reputational risk.

    Conversely, companies with mature governance frameworks can move faster because they have the confidence to deploy agents in mission-critical applications. They’ve already answered the hard questions about accountability, transparency, and control.

    The governance challenge is fundamentally about trust. Stakeholders — whether they’re customers, regulators, or board members — need to trust that AI agents will operate within acceptable bounds.

    How is your organization approaching AI agent governance? Are you treating it as a compliance requirement or as an enabler for innovation?

  • The Agentic Workflow: How AI is Changing Product Requirements

    The Product Requirement Document has been the backbone of product management for years. It tells engineering exactly what to build. But that model is breaking under the weight of AI-driven development.

    We are moving toward agentic workflows. Agents don’t read specs and wait for clarification. They take a directive, interpret it, and start building. For product teams, this fundamentally changes what a “requirement” even means.

    Instead of a 40-page document, requirements become a set of constraints and success criteria. The PM’s job shifts from writing specs to defining the logic the agent follows.

    Constraint-Based Requirements

    In a traditional workflow, the PM details every user story, edge case, and UI state. That level of granularity was necessary because developer time was expensive and misalignment was costly. Agents flip that cost equation. It is now cheaper to iterate on a high-level directive than to document every step in advance.

    The requirement is no longer a step-by-step instruction. It becomes a boundary.

    • Success metrics over user stories: Instead of “Add a filter dropdown,” the directive is “Users must be able to narrow results to under 50 items with two clicks.” The agent figures out the implementation.
    • Rapid prototyping: Agents can generate working drafts or code skeletons in minutes. PMs validate against the output rather than a theoretical spec, turning discovery into a feedback loop.
    • Technical and persona guardrails: The agent needs rules. “Must use existing API,” “Must comply with WCAG 2.1,” “Target audience: enterprise admins.” These constraints keep the agent’s output aligned with reality.

    From Writer to Orchestrator

    This transition moves the product manager away from documentation and toward system management. The value is no longer in how well you write a spec, but in how effectively you coordinate the agents that execute it.

    Three responsibilities become central:

    • Strategic direction: Agents optimize for what they’re told. They don’t know about the Q3 revenue target or the recent customer churn spike. The PM provides the business context that prevents local optimization.
    • Governance: Autonomous systems need hard limits. PMs define the non-negotiables—data privacy boundaries, brand standards, compliance requirements. The agent handles the rest.
    • Human alignment: An agent can draft a feature, but it can’t negotiate with engineering on technical debt or align with sales on a launch timeline. That human coordination is still a PM’s core responsibility.

    The Friction Is Real

    Adopting this workflow is not trivial. Data security is the first hurdle; teams are understandably cautious about feeding roadmaps into external models. Then there’s reliability. Agents hallucinate. They misinterpret nuance. They produce confident but incorrect outputs.

    The practical approach is hybrid. Use agents for the heavy lifting of documentation, test case generation, and initial prototyping. Keep human review before anything reaches production.

    Teams that do this well report significantly shorter cycles from concept to working software. But it requires a new level of discipline. The spec isn’t gone—it’s just executable now.

    How is your team approaching this? Are you using AI to accelerate the discovery phase, or are you still keeping it strictly out of the requirements process?

Powered by
►
Necessary cookies enable essential site features like secure log-ins and consent preference adjustments. They do not store personal data.
None
►
Functional cookies support features like content sharing on social media, collecting feedback, and enabling third-party tools.
None
►
Analytical cookies track visitor interactions, providing insights on metrics like visitor count, bounce rate, and traffic sources.
None
►
Advertisement cookies deliver personalized ads based on your previous visits and analyze the effectiveness of ad campaigns.
None
Powered by