A team builds a dazzling prototype in a week. It answers questions, books meetings, drafts replies, and everyone in the room nods. Then it meets real customers, real data, and real consequences — and it quietly dies.

This is not a model problem. The model was never the hard part.

MIT's 2025 report The GenAI Divide: State of AI in Business found that 95% of generative AI pilots fail to deliver measurable profit-and-loss impact. Read that again: not 95% of bad ideas, but 95% of pilots — including many built by competent teams with good models. The gap between a demo and a production system is where almost every agent project is won or lost.

Here is what that gap looks like, and how to survive it.

The demo is not the product

A demo runs on clean data, a friendly user, and a forgiving audience. Production runs on none of those.

In a demo, the input is the one happy-path example you rehearsed. In production, it is a form field full of typos, a CRM record missing half its fields, and an API that fails at 2 a.m. A wrong answer in a demo is a laugh; in production it is a refund issued twice or an email sent to the wrong client. Demos do not need error handling. Production is mostly error handling.

That is the first uncomfortable truth: the model is perhaps 10% of a working automation. The rest is the boring engineering nobody posts about — observability, retry logic, guardrails, a data layer that does not lie, and a human who can hit stop.

Why prototypes die: five production gaps

1. No permission boundaries

A demo agent usually runs with your credentials, or with a single admin token, because that is the fastest way to make it work. In production, that is a liability with an API key.

Every tool an agent can call is a door. A production agent needs scoped, least-privilege credentials per task, and an explicit list of actions it is never allowed to take without a human. Before you ship, answer this precisely: if this agent went rogue for ten minutes, what is the worst thing it could do?

2. Messy data, not model quality

Demos run on curated examples. Production data is duplicated, stale, half-empty, and contradictory. The same customer exists three times in the CRM under three spellings of their name. The price list has two versions. The calendar has a cancelled meeting that was never deleted.

An agent that trusts its data blindly will confidently act on lies. A production system validates before it acts. This is why so many agent projects stall at integration, not intelligence. The model was fine. The data layer was not.

3. No observability

When a workflow breaks, you can point at the exact step that failed. When an agent breaks, you often get a confident answer and no explanation of how it got there. Production agents need logs for everything — goal, tools called, arguments, outputs, and final action — so a run can be replayed step by step. When a customer asks why something happened, "the agent decided" is not an answer.

4. Cost and drift nobody budgeted for

A workflow costs roughly the same every time it runs. An agent does not. Token usage varies with every run, tool calls multiply, and a retry loop can turn a cheap task into an expensive one. Behaviour also drifts: the same prompt that worked in October can behave differently after a model update in November.

Production means budgets per run, capped retries, and regression tests run on a schedule against the live system. If you would not ship a normal feature without tests, do not ship an agent without them either.

5. No human at the money step

The most dangerous agent is one that can move money, message customers, or change records with no human in the loop — not because agents are malicious, but because they are confident. The rule I use with clients is simple: AI can prepare, draft, and recommend, but a human approves anything irreversible.

The architecture that survives: deterministic backbone, AI at the edges

The agents that reach production almost all look the same, and they do not look like the demos. They have a deterministic backbone — a workflow with defined steps, validations, and fallbacks — and they use AI only where judgement is actually needed.

Take a lead-handling automation. Four of its five steps are deterministic: take the form lead, enrich it, check it against rules, and create the deal in the CRM. Only one step needs judgement: scoring the lead. That is a workflow with one language-model call in the middle — not an agent problem.

Here is the shape of that pattern in code:

def handle_lead(lead):
    # Deterministic steps: same input, same output, every time
    lead = validate(lead)          # reject bad data early, loudly
    lead = enrich(lead)            # one known source, not a web search
    if not passes_rules(lead):     # business rules, not model vibes
        return queue_for_review(lead, reason="failed rules")

    # AI only where judgement is needed
    score = score_lead_with_llm(lead)

    # Human at the money step: nothing irreversible happens unseen
    if score >= 80:
        return request_approval(lead, score, action="create deal + notify sales")
    return queue_for_review(lead, reason="low score")

And the guardrails around it are explicit, not implied:

{
  "agent_policy": {
    "allowed_tools": ["crm.read", "crm.create_deal"],
    "forbidden_tools": ["email.send", "payment.refund"],
    "max_retries": 2,
    "max_cost_per_run_usd": 0.50,
    "requires_human_approval": ["create_deal", "send_to_customer"],
    "log_everything": true
  }
}

Notice what this buys you. When it breaks, you know which step broke. When it is wrong, a human saw it before the customer did. That is not less ambitious than a fully autonomous agent. It is the version that still exists in six months.

A production checklist before you launch

Run through this before any agent touches real customers:

• Permissions: scoped credentials, an explicit forbidden-action list, and a tested answer to "what is the worst ten minutes of rogue behaviour?"
• Validation: every input checked, every tool output sanity-checked before the next step runs.
• Observability: full run logs you can replay, plus alerts when a run fails, loops, or exceeds its cost budget.
• Retries and fallbacks: capped retries with backoff, and a graceful fallback — queue for a human — instead of retrying forever.
• Human approval: a person approves anything irreversible, especially anything involving money or customer-facing messages.
• Regression tests: your ten most important tasks, run on a schedule against the live system, with failures alerting a human.

If you cannot tick all six, you do not have a production agent yet. You have a demo with a launch date. Fix the checklist first — it is cheaper than explaining a failure to a customer later.

Stop asking which model to use

The model question is the easiest question in agent building, and the least important. Swap the model in a well-built system and the system still works. Start instead with the question that decides whether your agent survives contact with reality: what happens when it fails? Build the boring backbone first. Earn the agent later.

I break down systems like this daily — follow for more.

Traffic was always a proxy. It stood in for attention, then for interest, then — hopefully — for revenue. AI search has broken that chain in an interesting way: it sends fewer visitors, but the visitors it does send are much closer to buying. If your dashboard only counts sessions, AI search looks like a disaster. If it counts signups, it looks like the best channel you have.

The numbers that break the old dashboard

Ahrefs shared their own data: AI search accounted for just 0.5% of their traffic, but 12.1% of their signups. That is not a rounding error. It is a completely different ratio from organic search, where a large volume of visits produces a small share of conversions.

An Opollo study across 312 IT firms points the same way: AI referrals converted at 14.2%, against 2.8% for Google organic. Again, the story is not volume. It is intent. By the time an AI answer sends someone to your site, that person has usually already had the comparison, the objection handling, and the shortlisting done for them — inside the answer.

And Ethan Smith's analysis adds the part most teams miss: roughly 45 billion monthly AI sessions, with about 83% happening inside apps — ChatGPT, Gemini, Perplexity, Claude — rather than on the open web. Most AI activity is not happening on websites at all. Your site is not the destination any more. It is the source the answer was built from, and occasionally the place the buyer lands to finish the job.

Put those three data points together and the picture changes. The funnel did not disappear. It moved upstream, into the answer itself. The click is no longer the start of the buying journey. It is near the end of it.

Why an AI referral converts differently

A Google click is often the start of research. Someone types a broad query, opens five tabs, and bounces from four of them. An AI referral is different because the filtering already happened before the visit.

The AI has already:

• Understood the question in full sentences, not keywords
• Compared options against the buyer's actual criteria
• Answered the obvious objections in the answer itself
• Recommended a shortlist, with reasons

So the visitor who arrives is pre-qualified in a way a search click rarely is. They are not asking "what is this?" They are asking "is this the one, and how do I start?" That is why a tiny share of traffic can produce an outsized share of signups. The denominator shrank. The numerator did not shrink nearly as much.

This also explains why the usual SEO panic is misplaced. Yes, AI answers will eat some informational traffic — the "what is X" visits that never converted anyway. Losing those visits hurts a traffic chart. It does not necessarily hurt revenue. The visits worth keeping are the ones attached to a buying question, and those are exactly the visits AI answers are now routing.

Stop optimising for clicks. Start optimising for the answer.

If the click is now the end of the journey, the work moves earlier: into being the source the answer quotes, and into being the obvious next step once it does.

1. Say what you do in one quotable paragraph

AI answers quote pages that state a claim plainly, with evidence on the same page. A homepage that says "we help businesses grow with innovative solutions" gives an answer engine nothing to quote — because it says nothing specific. A page that says who you serve, what problem you solve, and how you are different, in the first paragraph, gives it a sentence it can lift whole.

Write the paragraph you would want quoted. Then check whether it actually appears on the page, in crawlable HTML, not buried in a JavaScript widget.

2. Answer the buyer's real question, not the keyword

Ranking content targets a keyword. Answer content targets a question a buyer actually asks, in the words they ask it: "which tool is best for a small team on a budget?", "what does it cost after the first year?", "what breaks when you scale past 10,000 users?" Pages built around those questions — with a direct answer first and the detail after — are the pages AI systems reach for.

3. Make the conversion step obvious from the answer

If an AI answer sends a buyer to your pricing page, that page has to finish the job: clear pricing, a visible next step, and proof — testimonials, case results, a comparison — on the same page. A buyer who arrives pre-sold and then has to hunt for a signup button is a buyer you will lose to a competitor whose page does not make them hunt.

4. Measure signups per AI referral, not AI sessions

This is the dashboard change that matters most. Split your analytics so AI referrals are a separate source, and track them all the way to signup, not just to session. A simple way to start is to segment referral traffic from the major AI platforms and compare its signup rate against organic search over the same period:

// Segment AI referrals in your analytics by referrer
const aiReferrers = ["chatgpt.com", "perplexity.ai", "claude.ai", "gemini.google.com"];
const isAiReferral = aiReferrers.some((domain) =>
  document.referrer.includes(domain)
);
// Tag the session, then compare signup rate: AI referrals vs organic

The point is not the code. It is the comparison. Once you can see that one AI referral is worth several organic visits, the internal argument about "losing traffic" ends, because the team is finally looking at the number that pays.

Ranking still matters — but the finish line moved

None of this means SEO is dead. Ranking still matters, for two reasons. First, the pages AI systems trust are usually the pages that already earned authority the old-fashioned way: consistent content, real expertise, third-party mentions. Second, not every buyer journey runs through an AI answer yet. Search still drives discovery for a huge share of queries.

What changed is the finish line. It used to be the most traffic. Now it is the most-quoted answer on the question buyers actually ask — followed by a page that converts the buyer who arrives. Traffic without that second half is just a chart going up. A cited answer with a working signup path is a business.

So stop asking whether SEO or GEO wins. Ask a better question: when an AI answer mentions your category, does it mention you — and when it sends you a buyer, does your page close?

That is the game now. The dashboards just have not caught up yet.

Then they asked ChatGPT what the site was about.

It had no idea the site existed. Not a wrong answer — no answer at all.

That's the quiet failure mode of the vibe-coding boom. These tools didn't just change who builds websites. They quietly changed what gets indexed. And most vibe-built sites ship on the one rendering model that makes them invisible to AI search.

The vibe stack defaults to client-side rendering

Most AI builders — Lovable, v0, Bolt, Replit's agent mode — scaffold React/Vite-style single-page apps. Your entire page is assembled in the browser at runtime. The HTML the server actually sends looks like this:

<!doctype html>
<html>
  <head>
    <title>Loading...</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/main.tsx"></script>
  </body>
</html>

That is the entire page. A few hundred bytes of "we'll get back to you." In your browser, JavaScript fills it in within milliseconds. To a crawler, it is a blank wall with a polite title.

Why bots don't see what you see

Fetching HTML is cheap. Executing JavaScript is expensive — a full browser context, memory, timeouts, multiplied across billions of pages. Crawlers operate on budgets, and those budgets are spent on the highest-value work first.

Google invests in a rendering pipeline for its index, and even that is queued and delayed — JS-rendered content routinely enters the index days behind the HTML version. AI crawlers — GPTBot, ClaudeBot, the bots that decide whether ChatGPT or Claude ever cite your site — run leaner still. The widely observed behaviour: they fetch the raw HTML and move on. No JavaScript execution, no hydration, no content.

So the question that matters is no longer "does my site look good?" It is: what does my site look like with JavaScript switched off? If the answer is an empty div, then as far as AI search is concerned, your content does not exist. There is nothing to cite, quote, or recommend — no matter how beautiful it renders in Chrome.

This is exactly why the new SEO conversation has shifted from rankings to citations. An AI answer engine doesn't browse your site like a visitor. It reads what its crawler could read, and it quotes what it could understand. An empty HTML shell gives it nothing to work with.

The 60-second test

curl is the most honest reviewer your site will ever meet. It shows you exactly what a no-JS bot sees:

curl -s https://your-site.com | wc -c
curl -s https://your-site.com | head -c 600

If the byte count is in the low hundreds and the output is a shell with an empty root div, your page is invisible to AI search. Try it right now — the result takes five seconds and has ended more than a few launch-day celebrations.

For comparison, run the same test on a server-rendered page and watch kilobytes of real headings, paragraphs, and links stream past. That is the difference between being quotable and being a loading spinner.

The fix — you don't have to give up the vibe

Vibe-coding the content is fine. Shipping the rendering client-side is the problem. The fixes are all on the delivery layer:

1. Server-render or statically generate. Frameworks that ship HTML on first byte — Next.js (App Router), Astro, Nuxt, SvelteKit with prerendering — give every bot the full page immediately. The browser still hydrates into the interactive app your users see; the crawler never needed the interactivity.

2. Prerender as a bridge. If you're locked into a client-side build, prerendering services generate static HTML snapshots of your routes and serve them to crawlers. It is not as clean as true SSR, but it converts an invisible page into a readable one in an afternoon.

3. Prioritise the pages AI needs to quote. Your homepage, pricing, and services pages are the ones answer engines will cite. A blog post buried three clicks deep matters less than the page that says, in plain words, what you do. Make sure those pages render as HTML.

4. Don't rely on "Google renders JavaScript" as your safety net. Even Google's rendering is budgeted and delayed — and AI crawlers are stricter still. "It'll probably get rendered eventually" is not a strategy; it is a hope.

Ship HTML, not apps

The irony is sharp: the tools that made building websites effortless quietly rebuilt the exact problem the industry spent a decade solving. We fought the client-side-rendering wars in the 2010s, learned that HTML-first wins for discoverability, and then vibe-coding reintroduced the SPA as the default — this time to people who never saw the first war.

So here's the practical checklist before your next launch:

• Build with the vibe tools for speed, render like it's 2009 for the bots: HTML first, JavaScript as enhancement.
• curl every launch. If a page serves under a couple of kilobytes of real text content, fix the rendering before you announce it.
• Say the plain truth in text. A single clear paragraph describing what you do beats a clever animation that no bot will ever watch. AI search rewards clarity, not cleverness.

Your site can be both stunning and visible. It just has to arrive as a page, not a promise.

I break down systems like this daily — follow for more on web architecture, AI automation, and getting cited by the machines that now decide what gets read.