GPT-4, the model that passed the bar exam, was given a straightforward job: plan a multi-day trip with flights, hotels, and restaurants within a budget. It succeeded 0.6% of the time. Not 60%. Not 6%. Zero point six. Meanwhile, a system that used the same LLM differently — as a translator rather than a decision-maker — hit 97%. That gap isn't about model intelligence. It's about architecture. And it's the reason most "AI agents" being shipped today are destined to break in production.
I've spent the last two years building AI systems that interact with some of the most unforgiving APIs on the planet — Global Distribution Systems for flight booking, built in the mainframe era, where a single misformatted field kills the entire transaction. That experience taught me something the AI hype cycle keeps ignoring: the hard part of building AI agents isn't making them smart. It's making them reliable.
The Wrapper Delusion
The AI industry has a naming problem. We started calling everything an "agent" — AutoGPT, BabyAGI, any LLM with access to a few API tools. The pitch is seductive: give a powerful language model a goal and some tools, and it'll figure out the steps on its own.
It doesn't work.
Language models predict the next most likely word. That's their entire mechanism. When you're writing an email or summarizing a document, "most likely" and "correct" overlap almost perfectly. But when you're executing a ten-step business workflow — where step 7 depends on the exact output of step 3 — "most likely" and "correct" diverge fast.
I call this the Wrapper Delusion: the belief that you can wrap a probabilistic text generator in a thin layer of tools and prompts and get a reliable business system. My team has watched it fail over and over — agents stuck in infinite retry loops, agents that confidently report a booking was made when no transaction ever occurred, agents that "autocorrect" an API field and silently break everything downstream.
The hard problem of AI agents isn't intelligence. It's that a 90% accurate model run across ten steps gives you 34% reliability.
That math is simple compounding. If your model makes the right call 90% of the time (generous for complex reasoning), then across ten sequential steps you get 0.9¹⁰ ≈ 34%. A real flight booking workflow often exceeds ten steps. You're below coin-flip territory before you've even reached payment.
What the TravelPlanner Benchmark Actually Proved
The TravelPlanner benchmark is the most revealing stress test I've seen for AI agents. Researchers asked language models to plan U.S. travel itineraries — coordinating flights, hotels, restaurants, and budgets across multiple days. The constraints are rigid: stay under budget, don't book a dinner reservation in a city you haven't flown to yet, don't schedule a hotel checkout after your flight departs.
GPT-4, running as a pure LLM agent, achieved a 0.6% overall success rate. Its hard constraint pass rate — meaning it satisfied all the non-negotiable rules — was roughly 4.4%. It delivered something 93% of the time, which is the dangerous part: it looked like it was working. The itineraries read beautifully. They just violated the actual constraints.
The system that reached 97% didn't use a better model. It used the same kind of LLM for a narrower job — parsing the user's request into structured data — and then handed that data to a deterministic solver. An algorithm, not a language model, enforced the budget. Code, not token prediction, ensured the hotel was in the right city on the right night.
We explored this architecture in depth in our interactive analysis, but the core insight fits in one sentence: treat the LLM as a translator, not a planner.
Why Smart Models Make Dumb Mistakes
When I first saw the TravelPlanner results, I assumed the failures would be obvious — wrong cities, absurd dates. They weren't. The failures were subtle and cascading, which made them far more dangerous.
Context drift is the first culprit. As the model works through a multi-step plan, its context window fills with intermediate data — flight options, hotel listings, pricing details. The attention mechanism, which decides what information to focus on, spreads thinner with every step. By step ten, the model has effectively "forgotten" the budget constraint it correctly identified in step two. The information is technically still in the window. The model just can't prioritize it anymore.
Then there's what we call the hallucination cascade. The model misreads a flight arrival time — 2:00 AM becomes 2:00 PM. It books a hotel check-in based on the wrong time. The API accepts the request because the API doesn't know intent, only input. The model sees a successful response and reinforces its own error. Every subsequent step builds on a foundation that's already cracked.
An AI agent that fails loudly is annoying. An AI agent that fails silently — reporting success while booking the wrong flight — is dangerous.
The most unsettling pattern is what researchers call reasoning-action mismatch. The model's internal reasoning correctly identifies a constraint — "I need a flight under $500" — and then its very next action calls an API for a $600 flight. The reasoning was right. The execution was wrong. Using text generation as a stand-in for logic execution is like using a poet to do your accounting. The words might be beautiful, but the numbers won't add up.
What Breaks When You Hit a Real API
My team builds against Global Distribution Systems — Sabre, Amadeus, Travelport. These are the backbone of airline booking worldwide, and they were designed in an era when "user-friendly" meant "the mainframe operator knows the three-letter command codes."
A GDS flight booking is a strict state machine. Think of it like a combination lock: you must enter the right numbers in the right order, or nothing opens. Authenticate first. Search for flights. Price the selected offer. Create a Passenger Name Record by adding segments, then passenger names (formatted exactly as LAST/FIRST MR), then contact info, then a ticketing time limit, then a "received from" field. Only then can you commit.
Skip a step? Error. Reorder two steps? Error. Change a date format from what the search returned? Error. The GDS doesn't care about your intent. It cares about your input, down to the byte.
I watched one of our early prototypes — before we rebuilt the architecture — attempt to commit a booking before adding the "received from" field. The GDS returned ERR 1209 - SEQUENCE ERROR. The LLM, trained to be helpful, interpreted this as a temporary glitch and retried the exact same request. Eleven times. Burning through tokens and API rate limits, banging against a wall it couldn't understand, because the error message gave it no semantic clue about what went wrong.
That was the moment I knew: control flow is not a language task. Deciding what step comes next in a rigid process shouldn't be a matter of token prediction. It should be an if-then statement in code.
The Architecture That Actually Works
We call our approach Neuro-Symbolic Orchestration, and the core idea is simple enough to sketch on a napkin. You take the two great traditions of AI — neural networks (pattern recognition, language understanding) and symbolic logic (rules, sequences, arithmetic) — and you assign each to the job it's actually good at.
The language model handles perception. It's the interface layer. When a customer says "I need to fly from London to New York next Tuesday, nothing too early," the LLM parses that into structured data: origin LHR, destination JFK, date 2025-01-14, preference no departures before 9 AM. That's what LLMs are brilliant at — understanding the mess of human language and converting it into clean data.
The graph handles execution. It's the logic layer. A hard-coded state machine — built with a framework called LangGraph — takes that structured data and runs the booking workflow step by step. The graph decides what happens next, not the LLM. The graph enforces that you can't attempt payment before selecting a flight, because the transition literally doesn't exist in the code until the selected_offer variable is populated.
We demoted the LLM from CEO to task worker. Code manages the workflow. The model fills in the blanks.
When the GDS returns a cryptic error code, a hard-coded error handler maps it to a specific recovery strategy. UC (Unable to Confirm) triggers a re-shop workflow. No LLM interpretation needed. No retry loops. No hallucinated fixes. The model is bypassed entirely during error recovery.
For the full technical methodology behind this architecture, see our detailed research.
Where Humans Stay in the Loop
One thing I've learned building enterprise AI: the goal is never full autonomy. It's augmented productivity — making people faster and more accurate, not replacing their judgment where it matters.
Our architecture has built-in "airgaps." When a flight exceeds a corporate policy limit, the graph doesn't ask the LLM what to do. It freezes execution, saves the complete state to a database, and sends an approval request to a manager. When the manager approves, the system reloads the exact state and resumes from where it paused. No re-reading chat history. No re-inferring context. The state is structured data in a database, not tokens in a context window.
This also solves the compliance problem that keeps enterprise legal teams up at night. Every node in our graph produces a log entry: what input it received, what rule it applied, what output it produced. When an auditor asks "why did the system book this flight?", we can show them a deterministic chain of logic — not a blob of token probabilities.
The Cost Nobody Talks About
Beyond reliability, there's a pure dollars-and-cents argument. When an LLM agent gets stuck in a retry loop against a GDS error it can't interpret, it generates thousands of tokens before timing out. A single stuck session can cost $5-10 in API credits and produce nothing.
Our architecture cuts this waste in two ways. First, hard-coded error handlers catch failures instantly — zero tokens spent on futile retries. Second, we never feed the LLM a raw 50-kilobyte GDS response. The code layer parses the JSON, extracts the five relevant fields, and passes only those to the model for summarization. That reduces context window usage by roughly 90%, which means lower inference costs and faster response times on every single interaction.
But What About Better Models?
The most common pushback I hear: "Won't GPT-5 or GPT-6 just solve this?" Maybe the success rate climbs from 0.6% to 15% or even 40%. But compounding probability across ten steps is a mathematical reality, not a model limitation. A 95%-accurate model across ten steps still gives you 60% reliability. For a system processing thousands of bookings a day, that's hundreds of daily failures.
And there's a deeper issue. Even if a future model reaches 99% per-step accuracy, you still can't audit it. You can't prove to a regulator why it made a specific decision. You can't freeze its state and resume it three hours later when a manager approves an exception. These aren't intelligence problems. They're engineering problems. And they require engineering solutions.
What This Means If You're Building AI Agents
If you're deploying AI into any workflow that touches real transactions — bookings, payments, inventory, compliance — ask yourself one question: who decides what step comes next? If the answer is "the language model," you're building on sand.
The shift isn't complicated conceptually. Identify every decision point in your workflow that has a deterministic answer — "if payment confirmed, then issue ticket" — and move it into code. Let the LLM handle the genuinely ambiguous parts: understanding what the user wants, summarizing complex data, resolving "the second one" into a specific offer ID. That's the work language models were built for. Everything else should be software.
The companies that figure this out first won't just have more reliable AI. They'll have AI they can actually trust with their customers, their money, and their reputation.
I'd love to hear from anyone wrestling with this in production — what's breaking, what's working, and where the line between "smart enough" and "reliable enough" falls for your use case.