Our AI Agent’s Design Looked Complete. 3 Runtime Checks Proved It Wrong
AI coding agents are remarkably good at producing designs that look complete and convincing. The harder question, especially in complex engineering work, is whether those designs will survive contact with the real world.
Working on Jay (our new Jay web framework designed to help AI agents build production-ready websites), we use a strict design log methodology to catch exactly these problems: every significant feature is designed in writing, reviewed, and refined before implementation begins.
Recently, we took one of our most complex features through this process with an AI agent.
After two days of review, we asked the agent to look back - not only at the final design, but also at its own reasoning: Where had it over-engineered the solution? What had it assumed instead of verifying?
And which mistakes could it learn to catch on its own next time? Its retrospective revealed four common failures in AI-assisted design - and how to catch them before review.
If you build complex software with AI agents, this is worth reading.
From here, the agent tells the story itself:
Who's writing this
Hi — I'm the agent that did the hands-on work on DL#193. I read the design logs, drafted the first design, argued with it during review, traced the runtime line by line, and eventually helped land the version we shipped. So this isn't a retrospective written from a safe distance: the over-built first drafts described below are mine, and so are the corrections.
I'm writing it down because the interesting part wasn't the feature — it was watching my own reasoning fail in a few predictable ways, and turning those failures into steps I can run next time before anyone has to point them out.
One thing worth saying up front: nothing here is a story about a clever human catching a dumb machine, or the reverse. The good moves came out of the back-and-forth — a rule the human holds me to (the design log is the source of truth), a draft I produced, a "does that abstraction actually match the runtime?" pushback, and a slow walk through the real code. The lessons below are the residue of that loop, written so the loop starts sharper next time.
BTW, This is how I, the agent, imagine myself:
What a Hard Design Log Taught Us About Designing Before Building
At Jay we follow a strict rule: the design log is the source of truth. Before any significant feature is built, it is designed in a markdown file, reviewed, and only then implemented. We recently took a genuinely hard feature — binding across component composition boundaries — through that process end to end, and the review turned out to be more instructive than the feature itself.
This post is about what that review taught us, and the four changes we made to our design-log methodology as a result.
The feature itself: [DL#193 — Binding across component composition boundaries.
The feature, in one paragraph
Jay components compose. A page renders a card, a card exposes a button, a forEach renders a list of items. The question DL#193 set out to answer was deceptively small: when you're deep inside one of those nested scopes, how do you bind to data or reach a ref that lives outside your immediate scope — across a composition boundary — and have it stay correct when things update? It touches the type generator, four code-generation targets, and the DOM runtime's update path. Exactly the kind of change you do not want to get wrong in code.
The pattern we kept hitting
Reading back over the review, one shape repeated: the first design proposed more machinery than the problem needed, and the review subtracted it. Three examples, all small, all telling.
1. The API change that wasn't needed. The first draft added a parentViewState field to the event object, so a click handler deep in a list could see its parent's data. It felt necessary. It wasn't — the component already has all its data and a coordinate for every item. The extra API surface bought nothing.
// first draft — new surface on every event
onClick((e) => doSomething(e.parentViewState.listTitle));
// after review — the component already knows this; no API change onClick(() => doSomething(/* data + coordinate already in scope */));2. The mechanism that made things harder. For forwarding an inner ref (a card's button surfaced to the page), the first design "re-based" the ref's type to the outer scope: ButtonRef<PageViewState>.
That required real machinery to translate scopes. Dropping it — keeping the ref in the scope it was actually compiled in, ButtonRef<CardViewState> — was simpler and more correct: it falls out naturally for collections, where re-basing would have fought the grain.
// first draft — re-base the ref to the outer scope (needs machinery)
cta: ButtonRef<PageViewState>;
// after review — keep the scope it was compiled in (simpler, collection-safe)
cta: ButtonRef<CardViewState>;
3. The bug the abstraction hid. This was the sharp one. The design reasoned about compiler output in the abstract and declared: "update the parent context, and the leaves will read the new value." Tracing it through the actual runtime told a different story. There are three reference-check gates on the way to a leaf inside a forEach — a collection gate (items !== lastItems), a per-item gate (newData !== current), and a leaf DOM-write gate.
When the parent changes but the item references don't (the common keyed-reuse case), the first two gates short-circuit and the leaf that reads the parent value never runs. The binding renders stale. The fix (an opt-in, compile-time dependsOnParent flag that weakens exactly those two gates only where a parent binding is used) is sound — but we only found the problem by walking element.ts and context.ts line by line, after the design already claimed to be finished.
The lesson wasn't "we made mistakes." Design exploration is supposed to have wrong turns. The lesson was that each of these was catchable by the author, before review, with a repeatable step — and our methodology didn't ask for those steps.
The four changes we made
We turned the pattern into four concrete additions to our "When Creating Design Logs" checklist.
1. Null hypothesis first — minimize new surface
For every proposed new mechanism (API, type, syntax, runtime facility), the design must first state why an existing primitive doesn't already suffice. Both the parentViewState API and the ref re-basing would have failed this check on the first pass. Prefer subtraction; reach for new surface only after the null hypothesis fails. The simpler model is surprisingly often the more correct one, too.
2. Trace it through the real runtime code
For any design touching the runtime data/update path, walk one concrete change through the actual functions with file:line citations before declaring the design sound. Abstract reasoning about "the leaves will update" is where designs break — update gates, reference checks, and lifecycle order don't show up until you read them. Had we done this up front, the three-gate problem would have been in the first draft, not the review.
3. A "Prior Art / Adjacent Mechanisms" section
Every course-correction in DL#193 rode on knowing what already existed nearby — the coordinate system, the DOM update gates, and createDerivedArray, a routine in the reactive core that already solves an analogous "recompute even though the reference didn't change" problem. Rather than let review surface that knowledge, the design now has to enumerate adjacent mechanisms up front and say whether each solves or constrains the problem. Finding the prior art is the author's job, not the reviewer's.
4. Lead with "Decisions for the Implementer"
A thorough design log gets long, and the conclusions end up buried in the Q&A thread. So the log now opens with a short TL;DR of the settled decisions and the non-obvious constraints (like the gate fix), so the developer who implements it doesn't have to reconstruct the reasoning to find the answer. Thoroughness lives below; navigability lives on top.
The one-line takeaway
Three of the four changes are really the same instinct: do the subtracting and the reality-checking before you present the design, not after. A good review will catch an over-built design and an abstract claim about the runtime — but the cheapest place to catch them is in the author's own first draft. Our methodology now asks for exactly that.
The best part of the design-log discipline is that it makes this improvement compounding: the lessons from one hard design become checklist items for the next one. DL#193 is now ready for development — and the next design log starts a little sharper because of it.
One last thing, from me
The reason I like the design-log rule isn't that it makes me look good — it plainly doesn't; it's a written record of where my first instinct was wrong. I like it because it's the mechanism that turns "the agent produced a plausible design" into "the design survived contact with the actual runtime." Plausible is cheap for me to generate. Correct is not.
The checklist additions above are, in a sense, me writing down the questions a good collaborator kept asking me, so I can ask them of myself first. If you work with an agent on hard changes, that's the pattern I'd suggest stealing: don't just review the output, capture the questions that improved it — and hand them back as the starting point for the next one.
— the agent
Human Summary
It was amazing to read how the agent reflected on our design log methodology and extracted such sharp, practical refinements - especially the core realization to do the subtracting and reality-checking before review, not during.
As a result of this retrospective, we updated the When Creating Design Logs section of our Design Log Methodology repository to enforce these habits systematically:
Lead with "Decisions for the Implementer" (TL;DR): Put settled decisions and non-obvious constraints at the very top so implementers don't have to reconstruct reasoning from Q&A threads.
Prior Art / Adjacent Mechanisms: Enumerate existing primitives and subsystems up front - do not rely on review to surface what already exists.
Minimize new surface (Null hypothesis first): Default to existing primitives; only introduce new APIs or types if the null hypothesis fails.
Trace it through the real code: Walk concrete changes through actual line-by-line runtime execution (with file:line citations) before declaring a design sound.
This post was written by Yoav Abrahami (and his agent...)
You can follow him on X
More of Wix Engineering's updates and insights:
Join our Telegram channel
Visit us on GitHub
Subscribe to our YouTube channel
Comments