{"id":27947,"date":"2026-07-25T18:56:06","date_gmt":"2026-07-25T13:26:06","guid":{"rendered":"https:\/\/deepubalan.com\/blog\/?p=27947"},"modified":"2026-07-25T18:56:16","modified_gmt":"2026-07-25T13:26:16","slug":"human-in-the-loop-isnt-just-about-trust-its-about-error-propagation","status":"publish","type":"post","link":"https:\/\/deepubalan.com\/blog\/2026\/07\/25\/human-in-the-loop-isnt-just-about-trust-its-about-error-propagation\/","title":{"rendered":"Human-in-the-loop isn’t just about trust. It’s about error propagation"},"content":{"rendered":"\n

Lessons from building a multi-agent AI system, and how I figured out where to actually trust the AI and where not to.<\/em><\/p>\n\n\n\n

I keep running into this question lately, in one form or another: how much should I trust an AI agent to just go and do the work without me checking it first?<\/p>\n\n\n\n

It shows up everywhere right now. Should an agent send an email on your behalf without you reading it first? Should it merge code on its own? For me, the question came up while building something bigger than a single task. I was putting together a system of AI agents meant to design and build reusable UI components for our component library, with very little manual work in between. So I had to actually sit down and think this through properly instead of guessing.<\/p>\n\n\n\n

#<\/strong><\/a> #<\/strong><\/a> #<\/strong><\/a> #<\/strong><\/a> #<\/strong><\/a><\/p>\n\n\n\n

Most people I talk to land on one of two answers. Either “AI isn’t reliable enough, so a human needs to check everything,<\/em>” which quietly means you haven’t really automated much. Or “the models are good enough now, just let it run,<\/em>” which is fine right up until the one time it isn’t, and by then the mistake is already a few steps downstream and harder to find.<\/p>\n\n\n\n

\"Article<\/figure>\n\n\n\n

Our component library isn’t a small thing. It ranges from simple components like buttons all the way up to something like a fully featured data table with sorting, filtering, inline editing, and a dozen interaction states. And every single one of these components comes with a very detailed spec written by our UX team. Our UX designers are genuinely good at their job, and they write down the interaction behaviour at a level of detail most engineering teams aren’t used to working with. Every state, every transition, every edge case and every pixel is well documented before anything gets built.<\/p>\n\n\n\n

That detail matters because of how the components eventually get judged. Before anything ships, it goes through a UX signoff session, and that session is strict. A slightly wrong interaction, a hover state that behaves a bit differently than spec’d, even a pixel-level spacing issue, can be enough to fail the review and send the whole thing back to dev. So accuracy isn’t a nice-to-have here. It’s the actual bar the work has to clear.<\/p>\n\n\n\n

That’s the backdrop I was building against, and it shaped a lot of how I thought about where AI should be trusted to run on its own, and where a person needed to look first.<\/p>\n\n\n\n

The basic pipeline<\/h3>\n\n\n\n

The system takes an approved UX design from Figma and turns it into a functionally perfect, visually reviewed, fully all usecase tested, storybook documented component. It does this across five steps.<\/p>\n\n\n\n

First, it reads the design and writes down exactly what the component is supposed to do, including its props, its states, its variants, and how it should behave during every interaction. Second, it creates the file structure for the component, following the conventions our codebase already uses and based on our framework standards. Third, it writes the actual implementation. Fourth, it writes unit tests covering all possible use cases. Fifth, it generates storybook documentation so other engineers know how to use the new component.<\/p>\n\n\n\n

Each of these steps has its own dedicated AI agent doing the work, and there’s one more agent on top of all of them whose only job is to call the right agent at the right time and pass information from one step into the next. That part isn’t really the interesting bit though. What I actually want to talk about is what happens between these steps, because that’s where I had to decide how much I trusted the AI to keep going on its own.<\/p>\n\n\n\n

My first instinct about this was wrong<\/h3>\n\n\n\n

When I started out, I thought about human review the simple way. More checking equals safer, less checking equals faster, find a balance somewhere in the middle. I think that’s how most people naturally think about it. But it’s the wrong way to think about it, and it took building this to actually see why.<\/p>\n\n\n\n

Checking the work isn’t one dial you turn up or down evenly across a process. It’s a separate decision at every single point where one step’s output becomes the next step’s input, and how much that decision matters changes a lot depending on where you are in the chain.<\/p>\n\n\n\n

A mistake near the end of a process is usually cheap to deal with. If the documentation step writes an awkward sentence about a prop, that’s a small, contained problem. Nothing downstream depends on it because there is no downstream, it’s the last step.<\/p>\n\n\n\n

But, a mistake at the very start is a completely different problem. Say the first step misreads the spec and misses that a button needs a distinct loading state, even though the UX team documented it clearly. That mistake doesn’t stay contained. It becomes part of the actual code in step three. It becomes part of the tests in step four, except now the tests are written against the wrong behaviour, so they pass happily and give false confidence that everything’s fine. Then it becomes part of the documentation too, quietly teaching every other engineer the wrong thing about how this component works. And the version that eventually walks into the UX signoff session is the one carrying that mistake, dressed up as a finished, tested, documented piece of work.<\/p>\n\n\n\n

One small misread at the start. By the time anyone notices, it’s spread through four more stages, and now you’re not just fixing a sentence, you’re re-doing code, tests, and docs, and possibly explaining to the design team why their signoff session caught something your process should have caught earlier.<\/p>\n\n\n\n

So that ended up being the real lesson. A mistake doesn’t cost the same no matter where it happens. It costs more the further it travels before someone catches it.<\/strong> Review effort should follow that, not be spread out evenly, and definitely not skipped just to move faster.<\/p>\n\n\n\n

Where I actually put the checkpoints<\/h3>\n\n\n\n

Once I started seeing it that way, the design got a lot more obvious. After every one of the five steps, a person looks at what the AI produced before anything moves forward. They can approve it, or they can send it back with feedback. Nothing advances on its own.<\/p>\n\n\n\n

\"Article<\/figure>\n\n\n\n

But I will be honest, not all five checkpoints are doing equal work, and pretending otherwise would be misleading. The checkpoint right after step one, the written specification, is the one I’d argue hardest to protect if someone ever tried to cut it to save time. Every step after that trusts the specification completely and never goes back to look at the original Figma design again. If the spec has a gap, every later step inherits that gap without knowing it exists. Given how detailed and exact our UX specs are, and how unforgiving the signoff process is about even small mismatches, this checkpoint is genuinely the one doing the most work out of all five, simply because it’s positioned furthest from where the actual problem would eventually surface.<\/p>\n\n\n\n

The checkpoints toward the end still matter, but for a different reason. Not because a mistake there spreads everywhere, but because they’re the last chance to catch something before an engineer starts relying on this component, or before it walks into a UX signoff session it has no business failing.<\/p>\n\n\n\n

The mistake I almost made, confusing checking with starting over<\/h3>\n\n\n\n

Here’s something that genuinely surprised me, and it changed the design. In my first version, when a reviewer sent feedback back to the AI, I just took that feedback, added it to the original instructions, and asked the AI to write the whole thing again from the top.<\/p>\n\n\n\n

The problem is, regenerating from scratch gives you no guarantee that the parts the reviewer already approved of come back unchanged. Picture a reviewer who flags one small thing, something like the hover state description being incomplete. If the whole document gets rewritten, you might get back something that’s also subtly different in five other places nobody asked it to touch. Now the reviewer has to read the entire thing again from the beginning, just to make sure nothing else shifted. That’s not really a quick fix loop at that point. That’s asking someone to proofread the full document every single round, and after enough rounds of that, people stop reading carefully and just start approving things to get it over with, which defeats the whole point of having a checkpoint.<\/p>\n\n\n\n

So I split it into two separate jobs handled by two separate agents. One agent writes the first draft. A second, different agent takes the existing draft plus the reviewer’s feedback, finds the specific part the feedback is actually about, fixes only that part, and leaves everything else exactly as it was. It also writes a short, plain summary of exactly what it changed. That summary turned out to matter just as much as the fix itself. A reviewer needs to read something like “added the disabled and loading combined state to the interaction section” and trust that’s genuinely everything that changed, rather than wondering what else might be different now and having to go check.<\/p>\n\n\n\n

This is really the same lesson again from a different angle. Deciding where a human needs to check the work isn’t enough on its own. You also have to make the checking itself easy to do properly. If checking is slow, or makes someone redo work they already finished once, people stop doing it carefully, and a checkpoint nobody actually pays attention to isn’t really a checkpoint anymore, it’s just a formality.<\/p>\n\n\n\n

This also matters more than ever for token usage<\/h3>\n\n\n\n

There’s another reason the targeted editing approach matters, and it’s become hard to ignore recently. AI tools are moving toward usage-based pricing where every token counts. GitHub Copilot is a visible recent example of this direction, and most enterprise AI tools are heading the same way. In a multi-agent system running several steps in sequence, token usage adds up fast if you’re not deliberate about what actually goes into each agent’s context.<\/p>\n\n\n\n

The way this architecture handles it is by being strict about what each agent actually needs to see. The refinement agent doesn’t get handed the entire project history or the full original design file every time someone asks for a small change. It gets the current draft and the specific piece of feedback. That’s it. The orchestrator only passes forward what the next stage genuinely needs, not everything that’s been produced so far. And because refinements are targeted edits rather than full rewrites, the context going into each revision pass stays small no matter how many rounds of feedback there are.<\/p>\n\n\n\n

This isn’t purely a cost-saving move either. Keeping context lean also keeps the agents more accurate. <\/strong>There’s a well-documented pattern where AI models start losing focus or drifting when their context window gets very long. Smaller, focused inputs tend to produce tighter and more reliable outputs. So “only pass what’s needed” isn’t just good for the budget, it’s good for the quality of the output too. In this architecture, both of those things point at exactly the same design decision, which is a good sign you’ve got the right answer.<\/p>\n\n\n\n

\"Token<\/figure>\n\n\n\n

A simple way to think about this on any project<\/h3>\n\n\n\n

Pulling back from this one pipeline, here’s roughly what I’d carry forward to a different project.<\/p>\n\n\n\n

Ask where an error would get caught if you weren’t the one catching it now. If the honest answer is nowhere, that it would just quietly become part of everything built afterward, that step needs a human in the loop, no matter how good the model is at that particular task. If the answer is that it’s the last step and someone would just notice and fix it locally, you can probably afford to relax there.<\/p>\n\n\n\n

Ask how much would actually need to be redone if this one thing turned out to be wrong and only got discovered a few steps later. This is really the same question asked differently, but it makes the cost feel more real. Redoing the code, the tests, and the docs is a very different amount of pain compared to fixing a typo in a doc nobody else depends on.<\/p>\n\n\n\n

Don’t treat “send it back for changes” and “start over from scratch” as the same thing. Most non-trivial reviews involve more than one round of feedback. If every round means a full rewrite, review quality gets worse with each round instead of better, because the reviewer can never fully trust what came back without re-checking all of it again.<\/p>\n\n\n\n

And be ready to be wrong about where the real risk actually sits. Going into this, I assumed the riskiest step would be the implementation, actually writing the code. It wasn’t. The riskiest step was the very first one, simply because everything else got built on top of it without anyone going back to double check it against the original design. The step that feels the most technically impressive to get right isn’t always the one doing the most damage when it’s wrong.<\/p>\n\n\n\n

None of this is really specific to AI, if I’m honest about it. It’s closer to ordinary advice about running any process with several handoffs, catch problems as close to where they started as you can, and don’t let a review step turn into a rubber stamp. AI just makes the question feel more urgent, because it’s tempting to assume a model that sounds confident needs less of this discipline than a new teammate would. Working through this project, that assumption didn’t hold up for me even once.<\/p>\n\n\n\n

This is based on real lessons from building a multi-agent AI system for our component library, five specialized agents handling the work, one agent coordinating the sequence, and a person checking the output after every single step. I might write a more technical follow-up about the actual setup if there’s interest.<\/p>\n\n\n\n

Building this system changed how I think about human-in-the-loop AI. The question isn’t how often humans should review AI output. It’s where. Review every step equally, and you waste effort. Skip the wrong checkpoint, and small mistakes quietly become the foundation for everything that follows. Human review isn’t about distrust. It’s about stopping errors before they become expensive.<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

Just published something on a question I keep running into while building with AI agents: how much should you actually trust the AI to work unsupervised, and where does human review genuinely matter. Real lessons learned, not theory.<\/p>\n","protected":false},"author":1,"featured_media":27949,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[467],"tags":[483,488,485,486,484],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/posts\/27947"}],"collection":[{"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/comments?post=27947"}],"version-history":[{"count":3,"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/posts\/27947\/revisions"}],"predecessor-version":[{"id":27954,"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/posts\/27947\/revisions\/27954"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/media\/27949"}],"wp:attachment":[{"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/media?parent=27947"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/categories?post=27947"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/deepubalan.com\/blog\/wp-json\/wp\/v2\/tags?post=27947"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}