How to build an AI agent: from idea to a working one in a day

I built a customer-question agent this afternoon. Version one got three out of twenty test questions wrong. Version two fixed those three and then broke a fourth one in a completely different way. That's the part nobody's guide mentions: the fix is never the end of the story.

This is for anyone who's read five "how to build an AI agent" articles and still doesn't know what to actually type into a terminal. By the end, you'll have seen a real agent built, tested, broken, and fixed, with the exact numbers at each step. The honest catch: this isn't a fully autonomous customer service bot you can point at your live inbox tomorrow. It's the classify-and-route core that every one of those bots is built on, which you can extend once you trust it.

Can you really build an AI agent in a day?

Yes, if you pick a narrow enough job. Most guides talk about "an AI agent" like it's one big thing: a brain, a memory, some tools, a runtime loop. That's true, but it's also why most people never finish, because they try to build the whole loop before testing any single piece of it. I built one piece: a small program that reads an incoming customer question, decides which team should handle it, and drafts a starting reply. That's the routing core underneath most real support agents, and it's small enough to build, test, and fix inside one afternoon.

What you need before you start

You need three things, and none of them are exotic. First, a way to run a short script, which can be Python on your own machine or a tool like Claude that can write and reason about code for you. Second, a test set of real examples: I wrote 20 sample customer questions across four categories (billing, technical, shipping, general) instead of trusting my gut about what would work. Third, patience for at least one rebuild, because the first version is not going to be the last one. Set aside an afternoon and skip signing up for anything until you've proven the logic works.

Step 1: build the simplest version that could possibly work

The simplest agent I could build was a keyword matcher: scan the question, count how many words match each category, pick the category with the highest score. No AI model needed for this part, just a dictionary of words per category and a suggested reply for each one.

Read this customer message and classify it as billing, technical, shipping, or general. If it genuinely fits two categories equally, say "needs human" instead of guessing. Message: "{customer question here}"

I ran the plain keyword version against my 20 test questions. It got 17 right, which is 85%. Not bad for an afternoon's work, and also not good enough to trust with a real customer.

Step 2: find out exactly why it failed

Here's where I lost time I didn't expect to lose. All three wrong answers had the same shape: a question that mentioned money ("charged", "card") while actually being about a bug or a lost package. "I was charged for shipping but never received a tracking number" is a shipping problem, but the word "charged" pulled it into billing every time, because my first version treated every matching word as worth the same one point.

Step 3: fix it, and watch the fix cause a new problem

The fix was straightforward: give unique, specific words (bug, crash, tracking, delivered) more weight than shared, ambiguous ones (charged, card). I also added a rule that if two categories end up truly tied, the agent flags the question for a human instead of picking one. Rerunning the test set, all three original misses fixed themselves.

Score: 19 out of 20.

Then I checked what happened to that twentieth question, and this is the part that actually taught me something. "My card was declined and now the app says my account is locked" is a plain billing question. My new tie-breaker rule flagged it as "needs human" anyway, because the word "app" matched a keyword I'd added for the general category, and that tied it with the billing score. The fix that stopped the agent from guessing wrong on real ambiguity also made it nervous about an ordinary word that happened to overlap. Two versions, one afternoon, and I ended up with a new, smaller problem instead of zero problems. That's the trade nobody mentions when they show you the finished agent instead of the process.

The result: what one afternoon of building actually gets you

Twenty test questions, two versions, one afternoon: version one at 85%, version two at 95% with one question correctly escalated for the wrong reason. That's not a shipped product. It's a routing core you could now plug a real inbox into, with actual evidence for where it's still shaky, instead of a demo that looked fine because nobody tried to break it. Build it once, win every week only works if the "once" includes finding out where it breaks, and I now know exactly which kind of sentence to double-check before I trust this thing with anything real.

When this doesn't work

This approach breaks down the moment your questions stop looking like short, single-topic sentences. My test set was clean on purpose. A real inbox has typos, two questions in one message, and customers who are angry enough to skip punctuation entirely. A keyword classifier like mine also can't handle a brand-new phrasing it's never seen, where an actual language model would still get the gist. If you're routing more than a handful of categories, or your messages are long and rambling, you'll want a real LLM doing the classifying, not a word-counting script, and you should budget for reviewing its mistakes the same way I reviewed mine.

Would I build one again?

Yes, and I'd start with the test set again before writing a single line of the agent itself. The twenty questions I wrote up front are the only reason I know my accuracy went from 85% to 95%, instead of just feeling like it improved. If you're weighing which tool should actually run this once it's built, that's less about the agent logic and more about the automation platform underneath it, which is exactly what I compared in n8n vs Zapier vs Make. And if a single agent isn't the whole job you're trying to offload, I stacked five smaller automations back to back in I automated 5 tasks in 5 days, which is the wider view of what this fits into.