Agent Middleware: What Kept Obelisk's Agents in Line

Obelisk is the AI marketing platform we build at Entropy Labs. It runs more than fifteen specialist agents: brand voice, SEO, email, paid ads, strategy and so on, all behind one API.

Most of the work ended up in what wraps each model call: which model to use, what to do when the conversation gets long, how to stop an agent calling the same tool forever, when to ask a human before spending money, and what to do when something fails. Early on, each agent handled those things its own way, which didn't last long.

We pulled every one of those concerns into middleware, small separate layers that run around every model and tool call and that each agent opts in or out of. LangChain's agent middleware gave us the hooks. Below are the decisions that mattered. Our thresholds and routing rules are left out.

One concern per layer

Each middleware does one thing: pick the model, trim the context, cap the calls, gate the dangerous tools, or recover from errors. An agent declares which ones it wants. The SEO agent, which uses a handful of tools, doesn't need tool filtering. The strategy agent, which can reach dozens of tools, does.

Order matters a lot. Model selection has to run first, so every later layer knows which model it's dealing with. Context trimming has to run before the call-limit check, or a long conversation burns its budget on retries. Approval gates have to sit close to the tool call, so nothing downstream can reroute around them. We found most of those orderings by getting them wrong.

Picking the model per request

Obelisk supports several model providers, and the model is chosen when a request arrives. The same agent can run on one provider for one customer and on another for a customer with a different contract or their own keys.

That was painful to build, because every provider streams, counts tokens and reports errors a little differently. In return we can use cheap models for cheap jobs like summarizing, accept customer-supplied keys, and fall back when one provider has a bad afternoon.

Summarizing long conversations

Long sessions with tool calls and research results fill context windows quickly. When a conversation gets close to a threshold, a summarization layer compresses the older messages with a cheap model and keeps the recent ones as they are.

The threshold is set well below the smallest context window we support, so switching models mid-conversation never overflows. Every summarization is also recorded, with when it fired and how much it compressed. When an agent "forgets" something, that record is the first thing we check.

Filtering tools per request

Obelisk has a lot of tools. Handing all of them to every call is a bad idea: the model spends tokens reading descriptions it won't use, and a longer menu means more wrong picks. For agents with large toolsets, a cheap classifier picks a small relevant subset per request, and a few tools, like knowledge-base search, are always included.

After that went in, the logs showed fewer irrelevant tool calls and a lot less prompt spent on tool descriptions.

Approval before sending or spending

Anything that sends, publishes or spends goes through an approval gate. The agent proposes the call, the graph pauses, and the person sees what's about to happen, including a spend warning when there's money involved. They can approve it, edit the arguments, or reject it.

HumanInTheLoopMiddleware(
    interrupt_on={
        "send_campaign_email": {"allowed_decisions": ["approve", "edit", "reject"]},
        "publish_ad": {"allowed_decisions": ["approve", "reject"]},
        "search_knowledge_base": False,
    },
)

This needs a checkpointer, because the conversation has to survive the pause, sometimes for hours. More than anything else in the system, this gate is what made people comfortable letting agents touch real accounts.

Retries

The default instinct with errors is "retry three times and hope". That's fine for a timeout. For a malformed email it means sending it malformed three times.

So errors get classified first. Timeouts and rate limits get retried with backoff; validation and auth failures don't. When retries run out, the agent gets the error as a message and decides what to do (try something else, ask the user, say it failed), and the conversation carries on.

@wrap_tool_call
def retry_transient(request, handler):
    for attempt in range(MAX_ATTEMPTS):
        try:
            return handler(request)
        except TransientError:
            if attempt == MAX_ATTEMPTS - 1:
                raise
            sleep(backoff(attempt))

Tool inputs are validated against their schemas before the call, so a bad argument fails before any side effect starts.

Splitting the general agent

The first version had one general agent with a very long system prompt that tried to cover every job. It wrote ad copy when asked for analytics and started research when someone wanted a quick answer.

Splitting it into specialists fixed most of that. Each agent has a short prompt, its own tools and its own output style, and a bug in one doesn't break the others. We route to agents directly from the product, since the UI already knows which kind of conversation it's in, so there's no agent picking agents. That's simpler to debug and saves a model call whose only job is deciding what the next model call should be.

Middleware first, logs from day one

We bolted the middleware onto working agents over several months. Building it first and adding agents on top would have saved a lot of refactoring.

Structured events should have been there from the start too. Debugging a many-agent system with print statements does not scale, and request IDs on every event came later than they should have. There's a post on structured logging about what we do now.

An orchestrator agent is tempting, and I'd still resist it. Direct routing plus good middleware is more predictable than a model deciding which model to call.