I'm going through Stanford's reinforcement learning course, and something in the lecture on imitation learning stuck with me. The professor explains the DAgger algorithm, a way to teach a policy to recover from its own mistakes by letting an expert correct it as it goes. The examples are all robotics and self-driving, but I kept thinking about LLM agents: give a model a bunch of tools and let it decide what to call, and its run drifts the same way a cloned driving policy does. Not the scripted flows where the LLM just picks a category and hands off, the open-ended kind, where it works out the tools and the order on its own. So I took the philosophy of DAgger and sketched out how it would map onto one of those.
DAgger
If you train a policy by imitation, say behavior cloning on an expert's demonstrations, it behaves fine on the cases the expert has seen but falls apart on states the expert never showed. A small early mistake nudges it somewhere the demonstrations never covered, which causes a bigger mistake, and so on. The errors compound, and the longer the task runs the faster they pile up, faster than the step count itself.
DAgger's fix targets that specific failure mode. Instead of training once on the expert's trajectories and deploying, you iterate:
- Deploy the current student policy to the environment.
- At every state the student visits, record it. If its uncertainty is above a threshold, ask the expert what they would have done here.
- Add those
(state, expert action)pairs to a growing datasetD. - Retrain the student on all states, old and new.
- Repeat for several rounds, relying on the expert less each time.
That is not the original DAgger, the original asks the expert at every single step, which gets expensive fast, in this case we use a threshold, which is more of a gated version of DAgger.
The example: a support ticket agent
Take the messy support tickets, the ones a human resolves by reaching for tools in an order that depends on what each one returns. A customer writes "I was charged twice and my order still shows cancelled." The human looks up the order, sees a payment mismatch, pulls the payment history, checks the authorization hold. Each tool's output decides which tool comes next. There are too many branches to draw as a fixed flowchart ahead of time. The state is the ticket plus everything gathered so far. The action is the next tool call, just the next one.
A resolved ticket unrolls into one pair per step. The state at each step is the ticket plus every tool output gathered so far, and the action is the tool the human reached. For example on a refund ticket:
A routine run. "Where is my order?" The student policy looks up the order -> pulls the carrier tracking -> sees the package in transit -> drafts a reply with the delivery estimate. Familiar ticket, familiar path, no escalation.
The drift run, the one DAgger is actually for. Now the double-charge ticket. Refund tickets dominate the training data, so on a familiar ticket the student reaches for the refund tool without hesitating. This one is different. The payment history shows one real charge and one pending authorization hold, the kind that quietly drops off in a few days. The ticket sends mixed signals, a charge that looks refundable next to a hold that says wait, so the policy can't decide. The confidence gate sees the shaky guess and routes the ticket to a human. The human checks the authorization status, explains the hold to the customer, and issues no refund. That decision gets recorded, the human's tool choices become (state, expert action) pairs in D, saved at the exact spot where the student got stuck.
What piles up in D over months is the whole point, corrected next-tool decisions at the states the initally policy failed. Common ticket shapes stop escalating after a few retraining rounds, while the genuinely weird ones keep reaching a human, forever. That is fine. Weird tickets are exactly what you keep humans around for.
Training the student
The set of choices is small and fixed, the student is only picking one of a handful of tools, so we can treat the next-tool decision as plain classification, we can run the state through a pre-trained embedding model, the thing that turns text into a list of numbers that capture its meaning, and leave that model frozen so it never changes. Then train a single linear layer on top. The frozen model handles the language, and the only thing you train is a small classifier over the tool set.
At rollout time the same head hands you both things the loop needs, the next tool and the gate signal, in one pass:
The softmax turns the classifier's raw scores into probabilities, and that gives the gate a cheap confidence signal: when the top tool sits at 0.9 the gate stays quiet, and when the guess is split across a few tools the ticket escalates. You get it for free, no extra model to measure confidence, no probability bookkeeping. The classifier's own output is the signal.
There is an issue though, this only catches the mistakes the student knows it might be making. The dangerous case is when it is confidently wrong, aka it choses the wrong tool instead of spreading its guess, so the gate sees high confidence and stays quiet. What we can do is training a few heads on different random subsets of the data and comparing their votes gives a better read on when to trust the guess.
Another limit is that the head predicts which tool, not its arguments. Often that is enough, the arguments are mechanical once the tool is picked and the order id is already sitting in the state or the outcome of the previous tool is attached to the state. When they do need judgment, we need to add a small generation step or fall back to the full LLM.
The rollout loop
The DAgger loop itself is not a framework, it is a while loop. The student proposes the next tool, a gate checks whether the step needs correcting, and if it does the ticket goes to a human whose decision is recorded before the rollout continues.
In production the human answers later, so you back it with a queue or something that can pause the execution and resume it when the answer comes in. That is an engineering detail, not part of the algorithm.
Final thoughts
This is design is more of an inspiration I had, not an algorithm you drop in expecting some guarantees. DAgger originally come from Robotics and there someone has to "sit in the loop and grab the wheel". A support org already runs that expert tier and already pays for it, aka are the human agents resolving tickets DAgger is just the discipline of asking those people what to do at the states where the agent is lost, and not throwing the answer away but record it to help future iterations of the agent.