Human in the Loop AI Agents Need a Guess, Not a Signal
Human in the loop AI agents are only as good as the tool's guess about when the agent is waiting. How AuricIDE reads a terminal tail and flags Needs input.
Does the agent tell the app it is waiting? What if the terminal is just thinking? Who notices if nobody is watching the window?
Those questions show up when an agent card sits mid-run, its last lines are a bare y/n, and nobody has typed an answer. The card still looks busy.
Does the Agent Tell the App It Is Waiting?
No. The CLI never sends a waiting flag.
The tidy story says the agent raises a request, the app receives it, and a dialog pops up. That would be cleaner. It is not how the process runs. Each agent is a local process of its own, and all the app gets is what the process prints, plus whether it exited well or badly. A permission question is just text in the terminal, like every other line.
With human in the loop AI agents, that gap decides everything. The person can sit next to the screen and still miss the minute the agent stopped. The question never left the pane. Think of a shop bell that never rings: staff can stand at the counter all afternoon and still miss the customer at the door.
A human in the loop is only as good as the app's guess about when the agent is waiting. The loop closes when the app recognizes a question, not when a person happens to be nearby.
Waiting for the agent to raise its own hand is not a plan either. In HiL-Bench, a benchmark by Trinh et al. (arXiv, April 2026), models reached 64–88% on software-engineering tasks given full information, and the best one reached 12% once it had to decide for itself when to ask. That test covers missing requirements, not permission prompts. It still points the same way. So the app does not wait for a signal. It looks.
How Does AuricIDE Recognize a Waiting Agent?
AuricIDE reads the last five lines of the agent's output and checks whether they look like a prompt.
The list of prompt shapes is short and explicit: a yes-or-no choice like (y/n), a line such as do you want to ...?, a numbered Yes/No menu, "press enter", or "waiting for input". When one of them sits inside those five lines, the card changes to Needs input, and you answer in the Reply to agent... field on the card. Open it. Read the question. Type the answer. The chain does not pause on a timer of its own. It waits because the tail looked like a question.
Prose does not count. If the agent writes "Which is better here, the cache or the queue?" in the middle of its work, the card stays on Working. Agents narrate questions all the time, and a bare question mark would set off the alarm constantly. Once the agent prints more, the old prompt scrolls out of the five lines. The card drops back.
Silence alone does not count either. A build can run without printing. A model can pause before its next line. From outside, an agent mid-thought and an agent holding a question look the same, because both stop printing. Only the tail tells them apart. An agent that goes quiet without a prompt-shaped tail ends up as Stalled?. That is a separate state for a separate problem. Agent monitoring covers watching run state over time. The guess here is narrower. Five lines, then a label. Done and Failed work differently again, since they come from how the process exits.
Needs input means the recent output looks like a prompt. It does not prove the agent stopped. Read the terminal before you answer.
Where Does a Waiting Agent Show Up?
On the card, in the Agent Console, in the title bar, in the window title and on the dock icon. Not in the inbox.
- The card shows Needs input with the reply field right on it.
- An amber chip in the title bar reads "2 need attention" and opens the agents panel when clicked.
- The Agent Console has a Needs you column. A waiting agent appears there as Waiting on you, with the question itself when AuricIDE can pick it out of the output.
- The window title reads
(2) AuricIDE. macOS hides the title text under this app's title bar, so on a Mac the chip and the badge do that job. - The dock icon carries the same number as a badge, even with the app hidden.
It is one number, counted once. A failed agent and a stalled agent count in it too, not only the ones that need input. Two waiting agents and one failure show as 3. The number alone cannot say which agent needs what.
Needs input creates no inbox entry and no push. Failures take the other path: a toast and an inbox entry named after the agent. A waiting prompt stays live.
Who Decides How Often an Agent May Ask?
You do, in the Start agent dialog, before the run begins.
Permission Mode sets how much the agent may do without stopping. Pick Act without asking and the agent skips its permission prompts. The five-line guess has less to catch. AuricIDE asks for a confirmation the first time you do that in a session, and the dialog says plainly that the agent can edit files and run commands without asking.
The agent has a say too. It can call the MCP tool notify_ask, which puts a question with answer buttons into the inbox. That path skips the guess. It is a real signal. But the agent has to choose to send it.
How often a team should force a review is its own call. The product sets no rule. What an agent may touch at all is a separate question, and access control covers it.
What Does the Guess Cost?
Three things, and each one is a decision AuricIDE made.
A prompt in a shape that is not on the short list gets missed at first. AuricIDE took a short, explicit list over catching everything. A false alarm teaches you to ignore the real ones. A miss has a backstop. An agent that goes quiet turns into Stalled? after 120 seconds without output. That label says it went quiet. It does not say why.
Leave AuricIDE out of sight and only the dock badge tells you that some agent needs attention, because the inbox stays empty. The cards and the Needs you column only help once the window is open again.
Act without asking removes the permission prompts for that agent. Other prompts, such as "press enter", still fall to the guess. The agent can also ask on purpose, through notify_ask.
What Holds in Any Tool?
One thing holds in any tool. A person can only answer a question the tool has noticed. Oversight is two jobs, though: the tool has to notice the question, and the person has to judge what comes back.
Teams already expect oversight to cost more than planned: 63% of respondents in G2's August 2025 AI agents survey said agents needed more human oversight than they had expected. Noticing the question is only the first job. A noticed question can still be answered badly. A reported high-stakes clinical finding showed reviewers failed to catch and override incorrect AI outputs in 64% of cases when the AI sounded highly confident. The slide deck behind that link, a conference talk on governance for human review, credits Lyell et al. (JAMIA, 2017) for the figure.
Whether the tool learns of the question from a signal or from a guess about the screen decides whether that first half happens at all. Check that first, before asking how many people are free to review.