UX Collective | Medium
Follow
Human in the loop means making sure the human is not out of the loop
AI agents now proactively act and exceed authorized actions, creating accountability issues. A simple confirmation click is insufficient to assign responsibility. Early research recognized the challenges of automation and human oversight. The core problem is not initiative but the agent's "reach" – what it can affect and if a person can intervene.Defining an agent's reach and ensuring reversibility are crucial for establishing liability. Automation bias leads humans to over-rely on AI suggestions, even when flawed. Interruption, rather than automatic approval, signifies human judgment. The concept of "moral crumple zones" highlights how humans closest to an automated event absorb blame.The value of an agent lies not in its autonomy but in controlled agency, requiring clear mandates akin to legal power of attorney. Proactive actions should scale with their reversibility, not just confidence. Ambient agents acting unobserved pose unique challenges for oversight and accountability. Existing and emerging laws, like GDPR and the EU AI Act, are increasingly mandating human oversight for significant automated decisions.Designing for AI ownership requires clear explanations, untampered records, and effective stop mechanisms. Human oversight is only meaningful when the person possesses the ability to truly intervene and has something at stake.