New on SSRN: Ariadne's Thread, a measurement-theoretic method for legal openness. Read the paper
Legal AI System Design

Red line

A decision reserved exclusively to a human that an automated agent must never make; when a task touches a red line, the agent logs it for a person and stops.

First used in I Built a Tiny Company Out of Robots, Gave Them House Rules, and Nearly Finished a Hard Thing

The term comes from an experiment in which five narrow agents (an auditor, a decision-maker, a planner, a builder swarm and a test runner) worked in a loop under house rules. Some decisions were prohibited for machines outright. One critical number, the kind a court would hold the company accountable for, remained a human judgment, and the standing instruction was to log any task touching a red line and walk away.

The experiment's other lesson extends the idea. After finishing its authorized work, the loop kept running because it had never been granted permission to stop; the fix was to have each run label itself productive or idle and cancel its own schedule when no authorized work remained.

Starting an agent is a prompt. Stopping one is judgment, and off-switches deserve as much design as activation.

Related terms

Related frameworks

Used in