A crucial decision in payment processing is automated based on a confidence score exceeding 0.8, a threshold whose origin is unknown. This single number dictates whether refunds are processed instantly by a machine or delayed for human review. Misplacing this threshold can lead to automated processing of deserving cases or overwhelming human queues with unnecessary ones. The system utilizes rules, decision models, and human oversight, with the threshold determining which path a case takes.Decision models, unlike readable rules, are trained and can fail silently by returning numbers for unseen data. The third path, human review, fails by not happening, resulting in a growing backlog. The confidence score from a model represents a prediction of its accuracy, not a subjective feeling. This confidence score is often confused with overall system accuracy, which is an aggregate metric.Calibrating a model and choosing a threshold are distinct tasks requiring different judgments, as a model only informs costs of different thresholds, not the optimal cut. Setting the threshold too low leads to irreversible automated errors, while setting it too high creates an invisible burden of unnecessary human reviews. These errors are not symmetrical or equally visible, with the latter quietly doubling automation costs.The decision of where to set the threshold depends on the cost of different errors, implying a policy choice rather than a constant tuning. This policy needs to be informed by a reliability report, a curve showing accuracy bands at different confidence levels. This curve reveals the true cost of a chosen threshold in terms of error rate.Before trusting any threshold, consider where errors land, the base rate of the issue being screened, and who is responsible for escalation. The threshold is where management decisions translate into executable code, and it should not be a hidden configuration value. Instead, it requires an owner, a date, and triggers for review, reflecting a deliberate policy choice. Ultimately, a well-defined policy, supported by data and clear ownership, is essential for effective threshold management.