Start with a labeled evaluation set
Label representative cases before looking at model output. Include difficult examples and the mistakes that would be costly. Run the same cases at several thresholds and count correct automated outcomes, incorrect automated outcomes, and review cases.
Compare coverage and error cost
A higher threshold usually sends more cases to review. That can be appropriate for a sensitive route and unnecessary for a reversible internal tag. The playground starts at 0.85 as a configurable example, not a validated threshold for your business.
Handle missing confidence explicitly
If the upstream response omits confidence, the result requires review. We do not invent a score. Even a high reported confidence can be wrong when your data differs from the model’s evaluation distribution. Recheck performance after changing instructions or categories.