What triage actually is
Triage is not "look at the alert and decide if it's real." That framing is why the queue never empties. Triage is a repeatable procedure that (a) reduces an alert to a verdict, (b) attaches the evidence used to reach that verdict, and (c) either escalates to investigation or closes with a rationale that survives audit.
Every alert that leaves triage without a verdict is a permanent liability. It shows up in post-incident reviews as "detected but not investigated" — the worst possible outcome. A working triage system produces zero unverdicted alerts.
The ManySignal triage agent operationalises this: for every finding, it walks a structured question set, computes a confidence-weighted verdict, and lands the case on the timeline with the full reasoning attached. Humans review outliers; they don't clear the queue.
The question set — anatomy of a triage decision
A triage question set is 5–8 questions answered against your live data. Each question has a resolver (graph, behavioural, enrichment, correlation, business_context, human_confirmation) and a weight. The verdict is the weighted sum of answers, normalised.
Questions must be specific and executable. "Is this suspicious?" is not a question — it's a wish. "Is the source ASN present in this identity's 90-day baseline?" is a question with an answer.
For each finding, the graph resolver traverses the entity graph and returns typed facts. Behavioural resolvers score the anomaly against per-entity baselines. Enrichment resolvers pull threat-intel matches. Human confirmation resolvers page the user via Slack/email with a one-tap yes-no. Every answer is timestamped and evidence-linked.
# question-set for identity.impossible_travel
key: identity.impossible_travel
questions:
- id: same_identity
prompt: "Are both authentications from the same canonical identity?"
resolver: graph
weight: 20
- id: known_asn
prompt: "Are both source ASNs in this identity's 90-day baseline?"
resolver: behavioural
weight: 15
- id: same_device
prompt: "Same device fingerprint across both logins?"
resolver: behavioural
weight: 15
- id: vpn_egress
prompt: "Is either IP a known corporate VPN egress?"
resolver: enrichment
weight: 10
- id: prior_anomaly
prompt: "Prior travel-velocity anomaly for this identity in 90 days?"
resolver: data_platform
weight: 10
- id: concurrent_sessions
prompt: "Concurrent active SaaS sessions right now?"
resolver: graph
weight: 15
- id: user_confirm
prompt: "User confirms via Slack: 'Did you sign in from <city>?'"
resolver: human_confirmation
weight: 15
verdict:
autonomous_threshold: 80
approve_gated_threshold: 60
Verdict thresholds and the autonomy ladder
Every question set defines two thresholds: autonomous_threshold (above this, action can execute without approval) and approve_gated_threshold (above this, action executes with approval). Below the lower threshold, verdict is close-with-rationale.
Thresholds are per-action-class. A read-only enrichment can be autonomous at score 60; a session revocation might require 80 + approval. Autonomy is earned per action class per team — treat it as a ratcheted trust budget.
The autonomy ladder is not "AI decides everything" — it's "AI produces the verdict, humans set the policy". Every action that fires records the verdict score, the question set version, and the actor (agent or human) on the immutable timeline.
Evidence attachment: the audit-proof output
Every verdict lands with attached evidence: the raw finding, each question's answer, each resolver's data, the confidence score, and the version of the question set at the moment of verdict. Nothing is reconstructable-later — it's captured atomically.
The case timeline is append-only and hash-chained. If a verdict is reopened for review, the review is a new timeline entry — not an overwrite. Audit teams love this because "tell me why analyst X closed alert Y last February" has a real answer.
Handling human confirmation without breaking flow
The user_confirm resolver pings the affected user via Slack or email with a two-button prompt ("Yes, that was me" / "No — not me"). The reply resolves the question inline. If unanswered in a configurable window (default 15 minutes), the resolver returns unknown and the verdict continues with the other answers.
For high-risk verdicts (session revocation, credential rotation), human confirmation is often decisive. "Not me" should trigger autonomous revocation within 60 seconds — the point is speed under high confidence, not more approvals.
Metrics that indicate the system is working
Track these five weekly: (1) % of alerts with a documented verdict, (2) median time to verdict, (3) TP:FP ratio per detection, (4) autonomous verdict rate, (5) alerts aged >24h without verdict (should be zero).
When any metric regresses, the question set — not the analyst rota — is the first place to look. Add a resolver, tighten a threshold, or split the detection into two.
Key takeaways
- Every alert must reach a documented verdict. Zero unverdicted alerts is the only acceptable target.
- Question sets are 5–8 executable questions, not "look and think".
- Autonomy is per-action-class, earned, and revocable — not a global switch.
- Evidence is atomic to the verdict — captured at decision time, not reconstructed later.
- Human confirmation via Slack is the highest-value resolver for identity attacks.
- Track TTV, autonomous %, TP:FP, and aged-alerts weekly. Regressions → question-set changes.