Alerting

Rule types that actually evaluate, the three comparators, and which notification channels work out of the box.

An alert rule evaluates a condition on a schedule and notifies a channel when it holds.

Before you start. You need telemetry arriving from at least one service — see Send data with OpenTelemetry.

How evaluation works

Rules are evaluated on a fixed 60-second loop. Each rule also carries its own evaluation window and a cooldown, both defaulting to 5 minutes.

The cooldown is what stops a flapping condition from paging you repeatedly: once a rule fires, it will not fire again until the cooldown elapses.

The 60-second loop is the floor on detection latency. Setting a rule’s evaluation window shorter than a minute does not make it check more often — the loop still runs once a minute. The window controls the span of data each check considers, not the checking frequency.

Rule types

These types are evaluated by the alerting loop:

Type	Fires when
Threshold	A value crosses a bound. The common case.
Absence	Expected data stopped arriving. This is the one that catches a crashed exporter, which no threshold on the data itself can detect.
SLO burn rate	An error budget is being consumed too fast.
Composite	Combines other rules.
Governance	Policy conditions on AgentSight data.

Two further types exist and are not driven by this loop. Anomaly rules are fired by the AI service; synthetic rules are fired by the synthetics scheduler (see Synthetic monitoring). Creating one of those two as an ordinary rule and expecting the evaluator to run it will not work — nothing evaluates it and it silently never fires.

Conditions

A threshold rule compares an aggregate against a value. Three comparators are implemented:

Comparator	Meaning
GT	Greater than
LT	Less than
EQ	Equal to

Only these three comparators exist. The condition field accepts any short string, so a rule saved with GTE, LTE or NE will be created without complaint and will then never fire — nothing matches it at evaluation time. There is no error and no warning; the rule simply sits there looking healthy.

Express the bound you want with GT or LT. Use the Test action after saving any rule to confirm it does something.

Severity uses the vocabulary LOW, MEDIUM, HIGH, CRITICAL, and drives how each channel renders the alert — PagerDuty urgency, OpsGenie priority, Slack formatting. This field is also not strictly validated on the normal create path, and a severity outside that set falls back to the equivalent of a warning rather than being rejected.

Notification channels

Five channels work with no additional configuration beyond their own credentials.

Slack

A formatted message to an incoming webhook. Needs webhookUrl.

PagerDuty

Events API v2. Needs routingKey. Severity maps to critical / error / warning / info.

Microsoft Teams

An Adaptive Card to an incoming webhook. Needs webhookUrl.

OpsGenie

Creates an alert. Needs apiKey, optionally teams.

Email

Sent through your SMTP server. Needs to, smtpHost, smtpPort, smtpUser, smtpPass and from.

There is no platform-operated mail relay; if you do not have an SMTP server to point at, use another channel.

Generic outbound webhooks are disabled by default. WEBHOOK is the schema’s default channel, and it is the one channel that does not work out of the box. It is gated behind a feature flag, off by default, pending SSRF protection, request signing and retry handling.

With the flag off, the API refuses to create or test a webhook-channel rule outright — so you get an error rather than a silently dead rule. If you want webhook delivery, ask your operator to enable the flag, and understand the gap it is currently open on.

Slack, PagerDuty, OpsGenie and Teams deliveries retry up to four times with 1s / 5s / 30s backoff and a 10-second timeout. 4xx responses are not retried, since a rejected payload will be rejected again.

Email is not retried. A failed send is lost, so do not make email your only channel for anything you must not miss.

Testing a rule

Every rule has a Test action, and rules can be tested before you trust them. Test notifications are prefixed [TEST] and use a separate deduplication key, so firing one never merges with a live incident or resolves one.

Given that an unimplemented comparator produces a rule that looks fine and never fires, testing after saving is not optional — it is the only thing that distinguishes a working rule from a decorative one.

Runbooks

A rule can carry a runbook URL, which is rendered as a link in every channel. The person woken at 3am by the alert gets a link to what to do about it, which is most of the value of the alert.

Next

Synthetic monitoring

Watch endpoints from outside.

Find the logs behind an alert

The query syntax.