Set up alerts

Alerts notify you when events occur in Braintrust. Send notifications to webhooks or Slack channels to catch errors, track quality issues, monitor usage patterns, or track prompt deployment changes.

Enable the Slack integration before creating an alert to send to a Slack channel.

Alert types

Braintrust supports three types of alerts:

Log alerts: Trigger when conditions are met on production logs.
Time window alerts: Trigger when a SQL calculation over a time window crosses a threshold.
Environment alerts: Trigger when prompt or dataset environments are assigned or removed.

Create a log alert

Go to Settings > Alerts.
Click Alert, or Create alert if you don’t have any alerts yet.
Enter alert name.
Select Log event as the event type.
Configure alert conditions:
- SQL filter: Query that defines which logs trigger the alert. See Write SQL filters for examples. When Loop is enabled, select Generate to create the filter from a natural-language description.
- Notify interval: The minimum time between notifications for this alert (5 min, 30 min, 1 hr, 4 hr, 12 hr, 24 hr).
  Braintrust evaluates logs in batches, so alerts aren’t instantaneous and a single alert can cover more than one matching log. When a batch triggers the alert, the notification counts only the matching logs in that batch, then the alert is suppressed for the length of the interval. Matching logs that arrive during that window don’t trigger another alert and aren’t reported later, so choose an interval short enough that you don’t miss what you need to act on.
Select an action type:
- Webhook: Enter the webhook URL to send a JSON payload to. Click Test next to the Webhook URL field to verify the configuration. See Test an alert.
- Slack: Select a Slack channel using the searchable dropdown. The channel list refreshes automatically every 7 days. To trigger a manual refresh, click Refresh channels. If your channel does not appear, you can enter its manually.
Click Create.

You can also create log alerts directly from the Logs or Dashboards pages. Apply filters to your logs, then select > Create alert from filters in the toolbar. The SQL filter automatically populates with your current filters.

Create a time window alert

Time window alerts evaluate a scalar SQL calculation over a time window and notify you when the numeric result crosses a threshold. Use them for aggregate conditions that log alerts cannot express, such as an average score dropping below a target, an error count exceeding a limit, or a breach that must persist before notifying.

Go to Settings > Alerts.
Click Alert, or Create alert if you don’t have any alerts yet.
Enter alert name.
Select Time window as the event type.
Configure the calculation with Builder or Query. The calculation must return one numeric value for each evaluated window. For example, to alert when errors exceed a count threshold:
SELECT COUNT(*) AS error_count FROM project_logs('<PROJECT_ID>') WHERE error IS NOT NULL
Set Window length (minutes) to the aggregation window.
Set Alert trigger to the comparison that fires the alert. For example, value is > 100.
Optionally, configure Advanced settings.
Select an action type:
- Webhook: Enter the webhook URL to send a JSON payload to.
- Slack: Select a Slack channel using the searchable dropdown. The channel list refreshes automatically every 7 days. To trigger a manual refresh, click Refresh channels. If your channel does not appear, you can enter its manually.
Click Create.

The Query editor must return at most one row. If it returns multiple numeric columns, select the Numeric result column to compare. If it returns one numeric column, Braintrust selects that column for you. Advanced settings include:

Setting	Purpose
Trigger delay (seconds)	Require the threshold condition to remain breached before sending a notification.
When no data is returned	Choose whether an empty evaluation keeps the last state, resolves the alert, or triggers the alert.
Send a recovery notification	Notify when the alert recovers after firing.
Evaluation schedule	Choose Fixed interval with Evaluate every (seconds), or choose Cron with Cron expression and optional Timezone.
Wait for late data (seconds)	Delay evaluation so late-arriving logs can land before the window is checked.
Renotify interval (seconds) (optional)	Send repeat notifications while the alert continues firing.

Use the Recent evaluation preview to confirm the calculation returns the expected value before saving the alert.

Create an environment alert

Environment alerts notify you when prompt or dataset environments are updated. Use them to track deployments, monitor version changes, or trigger downstream workflows when prompts or datasets are promoted across environments.

Go to Settings > Alerts.
Click Alert, or Create alert if you don’t have any alerts yet.
Enter alert name.
Select Environment update as the event type.
Optionally filter by specific environments (e.g., only alert on “production” changes).
Select an action type:
- Webhook: Enter the webhook URL to send a JSON payload to.
- Slack: Select a Slack channel using the searchable dropdown. The channel list refreshes automatically every 7 days. To trigger a manual refresh, click Refresh channels. If your channel does not appear, you can enter its manually.
Click Create.

Environment alerts trigger immediately when environments are updated. Unlike log alerts, they don’t have intervals or SQL filters.

Test an alert

Testing is available for log alerts that use the Webhook action. Click Test next to the Webhook URL field. Braintrust runs the SQL filter on recent logs. If matching logs exist, a test payload is sent. Check your webhook endpoint for the test message. Testing is not available for Slack actions or for environment alerts.

Pause an alert

Pause an alert to stop its notifications while keeping its conditions and delivery targets intact. For self-hosted deployments, pausing requires data plane v2.11.0 or later.

While an alert is paused, matching events don’t trigger notifications and aren’t reported later. Resuming the alert doesn’t notify you about anything that happened during the pause.

Go to Settings > Alerts.
Select the alert you want to pause.
Turn off Enabled.
Click Update.

The Status column shows Paused for a paused alert of any type. While running, log alerts and environment alerts show Active, and time window alerts show a health state such as Healthy or Evaluation failing instead. To resume a paused alert, turn Enabled back on.

Set config.status to paused with PATCH /v1/project_automation/{project_automation_id}. Send the complete config object, because the request replaces it rather than merging into it.

curl -X PATCH https://api.braintrust.dev/v1/project_automation/<project_automation_id> \
  -H "Authorization: Bearer $BRAINTRUST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "event_type": "logs",
      "status": "paused",
      "btql_filter": "error IS NOT NULL",
      "interval_seconds": 3600,
      "action": {
        "type": "webhook",
        "url": "https://example.com/webhook"
      }
    }
  }'

To resume the alert, send the same request with "status": "active".

Because the request replaces config, a later config update that omits status clears it and the alert resumes. Include status in every config update for an alert you intend to keep paused.

Edit or delete an alert

Go to Settings > Alerts.
Select the alert you want to change.
Update the alert configuration, then click Update.

If you close an alert with unsaved changes, Braintrust asks you to confirm before discarding them. To delete an alert, open it and click Delete in the footer, or click the delete icon in the alert’s row. Confirm the deletion in the dialog that appears.

Deleting an alert immediately stops its notifications. This cannot be undone.

Common alert patterns

Log alerts
Environment alerts

Error monitoring: Catch production issues immediately.

SQL filter: error IS NOT NULL AND metadata.environment = 'production'
Action: Post to Slack #incidents channel or create tickets in issue trackers

Quality thresholds: Track when model performance drops below acceptable levels.

SQL filter: scores.factuality < 0.8 AND metadata.environment = 'production'
Notify interval: 1 hr
Action: Send to monitoring systems or trigger automated remediation

Cost monitoring: Alert on expensive requests.

SQL filter: estimated_cost() > 1.0
Action: Webhook to cost tracking systems or budget management tools

Model-specific issues: Alert on problems with a particular model.

SQL filter: metadata.model = 'gpt-4o' AND (error IS NOT NULL OR scores.accuracy < 0.8)
Action: Post to team channel for model performance investigation

Feature-specific monitoring: Track specific workflows or user segments.

SQL filter: metadata.user_tier = 'enterprise' AND metadata.feature = 'summarization'
Action: Post to team channel for priority investigation

Combined conditions: Alert on multiple conditions for critical requests.

SQL filter: (scores.accuracy < 0.7 OR error IS NOT NULL) AND metadata.priority = 'high'
Action: Immediate Slack notification to on-call team

Usage spikes: Monitor when request volume exceeds normal levels.

Action: Use external systems to track historical rates and webhook alerts to capture spikes

Aggregate thresholds: Log alerts trigger on individual matching logs. To alert when an aggregate crosses a threshold, such as an average score or an error count over a window, use a time window alert instead.

Deployment tracking: Monitor when prompts are promoted to production.

Environment filter: production
Action: Post to Slack #deployments channel

Multi-environment monitoring: Track changes across all environments.

Environment filter: Leave empty to monitor all environments
Action: Webhook to deployment tracking system

Critical environment protection: Get immediate notification of staging or production changes.

Environment filter: staging, production
Action: Slack notification to team channel

Audit trail: Log all environment changes to external systems.

Environment filter: Leave empty
Action: Webhook to audit logging service

Webhook payloads

Webhooks can route alerts to any third-party incident, on-call, or automation tool that accepts inbound webhooks (for example, PagerDuty’s Events API, Opsgenie, or workflow platforms like Zapier). Braintrust delivers a fixed JSON payload (shown below) with the Content-Type: application/json header, and doesn’t support custom headers or payload formats. If the destination requires a specific payload shape or authentication headers, add a lightweight forwarder that transforms the Braintrust webhook payload before delivering it.

Webhook URLs must use http or https. To protect against server-side request forgery (SSRF), Braintrust blocks webhook delivery to private, internal, or otherwise reserved network addresses. Use Test to confirm your endpoint is reachable.

Log alerts
Time window alerts
Environment alerts

When a log alert triggers a webhook, it sends this JSON structure:

{
  "organization": {
    "id": "org_123",
    "name": "your-organization"
  },
  "project": {
    "id": "proj_456",
    "name": "your-project"
  },
  "automation": {
    "id": "c5b32408-8568-4bff-9299-8cdd56979b67",
    "name": "High-Priority Factuality",
    "description": "Alert on factuality scores for priority logs",
    "event_type": "logs",
    "btql_filter": "metadata.priority = 0 AND scores.Factuality < 0.9",
    "interval_seconds": 3600,
    "url": "https://braintrust.dev/app/your-org/p/your-project/configuration/alerts?aid=..."
  },
  "details": {
    "is_test": false,
    "message": "High-Priority Factuality: 5 logs triggered this alert. You won't be notified again for at least 1 hour.",
    "time_start": "2025-05-12T10:00:00.000Z",
    "time_end": "2025-05-12T11:00:00.000Z",
    "count": 5,
    "related_logs_url": "https://braintrust.dev/app/your-org/p/your-project/logs?search=..."
  }
}

When a time window alert triggers a webhook, it sends this JSON structure:

{
  "version": 1,
  "organization": {
    "id": "org_123",
    "name": "your-organization"
  },
  "project": {
    "id": "proj_456",
    "name": "your-project"
  },
  "automation": {
    "id": "c5b32408-8568-4bff-9299-8cdd56979b67",
    "name": "Hourly Error Count",
    "description": "Alert when error volume is high",
    "event_type": "windowed",
    "threshold": {
      "calculation": {
        "type": "btql",
        "btql_query": "SELECT COUNT(*) AS error_count FROM project_logs('proj_456') WHERE error IS NOT NULL",
        "value_column": "error_count"
      },
      "policy": {
        "condition": {
          "type": "threshold",
          "operator": "gt",
          "threshold": 100
        },
        "pending_seconds": 0,
        "no_data_behavior": "keep_last",
        "renotify_interval_seconds": null,
        "notify_on_recovery": true
      }
    },
    "window": {
      "window_seconds": 3600,
      "schedule": {
        "type": "interval",
        "evaluation_interval_seconds": 300
      },
      "evaluation_delay_seconds": 0
    },
    "url": "https://braintrust.dev/app/your-org/p/your-project/configuration/alerts?aid=..."
  },
  "details": {
    "transition": "firing",
    "value": 125,
    "time_start": "2026-08-07T10:00:00.000Z",
    "time_end": "2026-08-07T11:00:00.000Z",
    "evaluation_error": null,
    "evaluation_failure_count": null,
    "message": "Hourly Error Count is firing for the evaluated window."
  }
}

The top-level version field indicates the time window payload schema version. The details.transition value describes why the notification was sent: "scheduled", "firing", "recovery", "renotify", "evaluation_failed", or "evaluation_recovered".

When an environment alert triggers a webhook, it sends this JSON structure:

{
  "organization": {
    "id": "org_123",
    "name": "your-organization"
  },
  "project": {
    "id": "proj_456",
    "name": "your-project"
  },
  "automation": {
    "id": "c5b32408-8568-4bff-9299-8cdd56979b67",
    "name": "Production Environment Changes",
    "description": "Alert when production environment is updated",
    "event_type": "environment_update"
  },
  "details": {
    "environment": {
      "slug": "production"
    },
    "object_type": "prompt",
    "prompt": {
      "id": "prompt_789",
      "slug": "summarizer"
    },
    "new_version": "8234923849293849...",
    "action": "update"
  }
}

For dataset environment changes, object_type is "dataset" and the payload includes a dataset field instead of prompt:

{
  "organization": { "id": "org_123", "name": "your-organization" },
  "project": { "id": "proj_456", "name": "your-project" },
  "automation": {
    "id": "c5b32408-8568-4bff-9299-8cdd56979b67",
    "name": "Production Environment Changes",
    "description": "Alert when production environment is updated",
    "event_type": "environment_update"
  },
  "details": {
    "environment": {
      "slug": "production"
    },
    "object_type": "dataset",
    "dataset": {
      "id": "dataset_789",
      "name": "golden-set"
    },
    "new_version": "8234923849293849...",
    "action": "update"
  }
}

The action field will be either "update" (environment assigned to object) or "delete" (environment removed from object). When action is "delete", the new_version field will be null.

Limitations

For self-hosted deployments:

Alerts are available starting with data plane v0.0.72.
The Slack integration is available starting with data plane v1.1.29.
Time window alerts are available starting with data plane v2.10.0.
Pausing and resuming alerts is available starting with data plane v2.11.0.

For Slack workspaces with more than 100,000 channels:

The channel list shows the most recently active channels. The channel list refreshes automatically every 7 days. To trigger a manual refresh, click Refresh channels. If your channel does not appear, you can enter its manually.

Next steps

Export to cloud storage and configure data retention
View logs to understand alert triggers
Monitor deployments with dashboards
SQL reference for advanced filter queries

Start

Instrument

Observe

Annotate

Evaluate

Deploy

Admin

Best practices

Alert types

Create a log alert

Create a time window alert

Create an environment alert

Test an alert

Pause an alert

Edit or delete an alert

Common alert patterns

Webhook payloads

Limitations

Next steps

​Alert types

​Create a log alert

​Create a time window alert

​Create an environment alert

​Test an alert

​Pause an alert

​Edit or delete an alert

​Common alert patterns

​Webhook payloads

​Limitations

​Next steps

Alert types

Create a log alert

Create a time window alert

Create an environment alert

Test an alert

Pause an alert

Edit or delete an alert

Common alert patterns

Webhook payloads

Limitations

Next steps