Skip to content

Alert Runbooks

This page is the Excalibur operator's response guide for the Prometheus alerts fired by the Excalibur observability stack. Each alert email includes a View Runbook link that opens the matching entry below. Every entry follows the same Symptom → Cause → Resolution pattern so you can move from an alert to a fix quickly.

Prerequisites

  • You have kubectl access to the Excalibur namespace. Replace <namespace> with your deployment's namespace and <pod-name> with the pod named in the alert.
  • To open Prometheus or Alertmanager, see Access Grafana and Prometheus.
  • For cluster-level signals behind these alerts (node health, scheduling, resource pressure), see Cluster health signals.

How Alerts Reach You

Alerts are shipped by the centralized excalibur-observability Helm chart — kube-prometheus-stack (Prometheus and Alertmanager), Loki, and Alloy — together with a custom Excalibur PrometheusRule. Prometheus evaluates the rules; Alertmanager groups firing alerts and sends the branded email that links here.

Alerts are grouped by severity. Use the severity to decide how fast to respond.

Severity Meaning Response
critical A service is down, or data loss is imminent. Investigate immediately.
warning A component is degrading or trending toward failure. Investigate soon, before it becomes critical.
info An informational performance signal, not an outage. Review; act only if users are affected.

Individual pod restarts are normal

Most Excalibur services run multiple replicas, so Kubernetes absorbs a single pod restart or eviction transparently. Treat these alerts as signals to investigate, and reserve "service down" for the Deployment Down condition. See High availability and what "down" means.

Alerts at a Glance

Alert Severity Fires when
ImagePullBackOff critical A container cannot pull its image for 5m.
ErrImagePull critical An image pull fails for 3m, before backoff.
High Restart Rate warning A container restarts more than 3 times in 15m.
Deployment Down critical A Deployment has 0 available replicas for 5m.
Container OOMKilled critical A container is killed for exceeding its memory limit.
Backup CronJob Failed critical A backup Job reports failure for 5m.
CronJob Missed Schedule warning A CronJob is 10m+ past its schedule and not suspended.
PVC Filling Up warning A volume is over 85% full for 10m.
PVC Almost Full critical A volume is over 95% full for 5m.
Pod Pending warning A pod cannot be scheduled for 10m.
Container Near Memory Limit warning A container uses over 90% of its memory limit for 5m.
CPU Throttling High info A container is CPU-throttled over 25% of the time for 15m.

Image Pull Failures

These alerts mean Kubernetes cannot download a container image. Excalibur images are pulled from ghcr.io/excalibur-enterprise/ using the GitHub Container Registry (GHCR) pull secret excalibur-registry.

ImagePullBackOff

Symptom

One or more pods stay in ImagePullBackOff and never become Ready. Kubernetes has given up retrying the image pull and is backing off. The alert fires when kube_pod_container_status_waiting_reason{reason="ImagePullBackOff"} > 0 for 5 minutes (critical).

NAME                          READY   STATUS             RESTARTS   AGE
api-5884bcbf58-2r6kk          0/1     ImagePullBackOff   0          6m

Cause

  • The image tag or digest does not exist in ghcr.io/excalibur-enterprise/ — usually a wrong version in the Helm values.
  • The excalibur-registry pull secret is missing, expired, or lacks read:packages scope.
  • The registry is unreachable because of DNS, egress, or proxy restrictions.

Resolution

  1. Read the exact pull error from the pod events:

    kubectl describe pod <pod-name> -n <namespace>

    Look under Events for the underlying reason:

    Failed to pull image "ghcr.io/excalibur-enterprise/api:<tag>": not found

  2. Confirm the image reference points to a real tag:

    kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].image}'

    Compare the output against the version in your Helm values and the tags published in GHCR.

  3. Verify the pull secret exists and is referenced by the pod:

    kubectl get secret excalibur-registry -n <namespace>
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.imagePullSecrets[*].name}'

  4. Fix the root cause:

    • Wrong tag — correct the image tag in your Helm values and run helm upgrade.

    • Missing or expired secret — refresh excalibur-registry, then run helm upgrade so the value persists across future upgrades:

      kubectl create secret docker-registry excalibur-registry \
  --docker-server=ghcr.io \
  --docker-username=<github-username> \
  --docker-password=<github-token> \
  -n <namespace>

    The token is a credential

    The --docker-password value is a GitHub token with read:packages scope. Do not paste it into shared logs or tickets, and store it only in your Helm values or a secret manager.

  5. Delete the failing pod so its Deployment recreates it with the corrected settings:

    kubectl delete pod <pod-name> -n <namespace>

    For per-tenant workloads, the pod name follows the pam-tenant-<n>-deployment-* pattern.

  6. Verify the pod pulls its image and becomes Ready:

    kubectl get pods -n <namespace>

    Confirm the pod reports 1/1 and Running.

ErrImagePull

Symptom

A pod reports ErrImagePull on its first pull attempt, before Kubernetes starts backing off. The alert fires when kube_pod_container_status_waiting_reason{reason="ErrImagePull"} > 0 for 3 minutes (critical). It catches the same failure as ImagePullBackOff, only earlier.

NAME                          READY   STATUS         RESTARTS   AGE
core-7b9fdc7888-st8xb         0/1     ErrImagePull   0          3m

Cause

Identical to ImagePullBackOff: a bad image tag, a missing or expired excalibur-registry secret, or an unreachable registry. This is the first pull error; if it persists, Kubernetes escalates it to ImagePullBackOff.

Resolution

  1. Read the immediate pull error before backoff begins:

    kubectl describe pod <pod-name> -n <namespace>

  2. Apply the diagnosis and fix from the ImagePullBackOff resolution — the causes and steps are the same.

  3. Verify the pod becomes Ready:

    kubectl get pods -n <namespace>

    Confirm the pod reports 1/1 and Running, and that it did not escalate to ImagePullBackOff.

Workload Health

These alerts track whether your workloads stay up and stable. A single restart is routine; sustained restarts or every replica down is not.

High Restart Rate

Symptom

A container restarts repeatedly. The alert fires when increase(kube_pod_container_status_restarts_total[15m]) > 3 (warning) — more than three restarts in a 15-minute window. kubectl get pods shows a climbing RESTARTS count, often with CrashLoopBackOff.

NAME                          READY   STATUS             RESTARTS      AGE
pam-tenant-0-deployment-xxx   0/1     CrashLoopBackOff   5 (30s ago)   12m

Cause

  • The application crashes on startup because of a bad config value, a missing secret, or a failed migration.
  • A liveness probe kills a container that is healthy but slow to start.
  • The container is OOMKilled and restarts in a loop — see Container OOMKilled.

Resolution

  1. Identify the restarting container and its restart count:

    kubectl get pods -n <namespace>

  2. Read the logs from the previous, crashed instance:

    kubectl logs <pod-name> -n <namespace> --previous --tail=100

  3. Check the termination reason and probe events:

    kubectl describe pod <pod-name> -n <namespace>

    Under Last State, a Reason of Error points to an application crash; OOMKilled points to memory. Under Events, Liveness probe failed points to a probe timing issue.

  4. Route to the fix:

    • OOMKilled — follow Container OOMKilled.
    • Liveness probe too aggressive — increase the probe initialDelaySeconds or timeoutSeconds in the Helm values, then run helm upgrade.
    • Config or secret error — correct the value and roll out the change with kubectl rollout restart deployment/<service-name> -n <namespace>.

  5. Verify the restarts stop and the pod stabilizes:

    kubectl get pods -n <namespace>

    Confirm RESTARTS stops increasing and the pod reports Running.

Deployment Down

Symptom

Every replica of a Deployment is unavailable. The alert fires when available_replicas == 0 while spec_replicas > 0 for 5 minutes (critical). Because no replica can serve traffic, this is a genuine service outage.

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
api     0/2     2            0           139d

Cause

  • All replicas are crashing from a shared cause — a bad config, or a failed dependency such as the database or cache.
  • No replica can be scheduled — see Pod Pending.
  • The image cannot be pulled across all replicas — see ImagePullBackOff.

Resolution

  1. Identify the affected Deployment and its available count:

    kubectl get deploy -n <namespace>

  2. Inspect the Deployment's pods to see the shared failure mode:

    kubectl get pods -n <namespace> -l app=<service-name>

  3. Follow the runbook that matches what you see: ImagePullBackOff, Pod Pending, Container OOMKilled, or High Restart Rate.

  4. Check shared dependencies. Many services depend on the 3-node MariaDB StatefulSet — confirm it has quorum:

    kubectl get statefulset database -n <namespace>

    Expected output:

    NAME       READY   AGE
database   3/3     139d

    A READY count below 3/3 means one or more of database-0, database-1, database-2 is down and dependent services cannot recover until quorum returns.

  5. After fixing the root cause, watch the rollout complete:

    kubectl rollout status deployment/<service-name> -n <namespace>

  6. Verify the service recovers:

    kubectl get deploy <service-name> -n <namespace>

    Confirm AVAILABLE matches the desired replica count.

Container OOMKilled

Symptom

A container was terminated with reason OOMKilled and exit code 137 — it exceeded its memory limit. The alert fires immediately (for 0m) when kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} > 0 (critical).

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Cause

  • The container's memory limit is too low for its real workload.
  • A long-running service such as pam or core has a slow memory leak that grows over days.
  • A workload spike — for example, many concurrent sessions on a single tenant pod.

Resolution

  1. Confirm the kill reason and identify the container:

    kubectl describe pod <pod-name> -n <namespace>

    Under Last State, confirm Reason: OOMKilled and Exit Code: 137.

  2. Compare current usage against the container's limit:

    kubectl top pod <pod-name> -n <namespace> --containers
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources.limits.memory}'

  3. Review the trend in Grafana — the Memory by pod panel on the Excalibur Kubernetes Metrics dashboard. A line that keeps rising without dropping indicates a leak; a flat line near the limit indicates an undersized limit.

  4. Apply the fix:

    • Leak in a long-running service — reclaim memory now with a rolling restart, which returns usage to baseline without an outage:

      kubectl rollout restart deployment/<service-name> -n <namespace>

      For per-tenant workloads, restart the pam-tenant-<n>-deployment instead.

    • Undersized limit — raise the container's memory limit (and request) in the Helm values and run helm upgrade.

  5. Verify recovery:

    kubectl get pods -n <namespace>

    Confirm the container reports Running, RESTARTS is stable, and no new OOMKilled events appear.

Scheduled Jobs

These alerts cover the CronJob-based backups and any other operator-defined CronJobs. Backups write to the backup-repository volume and are your primary recovery mechanism.

Backup CronJob Failed

Symptom

A backup Job failed. The alert fires when kube_job_status_failed{job_name=~".*backup.*"} > 0 for 5 minutes (critical). Your most recent recovery point may be missing.

NAME                        STATUS     COMPLETIONS   DURATION   AGE
backup-28912345             Failed     0/1           2m         6m

Cause

  • The job could not write to the backup-repository volume because it is full, unbound, or has a read-write access problem.
  • The database dump failed because the database was unavailable or the credentials were wrong.
  • The job exhausted its backoffLimit or hit its activeDeadlineSeconds and gave up.

Resolution

  1. List recent jobs and find the failed backup run:

    kubectl get jobs -n <namespace> | grep backup

  2. Read the failed job's logs to find the failure point:

    kubectl logs job/<job-name> -n <namespace> --tail=100

    Look for database dump or snapshot creation errors.

  3. Check backup-repository capacity — a full repository is a common cause:

    kubectl get pvc backup-repository -n <namespace>

    If it is near capacity, follow PVC Almost Full before retrying.

  4. Confirm the database is healthy, since each run dumps the database:

    kubectl get statefulset database -n <namespace>

    Confirm READY is 3/3.

  5. Re-run the backup manually once the cause is fixed:

    kubectl create job --from=cronjob/<backup-cronjob-name> <backup-cronjob-name>-manual -n <namespace>

  6. Verify the manual run completes:

    kubectl get jobs -n <namespace>

    Confirm the manual job reports COMPLETIONS 1/1.

CronJob Missed Schedule

Symptom

A CronJob stopped creating Jobs on schedule. The alert fires when the CronJob is more than 600 seconds (10 minutes) past its next scheduled time, is not suspended, and its schedule has advanced — evaluated for 10 minutes (warning). The rule is (next_schedule_time - last_schedule_time > 0) and (time() - next_schedule_time > 600) and (spec_suspend == 0).

NAME             SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
backup-cronjob   0 * * * *     False     1        95m             139d

Cause

  • A previous Job is still Active and the CronJob uses concurrencyPolicy: Forbid, so new runs are skipped.
  • Too many missed start times (over 100) caused the controller to stop scheduling, governed by startingDeadlineSeconds.
  • An admission webhook or a ResourceQuota rejects new Job creation.

Resolution

  1. Inspect the CronJob and confirm it is not suspended:

    kubectl get cronjob -n <namespace>

    Confirm SUSPEND is False. If it is True, the CronJob was suspended intentionally — resume it with kubectl patch cronjob <cronjob-name> -n <namespace> -p '{"spec":{"suspend":false}}'.

  2. Check for a stuck active Job blocking the schedule:

    kubectl get jobs -n <namespace>

  3. Read the CronJob events for the scheduling error:

    kubectl describe cronjob <cronjob-name> -n <namespace>

    Look for too many missed start times or webhook and quota rejection messages.

  4. Clear a genuinely hung Job so the next run can start:

    kubectl delete job <job-name> -n <namespace>

    Confirm the job is hung first

    Deleting an active Job terminates its pod. Delete only after you confirm the Job is stuck — a running backup should be allowed to finish.

  5. Recover the schedule with a manual run if the missed-start deadline was exceeded:

    kubectl create job --from=cronjob/<cronjob-name> <cronjob-name>-manual -n <namespace>

  6. Verify scheduling resumes:

    kubectl get cronjob <cronjob-name> -n <namespace>

    Confirm LAST SCHEDULE updates at the next tick and a fresh Job appears in kubectl get jobs.

Storage Capacity

These alerts track PersistentVolumeClaim (PVC) usage. When a volume fills up, the service writing to it stops accepting writes — and because storage is shared across a stateful service's replicas, the whole service is affected.

PVC Filling Up

Symptom

A PersistentVolumeClaim is over 85% full. The alert fires when used/capacity > 0.85 for 10 minutes (warning). This is an early signal — writes still succeed, but the volume is trending toward full.

Cause

  • Session recordings and application data are growing on excalibur-data, typically the largest and fastest-growing volume.
  • Loki or Prometheus retention is set too high for the loki-data or prometheus-data volume size.
  • The backup-repository volume is accumulating snapshots faster than retention prunes them.

Resolution

  1. Identify which volume is filling. Use the Persistent Volume Usage time-series panel on the Excalibur Kubernetes Metrics dashboard, or list the PVCs:

    kubectl get pvc -n <namespace>

  2. Confirm live usage from inside the pod that mounts the volume:

    kubectl exec <pod-name> -n <namespace> -- df -h

  3. Choose a remedy:

    • Expand — if the StorageClass supports volume expansion, increase the size in the Helm values (or patch the PVC) and run helm upgrade.
    • Reclaim — tune Loki, Prometheus, or backup retention down. See Persistent storage for the volume inventory and defaults.

  4. Verify usage falls below 85%:

    Recheck the Persistent Volume Usage panel, or re-run kubectl exec <pod-name> -n <namespace> -- df -h and confirm the volume has headroom.

PVC Almost Full

Symptom

A PersistentVolumeClaim is over 95% full. The alert fires when used/capacity > 0.95 for 5 minutes (critical). Writes may start failing at any moment.

Cause

The same drivers as PVC Filling Up — recordings, observability retention, or backup accumulation — but now urgent. Under time pressure, expanding the volume is usually the safest option.

Resolution

  1. Identify the near-full volume immediately:

    kubectl get pvc -n <namespace>

  2. Expand the volume if the StorageClass allows online expansion:

    kubectl patch pvc <pvc-name> -n <namespace> \
  -p '{"spec":{"resources":{"requests":{"storage":"<new-size>"}}}}'

    Also update the corresponding size in your Helm values so the change survives future upgrades.

  3. If expansion is not possible immediately, reclaim space:

    • loki-data / prometheus-data — reduce retention to drop old data safely.
    • backup-repository — let retention prune expired snapshots, or increase the volume size.

    Do not delete application or database data

    Never delete files directly from excalibur-data (session recordings) or a database-data-database-<n> volume to free space. A full database-data volume stops that database node from writing — expand it or escalate, but do not delete data.

  4. Verify usage falls below 95% and writes resume:

    Recheck the Persistent Volume Usage panel and confirm the writer pod's logs no longer report write failures.

Scheduling and Resources

These alerts cover pods that cannot be scheduled and containers under CPU or memory pressure. They range from a warning (a stuck pod) to informational (throttling).

Pod Pending

Symptom

A pod stays in Pending and is never scheduled onto a node. The alert fires when kube_pod_status_phase{phase="Pending"} > 0 for 10 minutes (warning).

NAME                                     READY   STATUS    RESTARTS   AGE
virtual-browser-tenant-1-deployment-xxx  0/1     Pending   0          11m

Cause

  • No node has enough free CPU or memory to fit the pod.
  • Node affinity rules, taints, or a nodeSelector prevent placement.
  • The pod's PVC is unbound, so the scheduler waits for storage.

Resolution

  1. List the pending pods:

    kubectl get pods -n <namespace> --field-selector=status.phase=Pending

  2. Read the scheduling reason from the pod events:

    kubectl describe pod <pod-name> -n <namespace>

    Under Events, a FailedScheduling message names the cause:

    0/6 nodes are available: insufficient memory. preemption: 0/6 nodes are available.

  3. Route to the fix:

    • Insufficient resources — free capacity or add nodes. Node scaling is usually a platform-team task; see Cluster health signals. Per-tenant pods (pam-tenant-<n>, virtual-browser-tenant-<n>) can have large resource requests.
    • Taints or affinity — adjust the tolerations or nodeSelector in the Helm values and run helm upgrade.
    • Unbound PVC — check kubectl get pvc -n <namespace> for a Pending claim and confirm the StorageClass exists.

  4. Verify the pod schedules and starts:

    kubectl get pods -n <namespace>

    Confirm the pod moves from Pending to Running.

Container Near Memory Limit

Symptom

A container is using more than 90% of its memory limit. The alert fires when working_set/limit > 0.9 and limit > 0 for 5 minutes (warning). Without relief, the container risks an OOMKill.

Cause

  • Real load has grown to approach a limit that was set too low.
  • A long-running service is leaking memory gradually.

Resolution

  1. Identify the container and its usage:

    kubectl top pod <pod-name> -n <namespace> --containers
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources.limits.memory}'

  2. Review the trend in the Memory by pod panel on the Excalibur Kubernetes Metrics dashboard. Steadily rising usage indicates a leak; steady-high usage indicates an undersized limit.

  3. Apply relief:

    • Leak — run a rolling restart to reclaim memory now:

      kubectl rollout restart deployment/<service-name> -n <namespace>

    • Undersized limit — raise the memory limit and request in the Helm values and run helm upgrade, sizing to the observed peak.

  4. Verify usage falls below 90% of the limit:

    kubectl top pod <pod-name> -n <namespace> --containers

    Confirm no OOMKilled event follows.

CPU Throttling High

Symptom

A container is being CPU-throttled more than 25% of the time. The alert fires when the ratio of Completely Fair Scheduler (CFS) throttled periods to total periods exceeds 25% over 5 minutes, sustained for 15 minutes (info; node-exporter is excluded). This is a performance signal — users may see added latency, but the service is not down.

Cause

  • The container's CPU limit is too low for its workload, so the kernel throttles it against its CFS quota.
  • A sustained traffic spike on a specific service or tenant pod.

Resolution

  1. Identify the throttled container. Use the CPU seconds by container panel on the Excalibur Kubernetes Metrics dashboard, or:

    kubectl top pods -n <namespace> --sort-by=cpu

  2. Check the container's CPU limit:

    kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources.limits.cpu}'

  3. Choose a remedy:

    • Legitimate load — raise the CPU limit in the Helm values and run helm upgrade. For stateless services, adding a replica often relieves pressure better than a larger limit.
    • Single tenant driving load — review that tenant's pam-tenant-<n> or virtual-browser-tenant-<n> sizing.

  4. Verify throttling drops below 25%:

    Recheck the CPU seconds by container panel after the change and confirm throttling subsides.

    Info severity — no immediate action required

    Throttling alone does not cause an outage. Act on this alert only when the added latency affects users or the throttling is sustained.

Contact Support

If none of these runbooks resolves the alert, export the relevant logs and share them with the Excalibur support team using Excalibur Chronicler. See Collect diagnostic data for full instructions, including encrypted exports.