Self-hostingAgent runbook

Upgrades

Move a running install to a newer Agent Relay release safely - the EKS upgrader, manual GKE upgrades, and what to do if one fails.

Human version of this guide · Markdown

Written for a coding agent working for you. Lines marked Agent: are its checks and stop points. Everything else matches the human guide.

Agent Relay releases are tags of the public AgentWorkforce/infra repository, named agent-relay-<version>, each with a signed chart, an image manifest and checksums. Normally upgrade by moving to a whole release. Do not change one image digest on its own; a coordinated patch requires the reviewed plan described below.

Rules for every upgrade

  • Back up first. Every release can run database migrations, and a Helm rollback does not undo them. See Backup and restore.
  • Read the release notes of every release you skip, on the releases page.
  • Plan, review, then apply. Look at what will change before anything does.
  • Check the migrations after. Helm reports success even when a migration Job failed.
  • Use helm upgrade, never helm upgrade --install, for a release that already exists. Don't add --atomic.

Agent: Every upgrade changes the human's production install. ASK THE HUMAN for approval after showing the plan, and again before any --apply. STOP and report if the plan shows image changes outside the target release, or if a backup has not been taken in the last 24 hours.

Message receipts and Grok Bot

Message receipt lookup over MCP is available in Control 0.5.24 and Relaycast 12.10.0, published in component charts relay-control 0.4.17 and relaycast 0.3.14. Both services and their migrations are required. Updating a Desktop app or refreshing the Helm repository index alone does not upgrade those servers.

For an existing agent-relay umbrella install, choose a published whole release whose image manifest contains both versions or a compatible newer pair. A component chart release or an open suite candidate is not that release. If no published suite contains the pair yet, contact us for a reviewed coordinated patch plan for your existing chart and values. Do not run two new component installs over workloads owned by the umbrella, or change just one server image.

Keep the existing install's values, Secret references, database mode and image mirror configuration. Review the backup and migration plan before applying any upgrade. The cloud-specific steps below describe the normal whole-release path; they do not authorize installing an unreleased candidate.

After all required migrations complete and both servers are healthy, reconnect the bot's MCP connection and check that get_relay_message_status appears. Send one authorized test message, retain its message_id, then check that ID using the same sending identity. See Messaging and Grok Bot for the receipt states, webhook wake-up setup and unconfirmed-message handling.

Done when: the intended image digests are running, both services' migrations completed, the new MCP tool is listed, and you have either a confirmed acknowledgment/reply or an explicitly reported unconfirmed message.

Agent: Do not infer that a customer's server has upgraded because a chart or image was published. Identify the existing Helm release and values-file order first. Never print helm get values -a output or Secret data. Show a backup, migration and image plan to the human before any cluster change. Missing receipts alone never authorize resending a message or changing Desktop launch settings.

Coordinated receipt patch before a suite release

A paired image patch has been tested on a PostgreSQL-backend GKE install with its original chart (agent-relay 0.7.8, suite 2026.11.3). It has not been exercised with SQLite or proven on AKS. A full chart upgrade is a separate qualification. For an Azure install, have your Agent Relay contact review the existing release, database backend and backup plan before applying this patch. The EKS upgrader does not support Azure.

This procedure keeps the deployed chart and every existing values file, in the same order, and adds one final overlay. It changes only Relaycast and Control. Use this patch only for SQLite or PostgreSQL Relaycast backends. If the install uses the actors backend, stop for its separate compatibility plan. Do not change backends or enable new features with this patch.

  1. Record the deployed release, namespace, Helm revision, original chart archive and exact ordered values-file list. Keep these files private; they can contain credentials. Do not paste helm get values --all or Secret data into chat. If the original chart or values are missing, stop for support.
  2. Back up both services' data and wait for a successful backup before continuing. PostgreSQL requires a database backup, including Control's database. Control uses PostgreSQL on every Relaycast backend; back it up even when Relaycast uses SQLite. For SQLite, quiesce writes and take a consistent copy or volume snapshot with Relaycast stopped; an Azure SQLite PVC has no automatic chart backup by default. Record the recovery procedure and backup IDs. See Backup and restore and the infra backup runbook.
  3. Verify both published image attestations using the commands below. Stop if verification fails. If your install uses a private image mirror, copy both images there and verify their digests before applying; preserve the existing repository settings. Do not switch the install to public registries as a shortcut.
gh attestation verify oci://ghcr.io/agentworkforce/relaycast-self-host@sha256:ec062ef901af15cf575b53a71d7eb09f2e6ab6e7c0d5be2da35339a275a5c6cc --owner AgentWorkforce
gh attestation verify oci://ghcr.io/agentworkforce/relay-control@sha256:d1b6ca95fab240a9d41208d83de3b1f4245408b0765038762f1c419ee60f43bc --owner AgentWorkforce

Create receipt-images.values.yaml in your private install directory:

relaycast:
  image:
    digest: sha256:ec062ef901af15cf575b53a71d7eb09f2e6ab6e7c0d5be2da35339a275a5c6cc
relay-control:
  image:
    digest: sha256:d1b6ca95fab240a9d41208d83de3b1f4245408b0765038762f1c419ee60f43bc

These are ghcr.io/agentworkforce/relaycast-self-host:12.10.0 and ghcr.io/agentworkforce/relay-control:0.5.24. For a PostgreSQL Relaycast backend, also set relaycast.runtime.postgres.image.digest to the same Relaycast digest in this overlay; that backend selects its image from this separate key. Omit that addition for SQLite. Leave the database mode and all other components unchanged.

Render with the original chart and original ordered -f arguments, then add -f receipt-images.values.yaml last. Compare the rendered manifests with the deployed manifests privately. Expected changes are the two server images and their migration Job identities. Stop if anything else changes. Have the operator approve this diff, the backup plan and an outage window: Relaycast uses a Recreate deployment and is unavailable while its pod restarts and migrations/readiness finish. PostgreSQL pods wait for their migration Job.

For example, an existing Azure-guide install uses the following order. Replace the release, namespace and paths with those already used by that install; do not run this example on a different install or substitute a new chart:

helm template <existing-release> <original-chart.tgz> -n <namespace> \
  -f <original-values-eks.yaml> -f <site-values.yaml> \
  -f <original-release-images.values.yaml> -f receipt-images.values.yaml

# Only after reviewing the render and completing the backups:
helm upgrade <existing-release> <original-chart.tgz> -n <namespace> \
  -f <original-values-eks.yaml> -f <site-values.yaml> \
  -f <original-release-images.values.yaml> -f receipt-images.values.yaml \
  --timeout 25m

Wait for Control's migration Job to complete. PostgreSQL Relaycast also has a migration Job; SQLite migrates when its server starts, so check startup logs and readiness instead. Wait for both Deployments to become ready, check the API and Relaycast health endpoints and ingress, then perform the MCP receipt check above. Find the actual resource names from the release; do not invent a hash or assume a Helm release prefix. The Azure and GKE profile values use fullnameOverride, so their Deployments are normally relay-control and relaycast, with Jobs named relay-control-migrate-<identity> and relaycast-migrate-<hash> respectively:

kubectl -n <namespace> get jobs,deployments -l app.kubernetes.io/instance=<existing-release>
kubectl -n <namespace> wait --for=condition=complete job/<actual-control-migration-job> --timeout=15m
# PostgreSQL Relaycast only:
kubectl -n <namespace> wait --for=condition=complete job/<actual-relaycast-migration-job> --timeout=15m
kubectl -n <namespace> rollout status deploy/<actual-control-deployment> --timeout=15m
kubectl -n <namespace> rollout status deploy/<actual-relaycast-deployment> --timeout=15m

Keep the overlay as the record of this revision. When upgrading to a qualified whole release, review its manifest and remove this patch overlay from the values-file list so it cannot override the new release's images. Do not silently carry these fixed digests forward as the last overlay.

Done when: backups are recorded, the reviewed pair of digests is running, all required migrations completed, both services are healthy, and the bot can look up its own sent message. Publishing this procedure does not establish that an Azure rollout has been tested or that a customer's install is upgraded.

If a migration or rollout fails, collect its status and logs and stop for support. A Helm rollback does not restore the database. A downgrade needs a reviewed backup-restore plan; do not assume the older images can use a migrated schema.

Agent: Do not turn this patch into an automatic customer upgrade. Confirm the actual backend and original chart/values first. Keep rendered manifests and values private. For Azure, say that this path has not been proven on AKS and obtain a reviewed site-specific plan. Never run cluster commands from this guide without the human's approval of the exact commands and completed backup evidence.

EKS

The upgrader, scripts/selfhost/agent-relay-upgrade.sh in the infra repository, does the safe sequence for you. Plan-only by default, it:

  1. downloads the release's chart, manifest, image overlay and checksums, checks every checksum and verifies the GitHub build attestations;
  2. renders the chart with your values and prints every image and resource that would change; the last line is plan: changes=true or plan: changes=false;
  3. mirrors exactly the release's image digests into your ECR. From a laptop that takes about 15 minutes; from CloudShell or an EC2 instance in the same region it's much faster.

With --apply, after you confirm, it takes an RDS snapshot and waits for it, refreshes the database admin credentials, runs helm upgrade, and waits for the migration Jobs and Deployments. On a failure it stops and prints the previous Helm revision, the rollback command and the snapshot ID.

It needs gh, jq, yq, helm, kubectl, aws, crane, openssl and curl, and takes credentials only from your normal AWS, Kubernetes and GitHub environment. It always mirrors into an ECR registry in your account, so after the first upgrade the install pulls its images from there, even if it pulled from ghcr.io before. The nodes need ECR pull access (the default for EKS node roles in the same account).

1. Check out the target release

cd agent-relay-install && source settings.sh
export AGENT_RELAY_VERSION=<target version, e.g. 2026.11.1>
git -C infra fetch --tags && git -C infra checkout "agent-relay-$AGENT_RELAY_VERSION"

Done when: git -C infra describe --tags prints agent-relay-<target version>.

checkout refuses because of local changes? That's the single-node OpenSearch edit to main.tf from install step 4a. Keep it across the checkout, because a later terraform apply without it grows the domain to six nodes:

git -C infra stash && git -C infra checkout "agent-relay-$AGENT_RELAY_VERSION" && git -C infra stash pop

2. Plan

export ECR_REGISTRY=<account>.dkr.ecr.<region>.amazonaws.com
export RDS_INSTANCE_ID=<rds instance id>
infra/scripts/selfhost/agent-relay-upgrade.sh \
  --version "$AGENT_RELAY_VERSION" --registry "$ECR_REGISTRY" \
  --profile eks-native --rds-setup --values "$INSTALL_DIR/values.yaml"

Done when: the last line is plan: changes=true, and you have reviewed the image changes and the resource diff it printed.

3. Apply

infra/scripts/selfhost/agent-relay-upgrade.sh \
  --version "$AGENT_RELAY_VERSION" --registry "$ECR_REGISTRY" \
  --profile eks-native --rds-setup --values "$INSTALL_DIR/values.yaml" \
  --db-instance "$RDS_INSTANCE_ID" --apply

Done when: it completes without an error, and every migration Job and pod is healthy:

kubectl -n agent-relay get jobs
kubectl -n agent-relay get pods

Then run step 15's checks from Install on EKS again, and the Output coverage step in After the install with a new runId: lowercase letters, digits and dashes, at most 20 characters, for example upgrade-2026-11-5.

--rds-setup runs rds-setup.sh refresh-admin, which rewrites the two credential files derived from the database admin login and updates their Secrets. RDS can rotate that password, so this runs before every upgrade. If you manage Secrets with helm-secrets or vals, rerun your secrets commands after it. Never run rds-setup.sh pre again.

Installs older than 2026.11.0: two revisions for the scrubber

Installs made from these guides already run the strict secret scrubber (global.scrubEngine in values.yaml). An older install with data in it must move readers first, in a separate revision, before the writer changes:

cp "$INSTALL_DIR/values.yaml" /tmp/agent-relay-reader-first.yaml
yq -i '.global.scrubEngine.engine = "" | .global.scrubEngine.required = ""' /tmp/agent-relay-reader-first.yaml
infra/scripts/selfhost/agent-relay-upgrade.sh --version "$AGENT_RELAY_VERSION" --registry "$ECR_REGISTRY" \
  --profile eks-native --rds-setup --values /tmp/agent-relay-reader-first.yaml \
  --db-instance "$RDS_INSTANCE_ID" --apply

Check that every sessions Deployment is ready, every sessions Job completed, and every services container has READER_MAX_RULESET=10. Then plan and apply again with your real values.yaml. Never combine the two revisions on a populated database, and keep the reader ceiling at 10 on any rollback.

GKE

There is no automated upgrader on GKE yet. Upgrade by hand:

  1. Back up. Take an on-demand Cloud SQL backup and wait for it: gcloud sql backups create --instance "$CLOUD_SQL_INSTANCE".
  2. Get the new release. In a new folder, repeat step 1 of Install on GKE for the target version, and step 5 to mirror its images and write a new release-images.values.yaml. Keep your existing values.yaml and managed-values.json.
  3. Compare the profiles. diff -u <old>/release/agent-relay/values-gcp.yaml <new>/release/agent-relay/values-gcp.yaml. Add any new required keys to your values.yaml.
  4. Review the change. With the helm-diff plugin, run step 11's command as helm diff upgrade instead of helm upgrade --install.
  5. Apply with step 11's command, using helm upgrade and the new chart file. Put the new release-images.values.yaml last.
  6. Check the migrations (step 12), the pods, and step 14's checks.

Done when: the three migration Jobs completed, every pod is ready, and api ok, cast ok, dashboard 200 and history 401 print.

If an upgrade fails

  1. Stop. Don't retry, uninstall, roll back or delete anything yet.

  2. Keep the evidence:

    helm -n agent-relay history agent-relay
    kubectl -n agent-relay get jobs,deploy,pods
    kubectl -n agent-relay get events --sort-by=.lastTimestamp | tail -40
    kubectl -n agent-relay logs job/<failed job> --all-containers --tail=200
  3. Find out which migrations ran. If none did, fix the cause and run the upgrade again.

  4. If migrations ran, the database is already on the new schema. A helm rollback leaves it there. Restore the backup you took (side by side, then switch traffic) unless the release notes say the previous version runs on the new schema. Ask us when in doubt.

Agent: On a failed upgrade: collect the evidence above (check it for secrets before showing it), then STOP and ask the human. Never run helm rollback, delete PVCs, delete Jobs or restore a database without explicit approval.


Stuck? See troubleshooting or support.