Run the steps in order. Copy each block, replace every <...>, and check the
Done when line before you move on. Expect about an hour and a half, most of
it waiting on Google Cloud: the HTTPS certificate alone can take up to an hour.
Release: agent-relay-2026.11.16 (infra commit ece71b9d, chart 0.7.54).
The downloadable values.yaml renders against this release, which was not
installed on GKE. Read Known limits before a production rollout.
You need:
- Owner in the Google Cloud project (or the roles listed under
Who runs the install),
and
kubectladmin on the GKE cluster; - whoever manages DNS for your domain (step 6);
- the person who will be the first admin (step 15).
The cluster: GKE Standard with Workload Identity on, the HTTP load
balancing add-on on (the default), in a VPC with Private Services Access
configured: Cloud SQL and Memorystore get private IPs from it. For nodes, plan
on at least 3 × e2-standard-4 with 100 GB disks. Small nodes with 20 GB
disks run out of ephemeral storage while pulling the images.
No cluster or project yet? See Starting from an empty project
at the end, then come back to step 1. Creating a project needs
resourcemanager.projects.create on the organization (or folder) and
billing.user on the billing account; many organizations grant those to only a
few people, so ask early.
Network:
- Steps 1, 4, 5 and 6 need only Google Cloud access, not
kubectl. Start them while cluster access is being sorted out. - Nothing connects to the database from your machine. Every database step runs as a short-lived Job inside the cluster, so Cloud SQL needs no public IP.
1. Get the files
mkdir agent-relay-install && cd agent-relay-install
base=https://agentrelay.com/self-hosting/files/gke
curl -fsSLO "$base/SHA256SUMS"
for f in $(awk '{print $2}' SHA256SUMS); do curl -fsSLO "$base/$f"; done
shasum -a 256 -c SHA256SUMS
git clone https://github.com/AgentWorkforce/infra.git
git -C infra checkout agent-relay-2026.11.16
test "$(git -C infra rev-parse HEAD)" = ece71b9df93df64cb63676d2a0ca2767eccc6599 && echo "release ok"
mkdir release && gh release download agent-relay-2026.11.16 -R AgentWorkforce/infra -D release
(cd release && shasum -a 256 -c SHA256SUMS && tar -xzf agent-relay-0.7.54.tgz) | tail -1No gh? Download the same assets from the
release page
into release/.
Tools: gcloud (with gke-gcloud-auth-plugin), terraform 1.11 or later (or
OpenTofu), helm 3, kubectl, jq, yq, openssl, python3, and
crane for step 5.
Cloud Shell has everything except crane and a new enough Terraform.
Helm 3, not Helm 4. The chart needs Helm 3.10 or later. Helm 4 is
not supported yet. If helm version prints v4, install Helm 3 beside
it (for example as helm3) and use it for steps 10, 11 and 13.
Done when: every shasum line in the install folder ends in OK,
release ok prints, and the last line is release-notes.md: OK.
2. Connect to Google Cloud and the cluster
gcloud auth login && gcloud auth application-default login
gcloud config set project <project id>
gcloud services enable container.googleapis.com sqladmin.googleapis.com servicenetworking.googleapis.com \
redis.googleapis.com pubsub.googleapis.com secretmanager.googleapis.com dns.googleapis.com \
artifactregistry.googleapis.com compute.googleapis.com iamcredentials.googleapis.com
gcloud container clusters get-credentials <cluster> --location <region> # add --internal-ip for a private endpoint
kubectl auth can-i '*' '*' --all-namespaces
gcloud container clusters describe <cluster> --location <region> \
--format='value(workloadIdentityConfig.workloadPool,addonsConfig.httpLoadBalancing.disabled)'
gcloud container clusters describe <cluster> --location <region> \
--format='value(networkConfig.datapathProvider,networkPolicy.enabled)'
gcloud container node-pools list --cluster <cluster> --location <region> \
--format='table(name,config.workloadMetadataConfig.mode)'
gcloud compute addresses list --global --filter='purpose=VPC_PEERING' --format='value(name,network)'
kubectl get storageclassDone when:
can-iprintsyes;- the
describeline prints<project id>.svc.id.googand nothing after it (HTTP load balancing is not disabled); - every node pool shows
GKE_METADATA; - the
addressesline lists a Private Services Access range on your cluster's VPC; - one StorageClass is marked
(default); - the second
describeline printsADVANCED_DATAPATH(Dataplane V2), or it ends inTrue(the NetworkPolicy add-on) andkubectl -n kube-system get daemonset calico-nodeshows as many ready pods as the cluster has nodes. The add-on enforces policies only on nodes recreated after it was turned on.
Network policies must be enforced. The chart's NetworkPolicies, and
relaygate's rule that only the gate may reach Relaycast and Sessions, have no
effect on a cluster that doesn't enforce them: GKE accepts the objects and
ignores them. Dataplane V2 can only be chosen when a cluster is created. On an
existing cluster without it, turn on the add-on, then enforcement. If the
cluster already runs other workloads, check their policies first
(kubectl get networkpolicy -A): policies that were ignored start to apply.
gcloud container clusters update <cluster> --location <region> --update-addons=NetworkPolicy=ENABLED
gcloud container clusters update <cluster> --location <region> --enable-network-policyThe second command recreates every node pool's nodes, so pods restart. Each
pool follows its own upgrade settings (maxSurge, maxUnavailable), which
decide how many nodes restart at once. Check them first, and run the command
before you install or in a maintenance window.
Private control plane? Use the cluster's DNS endpoint
(get-credentials --dns-endpoint), or run the install from a machine inside the
VPC with --internal-ip.
billing account ... disabled from Cloud Storage or Service Networking?
Your Application Default Credentials bill API calls to a quota project that
has no billing. Point them at the install project:
gcloud auth application-default set-quota-project <project id>, or export
GOOGLE_CLOUD_QUOTA_PROJECT=<project id> and
CLOUDSDK_BILLING_QUOTA_PROJECT=<project id>.
3. Using your own Cloud SQL, bucket or other resources? (skip if Terraform creates them)
Follow Bring your own Google Cloud resources instead of step 4, then continue at step 5.
4. Create the Google Cloud resources (Terraform, about 5 minutes)
This creates a regional Cloud SQL for PostgreSQL 17 instance (private IP only, Google-managed shared CA, admin password in Secret Manager), the sessions bucket, a Standard HA Memorystore for Redis with TLS, a Pub/Sub topic and subscription with a dead-letter queue, a private DNS zone that resolves the Cloud SQL certificate name inside your VPC, and the Workload Identity grants. Every grant is on one resource, never on the project.
cd infra/terraform/examples/gcp
cat > terraform.tfvars <<'EOF'
project_id = "<project id>"
region = "<cluster region, e.g. us-central1>"
name = "agent-relay"
namespace = "agent-relay"
release_name = "agent-relay"
network_self_link = "projects/<project id>/global/networks/<cluster VPC>"
sessions_bucket_name = "<globally unique bucket name, e.g. acme-agent-relay-sessions>"
deletion_protection = true
image_mirror_repository = "agent-relay-mirror" # step 5's registry, readable by the GKE nodes
# desktop_feed_bucket_name = "<globally unique, e.g. acme-agent-relay-desktop-feed>" # optional: the desktop app's update feed
# gke_node_service_account = "<email>" # only if the node pool has its own service account
# kms_key_name = "projects/<p>/locations/<region>/keyRings/<ring>/cryptoKeys/<key>" # optional CMEK
labels = { app = "agent-relay" }
EOF
grep -n '<' terraform.tfvars
terraform init
terraform plan -out tfplan
terraform apply tfplan
terraform output -json chart_values > ../../../../managed-values.json
cd -Done when: the grep prints nothing, the plan shows 29 to add, 0 to change, 0 to destroy (5 more with desktop_feed_bucket_name, 2 fewer without
image_mirror_repository), the apply completes, and managed-values.json
exists.
desktop_feed_bucket_name (optional) gives your branded desktop app a
signed update feed in your project: a private bucket, and a Google service
account that relay-control uses through Workload Identity to write releases
there and sign download links. Creating that service account needs Service
Account Admin. managed-values.json carries the feed's settings into step 11.
If your values.yaml sets relay-control.extraEnv, copy the env entries
from terraform output desktop_feed into it: Helm replaces a list instead of
merging it. The feed also needs a signing key: see
Signed desktop update feed.
Where the Terraform state lives. The example has no backend block, so
terraform init keeps the state in a local file. To keep it in Cloud Storage
instead, create a private, versioned bucket and add a backend before
terraform init:
gcloud storage buckets create gs://<project id>-tfstate --location <region> \
--uniform-bucket-level-access --public-access-prevention
gcloud storage buckets update gs://<project id>-tfstate --versioning
printf 'terraform {\n backend "gcs" {}\n}\n' > backend_override.tf
terraform init -backend-config="bucket=<project id>-tfstate" -backend-config="prefix=agent-relay"Either way, keep the state: upgrades and teardown need it.
Error 403 ... servicenetworkingorno private services access connection: the VPC has no Private Services Access. Your network team configures it on the cluster's VPC; then apply again.- CMEK: first grant the Cloud SQL, Cloud Storage and Pub/Sub service agents
roles/cloudkms.cryptoKeyEncrypterDecrypteron the key (see Bring your own Google Cloud resources).
5. Mirror the images to Artifact Registry
Copy the release images, by digest, into the registry step 4 created. Run this from Cloud Shell or a VM in Google Cloud: there it takes about a minute, from a laptop it can take half an hour.
REGION=<region>
export MIRROR_ROOT=$(terraform -chdir=infra/terraform/examples/gcp output -raw image_mirror_root)
gcloud auth print-access-token | crane auth login $REGION-docker.pkg.dev -u oauth2accesstoken --password-stdin
for ref in $(jq -r '.images[].ref' release/release-manifest.json); do
crane copy "$ref" "$MIRROR_ROOT/${ref%@*}"
[ "$(crane digest "$MIRROR_ROOT/$ref")" = "${ref#*@}" ] && echo "ok ${ref%@*}"
done
sed "s|__IMAGE_MIRROR_ROOT__|$MIRROR_ROOT|g" release/release-images.values.yaml > release-images.values.yaml
grep -c "$MIRROR_ROOT" release-images.values.yamlThe GKE node service account pulls these images. Step 4 granted it
roles/artifactregistry.reader on this repository only: the project's default
Compute Engine service account, or gke_node_service_account. Check which
account your node pool uses:
gcloud container node-pools list --cluster <cluster> --location $REGION --format='value(name,config.serviceAccount)'default means the default Compute Engine service account. If it names another
account, set gke_node_service_account to it and apply step 4 again. In a new
project nothing else grants that read, and every pull fails with 403 (step
8's Job is the first pod that pulls).
Done when: MIRROR_ROOT is not empty, eight ok lines, and the grep -c
line prints a number above 10.
Using a registry Terraform doesn't manage? Leave image_mirror_repository
out of step 4, create the repository yourself, set
MIRROR_ROOT=$REGION-docker.pkg.dev/<project id>/<repository>, and grant the
node service account roles/artifactregistry.reader on it.
The release has eight images. A GKE install runs five of them (control API, dashboard, Relaycast, the sessions services and the PostgreSQL client); the other three are for other profiles, and mirroring them is harmless.
6. Reserve the IP address and point DNS at it (DNS owner)
The HTTPS certificate is a Google-managed certificate. Google issues it only after all four names resolve to the load balancer's address, so create the records now:
gcloud compute addresses create agent-relay-ingress --global --ip-version IPV4
gcloud compute addresses describe agent-relay-ingress --global --format='value(address)'Create A records for <domain>, api.<domain>, cast.<domain> and
history.<domain>, all pointing at that address.
Done when: dig +short api.<domain> prints the address, for all four names.
CAA records on your domain? They must allow pki.goog and letsencrypt.org.
7. Create the settings file
cat > settings.sh <<'EOF'
export DOMAIN=<app hostname, e.g. ar.example.com>
export ORG_ID=<stable id, e.g. org_acme: never change it>
export ORG_NAME='<organization name>'
export ADMIN_EMAIL=<first admin's email>
export GCP_PROJECT_ID=<project id>
export REGION=<region>
export MIRROR_ROOT=<region>-docker.pkg.dev/<project id>/agent-relay-mirror
export NAMESPACE=agent-relay
export HELM_RELEASE=agent-relay
export STATIC_IP_NAME=agent-relay-ingress
export SERVICES_IMAGE=$MIRROR_ROOT/ghcr.io/agentworkforce/sessions-pipeline@sha256:fd2efd61dca4761dfb55256aa4797cb7f61c3434cc8af4833287e0e7bb7b59a9
export PSQL_IMAGE=$MIRROR_ROOT/ghcr.io/cloudnative-pg/postgresql@sha256:ed6923f9ba2399cf9070379a353c710c29ced8a294be73986a2c3dae7e599ce3
export INSTALL_DIR="$PWD" # run "source settings.sh" from the agent-relay-install folder
export CHART="$INSTALL_DIR/release/agent-relay"
TF="$INSTALL_DIR/infra/terraform/examples/gcp"
PG="$(terraform -chdir="$TF" output -json postgres)"
export CLOUD_SQL_INSTANCE="$(jq -r '.connection_name | split(":")[2]' <<<"$PG")"
export CLOUD_SQL_HOST="$(jq -r .dns_name <<<"$PG")"
export CLOUD_SQL_ADMIN_USER="$(jq -r .admin_username <<<"$PG")"
export CLOUD_SQL_ADMIN_SECRET_ID="$(jq -r .admin_password_secret_id <<<"$PG")"
export CLOUD_SQL_CA_SHA256="$(jq -r '.["sessions-pipeline"].worker.postgresCa.sha256' "$INSTALL_DIR/managed-values.json")"
export SESSIONS_BUCKET="$(jq -r '.["sessions-pipeline"].worker.locker.bucket' "$INSTALL_DIR/managed-values.json")"
unset PG TF
EOF
source settings.sh
bash -c 'source ./settings.sh; for v in DOMAIN ORG_ID ORG_NAME ADMIN_EMAIL GCP_PROJECT_ID REGION MIRROR_ROOT CLOUD_SQL_INSTANCE CLOUD_SQL_HOST CLOUD_SQL_ADMIN_USER CLOUD_SQL_ADMIN_SECRET_ID CLOUD_SQL_CA_SHA256 SESSIONS_BUCKET; do [ -n "${!v}" ] && [ "${!v}" != null ] && [[ "${!v}" != *"<"* ]] || echo "MISSING: $v"; done; echo "settings checked"'Brought your own resources (step 3)? Replace the six lines between PG= and
unset with the values from
Bring your own Google Cloud resources.
Done when: only settings checked prints.
In every new terminal, first run cd agent-relay-install && source settings.sh.
8. Set up the databases
bash cloud-sql-setup.sh preIt creates the namespace, the cloud-sql-ca ConfigMap, the three databases and
their roles (in a Job inside the cluster), and writes every database credential
to ~/.agent-relay-install (mode 0700). It reads the Cloud SQL admin password
from Secret Manager and never prints it.
Done when: it ends with pre done, after tls: true and
extensions: btree_gin,pg_trgm,plpgsql,vector.
Copy ~/.agent-relay-install to your secret store: it holds the database
passwords. Never commit it. With Secret Manager, for example:
tar -C ~ -czf - .agent-relay-install | gcloud secrets create agent-relay-install-files \
--project $GCP_PROJECT_ID --replication-policy automatic --labels app=agent-relay --data-file=-Later copies add a version to the same secret: replace
secrets create ... --replication-policy automatic --labels app=agent-relay
with secrets versions add.
- It waits about 10 minutes, then says the Job did not succeed: look at the
Job's pod (
kubectl -n $NAMESPACE describe pod -l job-name=agent-relay-cloud-sql-pre).ImagePullBackOffor403 Forbiddenmeans the node service account can't read the mirror: grant it as in step 5, then runpreagain. Nothing was written to Cloud SQL, and the script recreates its Job. differs from CLOUD_SQL_CA_SHA256:managed-values.jsonis from an older apply. Rerun the last Terraform command of step 4, thensource settings.sh.- The Job is evicted (
low on resource: ephemeral-storage): the nodes' disks are too small (see "The cluster" above). - Run
preonce, before step 11. After step 13 it refuses to run again, by design.
9. Fill in values.yaml
sed -i.orig -e "s|__NAMESPACE__|$NAMESPACE|g" -e "s|__ORG_ID__|$ORG_ID|g" \
-e "s|__ORG_NAME__|$ORG_NAME|g" -e "s|__ADMIN_EMAIL__|$ADMIN_EMAIL|g" -e "s|__DOMAIN__|$DOMAIN|g" \
-e "s|__REGION__|$REGION|g" -e "s|__SESSIONS_BUCKET__|$SESSIONS_BUCKET|g" \
-e "s|__STATIC_IP_NAME__|$STATIC_IP_NAME|g" values.yaml
grep -n '__[A-Z_]*__' values.yamlDone when: the only line printed is __LLM_MODEL__. You set that one later,
when you turn on titles and summaries.
Optional: co-branding. The product stays Agent Relay; your logo goes beside it. The sign-in page shows your square mark under the Agent Relay lockup, and the dashboard shows one wide image with both logos. Put both PNGs in a ConfigMap before step 11:
kubectl -n $NAMESPACE create configmap relay-brand \
--from-file=mark.png=<your square mark>.png --from-file=cobrand.png=<Agent Relay + your logo>.pngThen uncomment the three "Optional co-branding" blocks in values.yaml
(global.branding, relay-control.login, relay-webapp.customerProfile) and
fill in their placeholders. For a branded desktop app,
customerProfile.desktop takes that build's link scheme and site.
$CHART/values-branding-example.yaml explains each value.
10. Create the secrets
This generates the remaining keys and creates the five Kubernetes Secrets. Nothing is printed.
S=~/.agent-relay-install/secret-files
( umask 077
for k in selfhost-bootstrap-token selfhost-setup-key relaycast-internal-secret \
relaycast-workspace-bootstrap-secret relaycontrol-introspect-secret session-read-api-secret \
selfhost-encryption-key; do
[ -s "$S/agent-relay-secrets/$k" ] || openssl rand -hex 32 | tr -d '\n' > "$S/agent-relay-secrets/$k"
done
[ -s "$S/agent-relay-secrets/SUMMARY_MASTER_KEYS" ] || printf 'k1:%s' "$(openssl rand -hex 32)" > "$S/agent-relay-secrets/SUMMARY_MASTER_KEYS"
cp "$S/agent-relay-secrets/relaycontrol-introspect-secret" "$S/agent-relay-ingest-edge-runtime/introspect-secret" )
put_secret() { # create the Secret, or replace it in place; never kubectl apply, which copies
# every value into a last-applied annotation (replace also drops an old one)
if kubectl -n $NAMESPACE get secret "$1" >/dev/null 2>&1; then
kubectl -n $NAMESPACE create secret generic "$@" --dry-run=client -o yaml | kubectl replace -f -
else
kubectl -n $NAMESPACE create secret generic "$@"
fi
}
for name in agent-relay-secrets agent-relay-read-api agent-relay-ingest-edge-runtime \
search-indexer-runtime search-indexer-owner; do
put_secret "$name" --from-file="$S/$name"
doneDone when: five lines end in created. Copy ~/.agent-relay-install to
your secret store again (step 8 shows how): it now holds the encryption keys.
Keep your secrets in SOPS (for example with Cloud KMS) or Google Secret Manager? Use one of the options in Secrets instead of this block. vals works with Google Secret Manager.
11. Install with Helm
helm upgrade --install $HELM_RELEASE "$INSTALL_DIR/release/agent-relay-0.7.54.tgz" --namespace $NAMESPACE \
-f "$CHART/values-gcp.yaml" -f "$CHART/values-gcp-native.yaml" \
-f "$INSTALL_DIR/managed-values.json" -f "$INSTALL_DIR/values.yaml" \
-f "$INSTALL_DIR/release-images.values.yaml" --timeout 25mThe order matters: the two GCP profiles, then Terraform's managed services,
then your site values, then the release images last. Without
release-images.values.yaml you would get the previous release's images.
Done when: it prints STATUS: deployed. Don't add --wait.
12. Wait for the database migrations
for j in $(kubectl -n $NAMESPACE get jobs -o name | grep -E 'relay-control-migrate|relaycast-migrate|agent-relay-worker-migrate'); do
kubectl -n $NAMESPACE wait --for=condition=complete "$j" --timeout=20m
doneDone when: three lines say condition met (about 2 minutes).
Expected at this point: the read API, ingest edge, search indexer and the
search-indexer-bootstrap Job fail. Step 13 fixes them.
13. Switch on the service logins
bash cloud-sql-setup.sh post
kubectl -n $NAMESPACE rollout restart deploy/agent-relay-read-api deploy/agent-relay-ingest-edge deploy/agent-relay-search-indexer
for j in $(kubectl -n $NAMESPACE get jobs -o name | grep search-indexer-bootstrap); do kubectl -n $NAMESPACE delete "$j"; done
helm upgrade --install $HELM_RELEASE "$INSTALL_DIR/release/agent-relay-0.7.54.tgz" --namespace $NAMESPACE \
-f "$CHART/values-gcp.yaml" -f "$CHART/values-gcp-native.yaml" \
-f "$INSTALL_DIR/managed-values.json" -f "$INSTALL_DIR/values.yaml" \
-f "$INSTALL_DIR/release-images.values.yaml" --timeout 25m
kubectl -n $NAMESPACE wait --for=condition=complete "$(kubectl -n $NAMESPACE get jobs -o name | grep search-indexer-bootstrap)" --timeout=10m
kubectl -n $NAMESPACE rollout status deploy --timeout=10m
kubectl -n $NAMESPACE get podsDone when:
post doneprints after threelogin ok ... tls=truelines;- the bootstrap Job shows
condition met; - every pod is
Runningand ready (completed Jobs showCompleted).
PostgresTlsConfigError in the read API, ingest edge or indexer logs: the
cloud-sql-ca ConfigMap doesn't match the pin in managed-values.json.
Compare kubectl -n $NAMESPACE get cm cloud-sql-ca -o jsonpath='{.data.ca\.crt}' | sha256sum
with echo $CLOUD_SQL_CA_SHA256.
14. Wait for the certificate and check (up to 60 minutes)
kubectl -n $NAMESPACE get managedcertificate agent-relay -o jsonpath='{.status.certificateStatus}{"\n"}'Repeat until it prints Active. Then:
curl -fsS "https://api.$DOMAIN/readyz" && echo " api ok"
curl -fsS "https://cast.$DOMAIN/health" && echo " cast ok"
curl -fsS -o /dev/null -w "dashboard %{http_code}\n" "https://$DOMAIN/"
curl -s -o /dev/null -w "history %{http_code}\n" "https://history.$DOMAIN/v1/sessions"Done when: Active, then api ok, cast ok, dashboard 200 and
history 401.
- Still
Provisioningafter an hour: one of the four names doesn't resolve to the address from step 6, or a CAA record blocks Google. Fix DNS; the certificate retries by itself. FailedNotVisible: the same: DNS doesn't point at the load balancer yet.- 502 for a few minutes after
Active: the load balancer's health checks are still converging. Wait 5 minutes.
15. First sign-in, then add users
The first admin does this before signing in any other way, including from the desktop app:
kubectl -n $NAMESPACE get secret agent-relay-secrets -o jsonpath='{.data.selfhost-setup-key}' | base64 -d; echo- Open
https://<domain>/loginand choose First-time setup: enter setup key. - Paste the key and choose a password.
No kubectl from the admin's machine? With a private control plane, the
first admin often can't reach the cluster. Whoever runs the install does the
same first sign-in through the API instead, without the key or the password
ever being shown:
bash first-admin.sh "$DOMAIN" "$GCP_PROJECT_ID" "$NAMESPACE"It stores a generated password in Secret Manager as
agent-relay-first-admin-password, then posts it with the setup key, and
prints first admin set. The admin then reads the password with
gcloud secrets versions access latest --secret agent-relay-first-admin-password --project <project id>
and signs in at https://<domain>/login. The setup key works once, within 24
hours: the script never retries, and it stops if the secret already exists. A
403 csrf_rejected means the request lacked your dashboard's origin; it is
refused before the key is read, so the key stays unused.
Then add users:
bash add-user.sh "$DOMAIN"It prints a temporary password for each user; they choose their own at first sign-in.
Done when: the admin is signed in to the dashboard.
16. Turn on agent search (once, after first sign-in)
Searching Sessions for @<agent name> answers "The session agent index isn't
ready yet" (501 agent_filter_not_ready from the API) until this one-time
activation runs. It checks that every session's agent names are indexed, then
marks the index ready; later sessions are indexed as they arrive. Run it once
the first session has uploaded, or on a brand-new install with
--fresh-install (below).
16a. Get the script and plan (read-only). It runs in a pod in the
cluster as the migration owner: the URL comes from HOOPDB_MIGRATION_URL in
agent-relay-secrets and never reaches your shell. The script isn't in the
services image, so this fetches it from the commit the image was built from:
REV="$(crane config "ghcr.io/agentworkforce/sessions-pipeline@${SERVICES_IMAGE#*@}" | jq -r '.config.Labels["org.opencontainers.image.revision"]')"
curl -fsSL -o catalog-activation.ts "https://raw.githubusercontent.com/AgentWorkforce/relayhistory-pipeline/$REV/scripts/native-agent-catalog-activation.ts"
kubectl -n $NAMESPACE create configmap agent-catalog-activation --from-file=catalog-activation.ts --dry-run=client -o yaml | kubectl apply -f -
catalog() { # catalog --plan | catalog --apply <fingerprint>
[ -n "$NAMESPACE" ] && [ -n "$SERVICES_IMAGE" ] || { echo "missing settings: run 'source settings.sh' (step 7) first"; return 1; }
ARGS="$(printf ', "%s"' "$@")"
kubectl -n $NAMESPACE delete pod agent-catalog --ignore-not-found
kubectl -n $NAMESPACE apply -f - <<YAML
apiVersion: v1
kind: Pod
metadata: { name: agent-catalog }
spec:
restartPolicy: Never
automountServiceAccountToken: false
securityContext: { runAsNonRoot: true, runAsUser: 1000, runAsGroup: 1000, fsGroup: 1000, seccompProfile: { type: RuntimeDefault } }
volumes:
- { name: tmp, emptyDir: { sizeLimit: 64Mi } }
- { name: ca, configMap: { name: cloud-sql-ca } }
- { name: script, configMap: { name: agent-catalog-activation } }
containers:
- name: catalog
image: $SERVICES_IMAGE
workingDir: /app
command: [bun, /app/scripts/native-agent-catalog-activation.ts$ARGS]
env:
- { name: HOOPDB_DB_MODE, value: selfhosted }
- { name: HOOPDB_URL, valueFrom: { secretKeyRef: { name: agent-relay-secrets, key: HOOPDB_MIGRATION_URL } } }
- { name: NODE_EXTRA_CA_CERTS, value: /etc/catalog-ca/ca.crt }
- { name: HOME, value: /tmp }
securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } }
volumeMounts:
- { name: tmp, mountPath: /tmp }
- { name: ca, mountPath: /etc/catalog-ca, readOnly: true }
- { name: script, mountPath: /app/scripts/native-agent-catalog-activation.ts, subPath: catalog-activation.ts, readOnly: true }
YAML
t=0
until p="$(kubectl -n $NAMESPACE get pod agent-catalog -o jsonpath='{.status.phase}')"; [ "$p" = Succeeded ] || [ "$p" = Failed ]; do
sleep 3; t=$((t + 3))
[ $t -lt 300 ] || { echo "agent-catalog still $p after 5 min"; kubectl -n $NAMESPACE describe pod agent-catalog | tail -20; return 1; }
done
kubectl -n $NAMESPACE logs agent-catalog | tail -1
rc="$(kubectl -n $NAMESPACE get pod agent-catalog -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}')"
echo "exit $rc"; [ "$rc" = 0 ]
}
catalog --planDone when: one JSON line, then exit 0 or exit 2. The plan is read-only
and safe to run any time. Its line shows "eligible" and a 64-character
fingerprint; "eligible":false with only unprojected above 0 is fine at
this point.
16b. Pause uploads, plan, apply, resume (about a minute). Apply only
works on a plan made while nothing writes, so stop uploads, give the indexer a
minute to catch up, then stop it too. pause waits until a Deployment's pods
are gone; resume puts back the replica counts read here:
IE="$(kubectl -n $NAMESPACE get deploy agent-relay-ingest-edge -o jsonpath='{.spec.replicas}')"
SI="$(kubectl -n $NAMESPACE get deploy agent-relay-search-indexer -o jsonpath='{.spec.replicas}')"
pause() {
sel="$(kubectl -n $NAMESPACE get deploy "$1" -o json | jq -r '.spec.selector.matchLabels | to_entries | map("\(.key)=\(.value)") | join(",")')"
[ -n "$sel" ] && kubectl -n $NAMESPACE scale deploy/"$1" --replicas=0 || return 1
until [ -z "$(kubectl -n $NAMESPACE get pods -l "$sel" -o name)" ]; do sleep 3; done
}
resume() {
kubectl -n $NAMESPACE scale deploy/agent-relay-search-indexer --replicas=$SI
kubectl -n $NAMESPACE scale deploy/agent-relay-ingest-edge --replicas=$IE
kubectl -n $NAMESPACE rollout status deploy/agent-relay-search-indexer deploy/agent-relay-ingest-edge --timeout=10m
}
if [ -n "$IE" ] && [ -n "$SI" ]; then
pause agent-relay-ingest-edge && sleep 60 && pause agent-relay-search-indexer && catalog --plan
else
echo "couldn't read the replica counts; nothing was scaled"
fiCheck the plan shows "eligible":true, then apply that plan's fingerprint.
Writers resume and the script is removed only if the apply succeeds:
catalog --apply <fingerprint from the plan above> && resume &&
kubectl -n $NAMESPACE delete pod/agent-catalog configmap/agent-catalog-activationDone when: the plan shows "eligible":true, the apply shows
"activated":true and exit 0, both rollouts finish, and searching Sessions
for @<agent name> lists sessions (or none) instead of "The session agent
index isn't ready yet".
- Brand-new install, no sessions yet: the plan shows
"liveRows":0and"eligible":false. Either upload a session first, or add--fresh-installas the last argument to both:catalog --plan --fresh-install, thencatalog --apply <fingerprint> --fresh-install. Use it only on an install that has never had a session, deleted ones included; anywhere else the plan stays ineligible. - It briefly pauses writers. Uploads are refused while
agent-relay-ingest-edgeis at 0, and the apply itself locks the session index (it waits at most 1 second for those locks). Expect about a minute. Run it at a quiet time, and before anyone deletes a session. - To give up or try later: run
resume. Uploads stay off until you do. unprojectedabove 0 in the paused plan: the indexer hadn't caught up. Runkubectl -n $NAMESPACE scale deploy/agent-relay-search-indexer --replicas=$SI, wait a minute, thenpause agent-relay-search-indexer && catalog --plan.native_catalog_plan_changed: something wrote between plan and apply. Plan again and apply the new fingerprint.native_catalog_operation_failedwithexit 1from apply: usually a writer still held the locks. Nothing was written and writers are still paused; run the samecatalog --apply ... && resume ...line again.native_catalog_schema_validation_required: the pod couldn't reach the database, or the migrations (step 12) haven't finished.- Any other count above 0 (for example
unknownUsers): runresumeand send us the JSON line.
Next
- Before people start: in the dashboard, under Settings › Policies, set what the desktop app uploads.
- Desktop app: Desktop app rollout.
- Search: transcript search is stored in Cloud SQL (
search.backend: postgres); there is no search cluster to run. - Problems: Troubleshooting, then Support.
Signed desktop update feed (optional)
With a feed, your branded desktop app offers each new version to its users and installs it when they choose to update. Without one, a new version means installing the new build over the old one. Agent Relay builds and publishes your desktop app, and gives you its profile ID and the SHA-256 hash of the token it publishes with. The token itself stays with Agent Relay.
-
Bucket: set
desktop_feed_bucket_namein step 4'sterraform.tfvars, then, from the install folder, apply it and refreshmanaged-values.jsonso it carries the feed's settings:(cd infra/terraform/examples/gcp && terraform plan -out tfplan && terraform apply tfplan \ && terraform output -json chart_values > ../../../../managed-values.json) -
Signing key: create an Ed25519 key for the profile and store it in the cluster. The private key never leaves your cluster and Secret Manager:
PROFILE=<profile id from Agent Relay>; BACKUP=agent-relay-desktop-feed-key-$PROFILE; umask 077 if out=$(gcloud secrets describe $BACKUP --format='value(name)' 2>&1); then BK=yes elif printf '%s' "$out" | grep -q NOT_FOUND; then BK=no else BK=error; fi if [ $BK = error ]; then echo "STOP: could not check for the backup $BACKUP (sign in to gcloud and run this again)" else if ! kubectl -n $NAMESPACE get secret desktop-feed-keys >/dev/null 2>&1; then if [ $BK = yes ]; then gcloud secrets versions access latest --secret $BACKUP > "$PROFILE.pem" # restore the saved key else openssl genpkey -algorithm ed25519 -out "$PROFILE.pem" fi if openssl pkey -in "$PROFILE.pem" -noout 2>/dev/null; then kubectl -n $NAMESPACE create secret generic desktop-feed-keys --from-file="$PROFILE.pem" fi rm -f "$PROFILE.pem" fi kubectl -n $NAMESPACE get secret desktop-feed-keys -o jsonpath="{.data.$PROFILE\.pem}" | base64 -d > "$PROFILE.pem" if openssl pkey -in "$PROFILE.pem" -noout 2>/dev/null && [ $BK = no ]; then gcloud secrets create $BACKUP --data-file="$PROFILE.pem" && BK=yes fi if openssl pkey -in "$PROFILE.pem" -noout 2>/dev/null && [ $BK = yes ] \ && gcloud secrets versions access latest --secret $BACKUP | cmp -s - "$PROFILE.pem"; then openssl pkey -in "$PROFILE.pem" -pubout -outform DER | tail -c 32 | base64 else echo "STOP: the cluster key is missing, unreadable or differs from its backup" fi rm -f "$PROFILE.pem" fiThis needs OpenSSL 1.1.1 or later, as in Cloud Shell. The
opensslthat ships with macOS (LibreSSL) has no Ed25519; use Homebrew'sopenssl@3. The last line prints the public key. Send it to Agent Relay, which builds it into the app. The key is created once. Running the block again reuses the key in the cluster, or restores it from the Secret Manager backup if the cluster lost it, so desktop builds keep trusting it. It makes a new key only when Secret Manager answers that no backup exists, and stores a key only after checking it is valid. If it printsSTOP, check that you can read the backup (gcloud secrets versions access latest --secret $BACKUP >/dev/null && echo readable, which never shows the key); if both copies exist and differ, keep the one whose public key Agent Relay has, and ask Agent Relay before changing either. -
Values: add this to
values.yaml, with the hash from Agent Relay, then run step 11'shelm upgradecommand again:relay-control: desktopFeed: signingKeys: existingSecret: desktop-feed-keys profiles: [<profile id>] extraEnv: - { name: SELFHOST_DESKTOP_FEED_KEY_LABEL, value: f1 } - { name: SELFHOST_DESKTOP_FEED_PUBLISHER_TOKEN_HASHES, value: '{"<profile id>":["<sha256 hex>"]}' }Keep the feed entries from
terraform output desktop_feedin the sameextraEnvlist: this list replaces the one inmanaged-values.json. -
Register the profile: an organization admin registers the profile and binds the workspaces whose users get the app, with Agent Relay. The public key the control API returns must match the one from step 2.
Done when: kubectl -n $NAMESPACE get pods shows relay-control ready after
the upgrade, and after Agent Relay publishes the first version to your feed,
the dashboard's Downloads card lists it.
Re-stamp sessions uploaded before 2026.11.6 (upgrades only)
Releases 2026.11.1 to 2026.11.5 recorded sessions uploaded to GCS as if they
were in S3, so the read API refuses them (501). 2026.11.6 records new uploads
correctly; this one-off tool fixes the rows written before. A fresh install of
2026.11.6 or later doesn't need it. It changes only rows whose objects it finds in your
bucket, writes nothing in plan mode, and is safe to rerun.
Run it in a terminal where you ran cd agent-relay-install && source settings.sh
(step 7): it needs NAMESPACE, REGION and SESSIONS_BUCKET from there.
IMG="$(kubectl -n $NAMESPACE get deploy agent-relay-read-api -o jsonpath='{.spec.template.spec.containers[0].image}')"
CA_SHA="$(kubectl -n $NAMESPACE get deploy agent-relay-read-api -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="SESSION_POSTGRES_CA_SHA256")].value}')"
restamp() { # restamp plan | restamp apply
[ -n "$NAMESPACE" ] && [ -n "$REGION" ] && [ -n "$SESSIONS_BUCKET" ] && [ -n "$CA_SHA" ] && [ -n "$IMG" ] \
|| { echo "missing settings: run 'source settings.sh' (step 7) first"; return 1; }
APPLY=""; [ "$1" = apply ] && APPLY=", --apply"
kubectl -n $NAMESPACE delete pod index-restamp --ignore-not-found
kubectl -n $NAMESPACE apply -f - <<YAML
apiVersion: v1
kind: Pod
metadata: { name: index-restamp }
spec:
restartPolicy: Never
serviceAccountName: agent-relay-read-api
automountServiceAccountToken: false
securityContext: { runAsNonRoot: true, runAsUser: 1000, runAsGroup: 1000, fsGroup: 1000, seccompProfile: { type: RuntimeDefault } }
volumes:
- { name: tmp, emptyDir: { sizeLimit: 64Mi } }
- { name: ca, configMap: { name: cloud-sql-ca } }
- { name: url, secret: { secretName: agent-relay-secrets, defaultMode: 0400, items: [ { key: HOOPDB_MIGRATION_URL, path: url } ] } }
containers:
- name: restamp
image: $IMG
command: [bun, run, /app/src/session-pipeline/index-provider-restamp.ts$APPLY]
env:
- { name: LOCKER_PROVIDER, value: gcs }
- { name: LOCKER_BUCKET, value: "$SESSIONS_BUCKET" }
- { name: LOCKER_REGION, value: "$REGION" }
- { name: SESSION_POSTGRES_TLS_PROFILE, value: custom-ca }
- { name: SESSION_POSTGRES_CA_FILE, value: /etc/restamp-ca/ca.crt }
- { name: SESSION_POSTGRES_CA_SHA256, value: "$CA_SHA" }
- { name: INDEX_PROVIDER_RESTAMP_DATABASE_URL_FILE, value: /run/restamp/url }
- { name: HOME, value: /tmp }
securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } }
volumeMounts:
- { name: tmp, mountPath: /tmp }
- { name: ca, mountPath: /etc/restamp-ca, readOnly: true }
- { name: url, mountPath: /run/restamp, readOnly: true }
YAML
t=0
until p="$(kubectl -n $NAMESPACE get pod index-restamp -o jsonpath='{.status.phase}')"; [ "$p" = Succeeded ] || [ "$p" = Failed ]; do
sleep 5; t=$((t + 5))
[ $t -lt 600 ] || { echo "index-restamp still $p after 10 min"; kubectl -n $NAMESPACE describe pod index-restamp | tail -20; return 1; }
done
kubectl -n $NAMESPACE logs index-restamp | tail -1
kubectl -n $NAMESPACE get pod index-restamp -o jsonpath='exit {.status.containerStatuses[0].state.terminated.exitCode}{"\n"}'
}
restamp planCheck the plan's JSON line, and keep a copy of it: restamp apply replaces the
pod, so its output is gone afterwards. scanned is the rows still recorded as
S3, proven those whose objects are in your bucket (they will be fixed) and
unproven those left alone. Then apply it:
restamp apply
kubectl -n $NAMESPACE delete pod index-restampDone when: the last JSON line shows "complete":true with restamped
equal to the proven in the plan line you kept, and exit 0.
-
missing settings: this terminal hasn't runsource settings.sh. -
still Pending after 10 min: thedescribeoutput shows why (usually an image pull from your mirror). Fix it, then run the same command again. -
exit 3: incomplete (a page limit, or a row refused mid-run). Runrestamp applyagain. -
exit 5: done, butguard_skippedrows were kept by a database guard (an erase fence, or a publication still in progress). Rerunning won't change them; send us the JSON line. -
exit 2: misconfigured (for exampleLOCKER_BUCKETis empty). Nothing was read or written. -
exit 4: any other error; checkkubectl -n $NAMESPACE logs index-restampand run it again.
Known limits
What release agent-relay-2026.11.16 doesn't do on GKE yet:
- Upgrading from 2026.11.5 or earlier: re-stamp once. Those releases
recorded sessions uploaded to Cloud Storage as if they were in S3, so their
reads answer
501. After the upgrade, run Re-stamp sessions uploaded before 2026.11.6 once. A fresh install doesn't need it. - Long sessions need the longer read deadline. The read API reads a
session's chunks one after another (about 30 ms each from Cloud Storage), so
the default 3 s deadline fails sessions of more than about 100 chunks
(
501 turns_canonical_deadline). The downloadablevalues.yamlsetsREAD_API_CANONICAL_DEADLINE_MS=15000; keep it. Parallel reads come in a later release. /turnsand/metadataare empty on self-host. They serve Agent Relay conversation-turn records, which only the hosted service produces, so/turnsanswers with no turns and/metadatawith404./eventsand the dashboard carry the whole transcript.- No automated install or upgrade. The install is the manual sequence above, and upgrades are manual Helm upgrades with a Cloud SQL backup first.
- Search runs in PostgreSQL, not OpenSearch. Fine for a first install, but not sized for search at large scale.
- Memorystore without AUTH. The services can't send Redis credentials, so the instance uses TLS with AUTH off. Keep it reachable only from the cluster's VPC.
- Not highly available. The control API, dashboard, Relaycast and search indexer each run one replica.
- Not covered by this guide: GKE Autopilot, Shared VPC, and titles and summaries with a model.
Starting from an empty project
The steps above assume a GKE cluster in a VPC with Private Services Access. If you have neither, this creates a dedicated project with a minimal shape that meets "The cluster" at the top: private nodes, Cloud NAT for their outbound traffic, and a public control plane open only to the addresses you list. Change the ranges if they overlap your other networks.
P=<project id>; R=<region, e.g. us-central1>
gcloud projects create $P --organization <organization id> --labels app=agent-relay
gcloud billing projects link $P --billing-account <billing account id>
gcloud config set project $P
gcloud services enable container.googleapis.com compute.googleapis.com servicenetworking.googleapis.com \
cloudresourcemanager.googleapis.com iam.googleapis.com storage.googleapis.com
gcloud compute networks create agent-relay-vpc --subnet-mode custom
gcloud compute networks subnets create agent-relay-subnet --network agent-relay-vpc --region $R \
--range 10.10.0.0/20 --secondary-range pods=10.20.0.0/16,services=10.30.0.0/20 \
--enable-private-ip-google-access
gcloud compute addresses create agent-relay-psa --global --purpose VPC_PEERING \
--prefix-length 16 --network agent-relay-vpc --addresses 10.40.0.0
gcloud services vpc-peerings connect --service servicenetworking.googleapis.com \
--network agent-relay-vpc --ranges agent-relay-psa
gcloud compute routers create agent-relay-router --network agent-relay-vpc --region $R
gcloud compute routers nats create agent-relay-nat --router agent-relay-router --region $R \
--auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges
gcloud container clusters create agent-relay --location $R --num-nodes 1 \
--machine-type e2-standard-4 --disk-type pd-balanced --disk-size 100 \
--network agent-relay-vpc --subnetwork agent-relay-subnet --enable-ip-alias \
--cluster-secondary-range-name pods --services-secondary-range-name services \
--workload-pool=$P.svc.id.goog --workload-metadata=GKE_METADATA \
--addons HttpLoadBalancing,GcePersistentDiskCsiDriver --release-channel regular \
--enable-private-nodes --master-ipv4-cidr 172.16.0.0/28 \
--enable-master-authorized-networks --master-authorized-networks <your IP>/32 \
--enable-shielded-nodes --enable-dataplane-v2A regional cluster with --num-nodes 1 gets one node in each of the region's
three zones. Then continue at step 1 with <cluster> = agent-relay and
<cluster VPC> = agent-relay-vpc, and grant the node service account in step 5.
Run everything that needs kubectl from an address in the authorized list.
Done when: step 2's checks pass on the new cluster.