tembo-agent-host module for Google Cloud, and use its outputs to configure the chart.
Before you run Terraform
Starting from an empty project? The module’s complete example creates everything below: a VPC with a subnetwork for hosts, Cloud NAT with dynamic port allocation, and a private GKE Autopilot cluster that hosts can reach.Projects and APIs
- A project for agent hosts. The host controller can create instances anywhere in the hosts’ project, and change or delete only its own. We recommend a project of its own for agent hosts: set it as the module’s
host_project_id. Otherwise hosts run in the cluster’s project. - APIs enabled in those projects: Compute Engine, IAM, IAM Service Account Credentials and Cloud Storage (
compute.googleapis.com,iam.googleapis.com,iamcredentials.googleapis.com,storage.googleapis.com).
gcloud reports RESOURCE_EXHAUSTED for Resource Manager reads against a shared consumer project, use a project you control for CLI quota. Enable the APIs there before setting the quota project:
serviceusage.services.use on that project. This changes API request quota attribution, not Compute Engine CPU or disk quotas. Newly enabled APIs can take a short time to become usable; retry a SERVICE_DISABLED response after propagation.
Cluster
- A VPC-native GKE cluster (Autopilot or Standard) with Workload Identity. Autopilot clusters always have it.
- Your access to the control plane. Check authorized networks on a new cluster: an empty allowlist can leave
kubectlhanging, and enforcement on the private endpoint can also block agent hosts. Use the DNS endpoint for administrators and authorize the hosts’ subnet for the private endpoint (see below). - Ingress: the chart serves Tembo through GKE’s Ingress with a Google-managed certificate (
ingress.gcp.enabled), so you don’t need a certificate in advance. GKE issues it once your hostname resolves to the Ingress.
VPC and subnetwork
Agent hosts launch into a subnetwork you choose, in the cluster’s VPC:- Cloud NAT and Private Google Access. Hosts have no external addresses. They need NAT for egress to your Tembo URL, Tembo’s API, LLM providers and code hosts such as GitHub, and Private Google Access for Cloud Storage and IAM.
-
Dynamic port allocation on the NAT, with at least 1,024 ports per VM. A host carries the egress of every session on it, and Cloud NAT’s default of 64 ports per VM runs out with a couple of sessions:
- The cluster’s control plane reachable from it. Hosts read their configuration through the Kubernetes API, at the cluster’s private endpoint. If the cluster enforces master authorized networks on its private endpoint, add the subnetwork’s range to them. Hosts in a different region than the cluster need the control plane’s global access.
- Your own Postgres or Redis, if you use them instead of the chart’s, must accept connections from the subnetwork. Hosts reach the chart’s bundled Postgres and Redis through internal load balancers.
container.clusters.connect permission and the gke-gcloud-auth-plugin installed. Agent hosts still use the private IP endpoint. If it enforces authorized networks, add their subnet’s CIDR:
Organization policies
compute.disableNestedVirtualizationnot enforced on the hosts’ project: hosts run agents in VMs.compute.requireShieldedVmnot enforced: the agent host image boots with legacy BIOS.- If you restrict
compute.trustedImageProjects, allow Tembo’s image project,projects/tembo-prod.
Quotas
Each host needs 32 vCPUs and 1,500 GB of local SSD, times the pool’s maximum hosts (three by default). Free Trial billing accounts can’t run a single host or request more quota: upgrade to a paid account first.- N2 CPUs and local SSD in the hosts’ region. The defaults often cover a few hosts.
- Persistent Disk SSD in the hosts’ region: each host’s 100 GB boot disk counts toward it, as do GKE’s node disks. New-project quotas can be as low as 250 GB.
- CPUs (all regions), a project-wide limit that also counts the cluster’s nodes. New-project quotas can be as low as 12 vCPUs, which isn’t enough for a single host.
Quota request denied.
Start with one host on a quota-limited project. Add this to the values.yaml you pass after the module’s values:
e2-standard-4 nodes (12 vCPUs). Allow headroom for node upgrades too. Autopilot scales its nodes automatically and also consumes project quota; use Standard with a capped node pool when you need a predictable CPU budget. A regional node pool’s size applies per zone.
Raise maxHosts only after quota increases are approved. Hosts beyond your quota don’t launch; replacing a host without interrupting sessions also needs room for the replacement (see Apply changes).
Permissions to apply
Apply the module as the projects’ Owner, or with these roles, besides read access to the cluster (roles/container.clusterViewer):
Editor isn’t enough: it can’t create the controller’s custom roles or set IAM on the subnetwork.
Install the module
- The host controller’s service account, bound to the chart’s service account with Workload Identity. It can change and delete only instances and disks named with your install’s prefix, launch them only into your subnetwork, and attach only the hosts’ service account.
- The agent hosts’ service account, with access to the install’s bucket.
- The firewall rule for the hosts’ manager.
- A GCS bucket for project images, session snapshots and file uploads, and the control plane’s service account to use it. To use a bucket you have, set
object_storage_bucket_name.
kubernetes_namespace and release_name to the namespace and release you’ll install the chart with: the service accounts trust only the chart’s service accounts in that namespace. See the module’s README for the other options, such as machine types.
New IAM bindings can take a few minutes to take effect; pods that start before then retry.
Configure the chart with its outputs
Save the Helm values, including the source ranges derived from your host subnet in the output above:- GKE’s Ingress for the chart (
ingress.gcp). - The host controller, with its service account, the hosts’ project and your install’s instance name prefix.
- A host class named
default, with your subnetwork, the hosts’ service account,n2-custom-32-65536with four local SSDs, and Tembo’sstableagent host image. - A pool named
default, of one to three hosts. - Object storage, with the bucket and the control plane’s service account.
tembo.json first and your own values.yaml after it, so yours override them. To change the agent hosts, such as their machine type, see Agent hosts; to use managed Postgres or Redis, see Postgres and Redis.
Next, set up GitHub.
DNS
Use Cloud DNS private zones attached to the hosts’ VPC forpostgres.<your-host>, redis.<your-host> and sandbar.<your-host>. A zone for each internal hostname avoids shadowing the public app and preview names. With Shared VPC, attach the zones to the shared network and grant the DNS administrator access in the zone’s project.
Configure external-dns for Google Cloud DNS to watch Services and manage these zones. The chart already annotates each internal Service with its hostname, so external-dns creates and updates the records when GKE assigns load balancer addresses. Keep public and private zone filters separate.
The Terraform module currently creates neither these zones nor external-dns; use your existing DNS automation where available. Without external-dns, create private records from the Service addresses shown by kubectl -n tembo get svc. Public app and preview records can stay with your existing DNS provider.
Ingress and certificates
We use GKE’s built-in Ingress, external-dns and cert-manager, with a Let’s Encrypt ClusterIssuer that solves DNS-01 in Cloud DNS.- Load balancers:
ingress.gcpsets up both Ingresses for GKE, with health checks and the HTTPS redirect. To serve on a reserved IP, add akubernetes.io/ingress.global-static-ip-nameannotation. - Certificates: GKE gives
tembo.example.coma Google-managed certificate, and issues one foringress.public.hostwhen you enable the public Ingress. Those can’t be wildcards, so cert-manager issues*.previews.tembo.example.comintotembo-previews-tls. Use your ClusterIssuer’s name in the annotation.
Sign in with Google
Google sign-in works on any cloud. You need your Tembo hostname, permission to create a Google OAuth client, and access to thetembo Kubernetes namespace. Create the namespace first if you have not deployed yet.
- In the Google Cloud console, open Google Auth Platform and set up your app. Choose Internal under Audience to allow only your Google Workspace’s users.
-
Under Clients, create a client of type Web application, with the authorized redirect URI
https://<your-host>/api/auth/callback/google. -
Save the client secret when Google first shows it, then store it with the client ID in a Secret:
-
Add it to your
values.yaml, then install, or upgrade if Tembo is already installed (see Apply changes):