Skip to main content
Agent sessions run outside your cluster, on agent hosts: cloud instances that run each session in its own virtual machine. The chart’s host controller runs in your cluster and launches, tracks and replaces them. Agent hosts are EC2 instances on AWS, or Compute Engine instances on GCP, started from Tembo’s agent host image. The host controller gets it with your license key, and Tembo shares it with your AWS account or the controller’s Google service account. They need nested virtualization, and they keep VM disks, snapshots and saved memory on the instance’s local NVMe.

Classes and pools

You describe agent hosts in your Helm values, with two kinds of resources the chart creates. The Terraform module’s Helm values create one of each, named default.
  • A host class says what to launch: the image, instance types, subnets, security groups, instance profile and root volume.
  • A pool says how many to run: minHosts, maxHosts, and minIdleMemory, the free memory to keep for new sessions.
The controller creates a host resource for each instance it launches. List them with:

Lifecycle

  1. Launch. The controller starts an instance with the class’s settings and tags it as its own.
  2. Configure. The host reads its configuration from a Kubernetes Secret through the cluster’s API, with a short-lived token the controller gives it at launch. This is the same configuration your Helm values give the platform.
  3. Register. The host registers with your Tembo installation and becomes Ready, usually within a few minutes. The controller replaces a host that doesn’t register within 15 minutes.
  4. Run sessions. The host claims sessions and runs each one in its own virtual machine.
Hosts read their image and configuration only when they start. When either changes, the controller replaces the running hosts, one at a time by default (see Apply changes).

Scaling

The controller sizes each pool by memory: the memory its sessions use, plus the memory waiting sessions need, plus minIdleMemory. When that is more than the pool’s hosts provide, it launches another host, up to maxHosts. It never goes below minHosts. To scale to zero, set both minHosts: 0 and minIdleMemory: 0; otherwise the pool keeps the requested spare memory available even with no sessions. See Agent hosts configuration. When a pool has more hosts than its demand needs, the controller removes a host that has been idle for disruption.idleAfter (10 minutes by default): no session virtual machines, agent turns or host jobs. Warm VMs don’t count: removing the host stops them, and the remaining hosts keep their own. The controller removes the newest idle host, one per pool at a time, and never goes below minHosts. Removing a host drains it first. The host takes no new sessions, its virtual machines stop, and their snapshots upload to object storage; then the controller terminates the instance. If a session starts on the host while it drains, the controller keeps the host. To keep a host from being removed, annotate it:

Sessions on a host

  • Stopping. When a session goes idle, its virtual machine stops. The host saves a snapshot of its disk and memory, and uploads the disk to object storage.
  • Resuming. On the same host, a session resumes from its saved memory, with its processes still running. On another host, for example after its host was replaced, it resumes from the disk in object storage, without the processes that were running.
  • Project images. A project build creates an image of the project’s environment and stores it in object storage. Hosts download it on first use and keep a copy, so later sessions start in seconds.
  • Warm VMs. For projects with Warm pool on, hosts keep virtual machines booted from the project’s image, ready for the next session.
Saved memory only restores on a host with the same CPU, so keep all hosts on one processor generation (see Instance types).

Need Help?

If you run into any issues, contact support@tembo.io.