> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tembo.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent hosts

> How agent hosts run sessions, and how the host controller manages them.

Agent sessions run outside your cluster, on **agent hosts**: cloud instances that run each session in its own virtual machine. The chart's **host controller** runs in your cluster and launches, tracks and replaces them.

Agent hosts are EC2 instances on AWS, or Compute Engine instances on GCP, started from Tembo's agent host image. The host controller gets it with your license key, and Tembo shares it with your AWS account or the controller's Google service account. They need nested virtualization, and they keep VM disks, snapshots and saved memory on the instance's local NVMe.

## Classes and pools

You describe agent hosts in your Helm values, with two kinds of resources the chart creates. The Terraform module's Helm values create one of each, named `default`.

* A **host class** says what to launch: the image, instance types, subnets, security groups, instance profile and root volume.
* A **pool** says how many to run: `minHosts`, `maxHosts`, and `minIdleMemory`, the free memory to keep for new sessions.

The controller creates a **host** resource for each instance it launches. List them with:

```bash theme={null}
kubectl -n tembo get tembohostclasses,tembohostpools,tembohosts
```

## Lifecycle

1. **Launch.** The controller starts an instance with the class's settings and tags it as its own.
2. **Configure.** The host reads its configuration from a Kubernetes Secret through the cluster's API, with a short-lived token the controller gives it at launch. This is the same configuration your Helm values give the platform.
3. **Register.** The host registers with your Tembo installation and becomes `Ready`, usually within a few minutes. The controller replaces a host that doesn't register within 15 minutes.
4. **Run sessions.** The host claims sessions and runs each one in its own virtual machine.

Hosts read their image and configuration only when they start. When either changes, the controller replaces the running hosts, one at a time by default (see [Apply changes](https://docs.tembo.io/features/self-hosted/kubernetes/configuration.md#apply-changes)).

## Scaling

The controller sizes each pool by memory: the memory its sessions use, plus the memory waiting sessions need, plus `minIdleMemory`. When that is more than the pool's hosts provide, it launches another host, up to `maxHosts`. It never goes below `minHosts`.

To scale to zero, set both `minHosts: 0` and `minIdleMemory: 0`; otherwise the pool keeps the requested spare memory available even with no sessions. See [Agent hosts configuration](https://docs.tembo.io/features/self-hosted/kubernetes/configuration.md#agent-hosts).

When a pool has more hosts than its demand needs, the controller removes a host that has been idle for `disruption.idleAfter` (10 minutes by default): no session virtual machines, agent turns or host jobs. Warm VMs don't count: removing the host stops them, and the remaining hosts keep their own. The controller removes the newest idle host, one per pool at a time, and never goes below `minHosts`.

Removing a host drains it first. The host takes no new sessions, its virtual machines stop, and their snapshots upload to object storage; then the controller terminates the instance. If a session starts on the host while it drains, the controller keeps the host. To keep a host from being removed, annotate it:

```bash theme={null}
kubectl -n tembo annotate tembohost <host> hosts.tembo.io/do-not-disrupt=true
```

## Sessions on a host

* **Stopping.** When a session goes idle, its virtual machine stops. The host saves a snapshot of its disk and memory, and uploads the disk to object storage.
* **Resuming.** On the same host, a session resumes from its saved memory, with its processes still running. On another host, for example after its host was replaced, it resumes from the disk in object storage, without the processes that were running.
* **Project images.** A project build creates an image of the project's environment and stores it in object storage. Hosts download it on first use and keep a copy, so later sessions start in seconds.
* **Warm VMs.** For projects with **Warm pool** on, hosts keep virtual machines booted from the project's image, ready for the next session.

Saved memory only restores on a host with the same CPU, so keep all hosts on one processor generation (see [Instance types](https://docs.tembo.io/features/self-hosted/kubernetes/configuration.md#instance-types)).

***

## Need Help?

If you run into any issues, contact [support@tembo.io](mailto:support@tembo.io).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.