A self-hosted box
- has no inbound SSH port. Laptops reach sshd through AWS Session Manager, so the security group admits nothing from outside the org’s own boxes, which reach each other on UDP 51820 and TCP 14650-14655.
- never runs its user-data. The user-data is three fields of JSON: the org id, box id, and control plane URL. None of it is secret. The image turns cloud-init’s user-data processing off, so nothing in it is executed. On first boot a step baked into the image reads it from the instance metadata service, refuses to go on unless every field is well formed (an https control plane URL), fetches boxd, and installs it only if its release signature verifies.
- holds no Tesser credential. boxd proves which instance it runs on
with a short-lived token that AWS STS signs for the box’s role
(
sts:GetWebIdentityToken, outbound identity federation). The control plane accepts it only when the token names this box’s instance in your account, addressed to the control plane. - launches only the image your role allows. By default that is Tesser’s published image: your role may launch images owned by Tesser’s publishing account and no one else’s. To own the image outright, copy a release into your account and pin the role to the copy (Own the image).
The Pulumi program
You need to be an admin of the org, and the org needs to have no boxes: a box cannot move between clouds, sotesser rm any it has first.
The setup is a Pulumi
program you copy into your own infrastructure repository; it needs a
@pulumi/aws 7 release with aws.iam.OutboundWebIdentityFederation. It is two files:
tesser-policy.ts holds the IAM documents, and index.ts makes the
network, the roles, and the instance profile.
Two values come from Tesser:
tesser-policy.ts
index.ts
pulumi up finishes, run the command it exports as connect, with
the ~/.aws profile members’ laptops use for
Session Manager:
--account finds the role (tesser-<org id>), the boxes’ instance profile
(tesser-<org id>-box), and the subnet and security group tagged with the
org id. The account id can be pasted with dashes, the way the AWS console
shows it. From then on every
tesser make, dev, and exec launches in your account. Boxes sleep,
wake, and are removed exactly as before.
The program makes, in the region you run it in:
- A VPC (
10.73.0.0/16) with one public subnet, an internet gateway, and a route to it. The subnet sits in a zone that offers every box size:use1-az1in us-east-1,usw2-az1in us-west-2. Boxes need that egress to reach Session Manager,api.tesser.sh, andreleases.tesser.sh. - A security group: UDP 51820 and TCP 14650-14655 from the group itself, so boxes reach each other over WireGuard on 51820 (and on 14650 from boxes on an older boxd) and can move between EC2 and fleet hosts on 14652-14655, all outbound, and nothing else inbound. No port 22.
- The identity provider for
https://api.tesser.sh. An account holds one per URL; when a second org sets up in the same account, drop theOpenIdConnectProviderfrom its program and look up the existing one withaws.iam.OpenIdConnectProvider.get. - A role named
tesser-<org id>that only this org can assume, with the permissions policy fromtesser-policy.ts. - Outbound identity federation, switched on for the account, so the boxes’ role can ask STS for identity tokens. It is one account-wide setting; a second org in the same account drops it from its program.
- The boxes’ role and instance profile,
tesser-<org id>-box, withAmazonSSMManagedInstanceCore,sts:GetWebIdentityTokenfor tokens addressed tohttps://api.tesser.shonly and, withtesser:parameterStore, read access to the org’s parameters.
pulumi destroy removes everything it made;
disconnect the org first.
To use an existing VPC, a private subnet behind a NAT (the CLI then reaches
boxes at their private IP with --private), or your own role names, keep
tesser-policy.ts and change index.ts, then connect with the ids
(--role, --subnet, --sg, --instance-profile) instead of
--account. tesser cloud template --account <id> --region <region>
prints the same policies with your values filled in. PassBoxRole names
the IAM role inside the boxes’ instance profile, not the profile. The
program gives the role and the profile the same name, and template
assumes that: it uses the org’s instance profile name (or
tesser-<org id>-box before the org is connected). If your role’s name
differs from the profile’s, or has a path, pass it as boxRole (in the
program) or --box-role path/name (to template).
Own the image
By default boxes launch Tesser’s published image, and the role may launch any image Tesser’s publishing account owns. To take that choice away from Tesser, copy an image into your account and pin the role to the copy. Tesser then launches exactly that image and no other for EC2 boxes, and a new image reaches them only when you copy it and move the pin. The pin does not cover fleet hosts or the microVMs on them. Every image build publishes a public box image inus-west-2 and
us-east-1, named tesser-box-<build time>-<commit>: the build’s UTC time
as YYYYMMDDhhmmss, then the first seven characters of the Tesser commit it
was built from. Pin an image by its name. This prints the newest image in
your region:
image line of index.ts with:
sourceName to a newer image’s name to take a new image.
The permissions policy’s LaunchImage statement becomes
arn:aws:ec2:<region>::image/<your copy> with no owner condition;
tesser:imageOwner is then used only to find the source image. Copy an
image soon after you pick it: Tesser deregisters all but its few newest and
most recently deployed images, and a deregistered image can no longer be
copied. Then tell the control plane which image to
launch:
--image tesser goes back to the published image; move the policy back
first. The image can change while boxes run: running and sleeping boxes
keep the image they launched from, and new boxes take the new one. Boxes
hibernate when they sleep only on images from release build-40566052647
or later; boxes on an older image are powered down instead, and their dev
server is started again when they wake.
What connect checks
connect tests the account end to end and stores nothing if any check
fails. Each failure names what is wrong and how to fix it:
tesser cloud check runs them again at
any time without changing anything. tesser cloud shows the config, and AWS
errors from the last seven days: a call AWS keeps refusing, such as a
StopInstances a tightened policy no longer allows, stays listed until it
works again. A dry run cannot test stopping or terminating an instance that
does not exist yet, so those only surface there.
What the control plane can do
Your code, the disks, the dev server logs, and the traffic between boxes stay in your account. Security and data flow lists every hop and everything the control plane receives. What the control plane can still do to your boxes, through the role and boxd’s WebSocket:- Start, stop, and terminate the org’s instances, and launch new ones from the image the role allows, into your subnet and group.
- Start and stop the dev server declared in your manifest. A start names only the service; boxd reads the command from the manifest on the box’s own disk.
- Set its env. boxd drops keys that load a program (
NODE_OPTIONS,LD_PRELOAD,BASH_ENV,PROMPT_COMMAND, theGIT_andNPM_CONFIG_families, and more). The list is a denylist, so it covers the known hooks, not every program’s. With Parameter Store the control plane can write values but not read them. - Update boxd. An update makes boxd fetch a new bundle from the control plane, and boxd installs it only if it carries Tesser’s release signature; first boot checks the same signature. The control plane can pick which signed release runs, not write its own.
- Replace
authorized_keys. sshd has no network path except Session Manager, which your IAM gates, so a key alone reaches nothing. - Run fleet hosts. With a fleet, it can also launch and terminate hosts, start, stop, and move the microVMs on them, make and remove the volumes a move uses, and choose which hostd build and guest image each host runs.
How Tesser reaches your account
Tesser never holds a credential for your account. The control plane is an OpenID Connect issuer, and the role trusts it for exactly one org id. Each time the org cell needs EC2 it signs a token that lives five minutes, trades it for one hour of temporary credentials withAssumeRoleWithWebIdentity,
and uses those. Another org’s cell cannot assume your role: the trust policy
names the org in sub, and the token carries the org that signed it. The
permissions are fenced the same way: the role can only launch instances
tagged with the org id, and only stop, start, or terminate instances that
carry it.
Session Manager
Every ssh, sync, and daemon tunnel runstesser-ssm proxy as its
ProxyCommand: tesser’s own Session Manager client, which ships inside the
tesser binary, so no AWS CLI and no session-manager-plugin on the laptop.
The proxy hands each connection to a tesser-ssm daemon on the laptop,
which keeps one Session Manager session per EC2 box and carries every
connection to it as a stream inside that session. A fleet host runs the SSM
agent’s own ssm-session-worker, so each connection to a microVM on it gets
its own Session Manager session. The host key is filed under the instance
id.
On EC2 boxes, Tesser swaps the SSM agent’s ssm-session-worker for its own
build, downloaded from releases.tesser.sh. It serves port-forwarding
sessions without KMS encryption and hands every other session (shell, Run
Command, KMS) to the agent’s own worker, so those work as before.
What it needs:
- The instance profile carries
AmazonSSMManagedInstanceCore(the SSM agent ships in the Tesser image), and the role tesser assumes hasiam:PassRoleon its role. The program above does both. - Every org member needs an AWS profile in
~/.aws/config(or~/.aws/credentials) for the account, withssm:StartSessionandssm:TerminateSessionon the boxes. Any profile kind works (SSO, static keys,source_profile+role_arn,credential_process); tesser reads the same files and SSO token cache the AWS CLI does, soaws sso login --profile <name>is still how an SSO profile signs in.--aws-profile <name>oncloud connectis the org’s default profile name, the one every member’s laptop is expected to have; a laptop whose profile is named differently overrides it withtesser cloud aws-profile <name>, kept in that laptop’s tesser config. With--sso-start-url <portal> --sso-region <region> --sso-role <permission set>oncloud connect, members don’t have to set anything up: a laptop missing the profile, with nothing else for the account, gets it appended to~/.aws/config(account from the role ARN) the first time it makes or wakes a box, and the SSO sign-in below takes it from there. A profile the laptop already has is never touched. - Agents and CI don’t need the profile. When it is missing or will not
resolve, tesser uses the default AWS credential chain (env keys, an
instance or container role, the default profile) once STS says those
credentials are for the org’s account; credentials for any other account
are never used. When neither works (SSO session expired, keys rejected),
every ssh-driven verb fails immediately and the
daemon holds its tunnels until it works again. For an SSO profile the verb
also opens
/__tesser/aws?profile=<name>on the daemon’s port: the daemon starts the device-code sign-inaws sso login --use-device-codedoes, shows the code, links to the AWS approval page, and writes the token to~/.aws/sso/cachewhen it is approved, so no AWS CLI is needed. The tunnels resume on their own. The verb’s error tells an agent to runtesser aws-wait <name>once, in the background, and to rerun the command once it says signed in.tesser aws-waitreturns once the code is approved and fails with the reason if it is denied or expires. If it fails, the error tells the agent to ask the person at the laptop instead of starting another wait. If nobody answers, the daemon opens the sign-in page at most three times for a profile and then stops opening it. From then on the verb’s error says to finish signing in at that page and to rerun the command only once the page says signed in. The count starts over when someone uses the page (its approval link or Try again) or runs a verb in an interactive terminal.
ManageOwnSessions is the statement to get right. Scoping it by session ARN
the way AWS leads with
only matches an IAM user: a federated caller’s ${aws:userid} is
AROAEXAMPLEID:you@example.com while the session id is
you@example.com-<suffix>, so every terminate is denied. The denial is
silent: tesser retries the close in the background without reporting it, and
the sessions stay Active. The session-tag form
above works either way, with the same scoping: your own sessions only.
Laptop-to-box transfers (sync, git packs) run at roughly link speed through
Session Manager: tesser’s client sends 32 KiB frames with a 4 MiB window,
where the official plugin paces itself to ~0.7 MB/s.
Env in your Parameter Store
By defaulttesser env set stores a service’s values in the org, and the
control plane injects them when an instance starts. With
--env parameter-store they live in your account instead, one SecureString
per key under /tesser/<org id>/<service>/<KEY> in the org’s region,
encrypted with the account’s aws/ssm key:
tesser env set/unsetgo through the control plane, which writes the value into your Parameter Store and keeps only the key’s name, sotesser env lsand theenv.requiredcheck need no AWS call. The role it assumes needsssm:PutParameterandssm:DeleteParametersonparameter/tesser/<org id>/*and nothing else: nossm:GetParameter*, nossm:DescribeParameters. That one statement is what you audit. A parameter changed or removed outsidetesser envis not noticed byenv lsor theenv.requiredcheck; boxes see it the next time they read.- Each box reads the values itself with the instance profile’s
credentials: when its dev server starts (a service box reads its
service’s, a workbench every service of its repository, as with
cell-held env) and, for shells, on connect and after every
env set. That profile’s role needsssm:GetParametersByPathon the same prefix; the Pulumi program adds it withtesser:parameterStore. Any box in the org can read any of the org’s services’ env, which is the same trust boundary as the mesh.
connect proves the role can write and delete under the prefix before
storing anything, then moves the values the org already holds into
your Parameter Store and forgets them. A value is at most 4 KB (the standard
tier), and may not be blank; unset a key to remove it. If a box cannot read
its env at start, the run fails with the AWS error in tesser dev’s output
rather than starting without it. Moving --env back to tesser, or
disconnect, leaves the parameters in your account and the org with no
values: set them again. If you later move back to parameter-store, boxes
read whatever is still under the prefix, including values the org no longer
lists; delete the stale ones or set them again. tesser service rm deletes the service’s
parameters. Because these are your parameters, the control plane can also
set them, the same way it can start the dev server; what it cannot do is
read one.
Snapshots
Snapshots of a connected org live in an S3 bucket in your account. Until you name one, the org has no snapshots andtesser snapshot create fails. Make a bucket in the org’s region that holds
nothing but Tesser’s data, because Tesser deletes whatever in it nothing
references. Block public access, use default encryption SSE-S3 (or SSE-KMS),
and turn versioning on. Give it the lifecycle that tesser cloud template
prints, which has two rules:
- Abort incomplete multipart uploads after 1 day. An upload that stops partway is never listed, so S3 keeps billing for its parts until the rule removes them.
- Expire noncurrent versions after 2 days, and remove delete markers that have no versions left. A deleted or overwritten object stays in the bucket for those 2 days, so you can restore it if Tesser deletes data it still needed.
tesser cloud template prints the three policies the bucket needs:
- The boxes’ role gets
s3:GetObject,s3:PutObject, ands3:DeleteObjecton<bucket>/<org id>/*, ands3:ListBucketon the bucket for keys under<org id>/. For an SSE-KMS bucket, pass--snapshots-kms-key <arn>totemplateand it addskms:GenerateDataKeyandkms:Decrypton that key, through S3 only. - Tesser’s role gets
s3:ListBucketon the bucket,s3:DeleteObjecton<bucket>/*, ands3:GetBucketVersioningands3:GetLifecycleConfigurationon the bucket, and nothing else. - The bucket gets a policy that denies any request not made over TLS.
--snapshots-bucket none stops using snapshots.
Your own tags
The org can store its own tags, for example a cost center, and Tesser adds them to every instance it launches in your account, next to its own tags:- Tesser sets them at launch, which the
TagAtLaunchstatement already allows, so the policy needs nothing new. Instances that are already running, including pool boxes waiting to be claimed, keep the tags they launched with. - Only instances carry them. Their volumes and network interfaces, and the org’s Parameter Store parameters, are not tagged.
- The tags you pass replace the stored set.
--tag noneremoves them all, andtesser cloudlists them. - An org stores at most 30 tags, because Tesser’s own tags use part of AWS’s
limit of 50 per resource. Keys are 1 to 128 characters and values at most
256, using letters, digits, spaces, and
_ . : / = + - @. - Keys that start with
tesser:oraws:, in any case, and the keyNameare refused. Tesser’s own tags always win over the org’s. - When the tags change,
connectruns the launch check again with the new tags and stores nothing if it fails. A service control policy or tag policy that refuses them stops the change instead of every later launch.
Fleet hosts
By default each box is its own EC2 instance. With a fleet, Tesser can also run r8a.metal-24xl hosts (96 vCPUs each) in your account through the org’s role, with the org’s boxes on them as microVMs. The fleet keeps sleeping boxes in the snapshots bucket, so it needs--snapshots-bucket. Hosts write them under <org id>/ with the
boxes’ role. Make a role and an instance profile
for the hosts, trusted by ec2.amazonaws.com, in the connected account, and
connect with the role’s ARN and the name of the boxes’ role, the IAM role
behind the boxes’ instance profile:
--fleet-box-role is required with --fleet and has no default. Include the
role’s path when it has one. Pass --fleet-profile <arn> too when the
instance profile’s name differs from the role’s. tesser cloud template
prints the policies:
-
The host role gets
AmazonSSMManagedInstanceCoreand a policy that lets it mint its identity token for the control plane (sts:GetWebIdentityToken), uses3:GetObject,s3:PutObject,s3:DeleteObject, ands3:AbortMultipartUploadon<bucket>/vms/*, ands3:ListBucketfor keys undervms/, where hosts kept sleeping boxes before. For an SSE-KMS bucket it addskms:GenerateDataKeyandkms:Decrypton the key, through S3 only. It also getssts:AssumeRoleandsts:TagSessionon the boxes’ role. -
The boxes’ role needs a trust policy that allows
ec2.amazonaws.comsts:AssumeRole, and the host rolests:AssumeRoleandsts:TagSession. In the Pulumi program that role istesser-<org id>-box. Replace its definition inindex.tswith: -
Tesser’s role gets
iam:PassRoleon that host role alone, passed toec2.amazonaws.com. With your own image pinned, it also getsec2:RunInstanceson images from Tesser’s publishing account, for the hosts. For moves it getsec2:CreateVolumewith thetesser:orgrequest tag, tagging at create, and attach, detach, and delete on volumes taggedtesser:organdtesser:move. Hosts carry thetesser:orgtag andtesser:managed=true, so the existing launch and terminate statements cover them. A policy that lists instance types needsr8a.metal-24xladded. -
Members reach a microVM through Session Manager on its host, which
forwards one port on its own loopback address to that box’s port 22 only.
Hosts carry
tesser:managed=true, so the permission set covers them once it also allows theAWS-StartPortForwardingSessiondocument. Do not grantAWS-StartPortForwardingSessionToRemoteHost: it would let a member reach any address from the host, including the instance metadata service and the host role’s credentials.
- the org’s security group to allow TCP 14650-14655 (see the Pulumi program), UDP 51820, and TCP 22000-23999 from itself, so boxes move and hosts reach each other;
- subnets below
10.127.0.0, since hosts address their boxes above it; - an on-demand vCPU quota with room for at least one host (96 vCPUs) beside everything else the account runs.
- A new box is made as a microVM on a host when one has room. It is made on EC2 instead when no host has room.
- Each minute Tesser compares the awake vCPUs of all the org’s boxes, on EC2 and on hosts, their lowest point over the last 5 minutes, and the peak of the coming 15 minutes at the same time of week over the last three weeks against what a host costs, and starts or stops hosts to match. A host starts once either figure passes 80% of the vCPUs at which a host costs the same as those boxes on EC2, so a host can start a little before it pays for itself. With no awake boxes and no forecast, no host runs. Boxes are packed onto the fullest host with room.
- Tesser moves awake EC2 boxes onto hosts with room, and boxes off a draining host to another host, or to EC2 when no host has room. A box does not have to be idle to be picked for these moves. An EC2 box moves onto a host only once it has been awake on EC2 for a few minutes, and an older EC2 box only while the forecast alone calls for a host (see Timers).
- Tesser also moves asleep EC2 boxes onto a host, so their next wake is a wake on the host. It picks boxes that have been asleep for a while, newest sleep first, at most two of the org’s boxes at a time, and only while the org needs hosts and runs no more than it needs. Pool boxes and boxes close to their removal time are skipped. Tesser starts the EC2 instance for the move, but the box stays asleep to members, opens no usage, and is billed as asleep. AWS still charges for the few minutes the instance runs. After the move the box sleeps on the host. A move that fails puts the box back to sleep where it is, and Tesser tries again later, with a longer wait each time (see Timers).
- A member who wakes a box while it moves this way wakes it as usual. Tesser calls the move off when it can, and otherwise wakes the box where the move ends.
- When an awake box reaches its idle sleep time while it moves between EC2 and a host, it stays awake until the move ends and then sleeps where it landed. A move that has run longer than a move can take is called off instead, and the box sleeps where it was (see Timers).
- A box whose boxd does not speak the current release’s move protocol gets a boxd update before it moves off a draining host or onto a host with room, as long as the current release can move it. A box on an older boxd that already speaks the current move protocol moves without an update. The update waits until the box is idle (see Timers). The dev server keeps running across the update, and the box moves on a later pass once it runs the new version. An update that does not land is tried again later, with a longer wait each time (see Timers).
- A box moves between EC2 and a host only when it runs the current boxd release. Tesser refuses any such move for a box on an older boxd, including one an admin starts, with the reason that boxd must be updated first. A box on an older boxd can still move from one host to another.
- While any microVM of a size is awake, Tesser keeps one EC2 box of that size awake in your account, so a box leaving a host has an instance to land on.
- A host that runs out of memory or disk moves boxes to another host, or stops keeping a long-asleep box locally, which then wakes from the bucket.
- The emptiest host drains once the org has needed fewer hosts for a while, if the other hosts can take its boxes. A host on an old host image drains before any host on the current one. A host in its first hour drains only when no box, awake or asleep, is on it. Otherwise no host drains until that host is older (see Timers).
- A host that stops answering still counts as running for a while after Tesser last heard from it (see Timers). Until then Tesser starts no host and no spare EC2 boxes to replace it, and drains or moves nothing on it.
- Tesser moves asleep boxes off a draining host onto EC2, at most two each minute, and each needs a spare EC2 box of its size to land on. The box stays asleep, opens no usage, and is billed as asleep. Its next wake is a wake on EC2, and its dev server starts again because only its disks moved. A box that is waking or moving waits for a later pass, and a box whose move to EC2 failed is not tried again for a while (see Timers). A box that the current release cannot move stays on the host and is kept in the bucket.
- A draining host is terminated once no box on it is awake, every asleep box that can move has moved to EC2, and every box left on it is stored in the bucket. When every host drains, Tesser also moves boxes that are kept only in the bucket onto EC2, and keeps the last host running until they have moved, since only a host can read the bucket.
- Each minute Tesser also looks for instances in your account that carry the
tesser:fleet-hosttag and the org’stesser:orgtag and run with the fleet’s host instance profile, but that the fleet did not launch or no longer tracks. Such an instance is terminated once it has stayed that way for a while with no box awake or unstored on it (see Timers). - Each microVM of the org gets AWS credentials for the boxes’ role from its
own metadata endpoint, listed under the role’s name without its path, so it
has the same AWS access as an EC2 box. The host assumes the role once per
box, with the session name
box-<box id>and the session tagtesser:box=<box id>, so CloudTrail shows which box made each call. The host refreshes the credentials before they expire and drops them when the box sleeps, stops, or is removed. Only boxes of the host’s own org get them. - A box waits up to 10 seconds for its first credentials before it starts or
resumes. When the host cannot assume the role, the metadata endpoint
answers
AssumeRoleUnauthorizedAccesswith the reason. With no box role set, it answers that the org has no fleet box role set. - Sleep and wake on a microVM covers how a wake finds room on a host or falls back to EC2.
- Hosts launch Tesser’s host image, and microVMs run the guest image and
hostd build that Tesser releases, whatever image the org pins with
--image. A host downloads both from Tesser’s S3 bucket with short-lived credentials from the control plane, and checks each file against the sha256 the control plane sends. No Tesser release signature is checked on that path. - When Tesser releases a new hostd or guest image, hosts update in place one at a time, and the next host waits until the last one is back on the new release. A host on the old release that has no work in flight updates first. Work in flight means a box moving onto or off the host, a box waking, sleeping or starting on it, a disk filling from the bucket, or a sleeping box still being stored. When every such host is busy, one of them waits to update until it is quiet, up to a limit (see Timers), and then updates anyway. While a host waits and until it is back on the new release, Tesser starts no scheduled moves onto or off it, and puts new boxes and wakes on other hosts first. When Tesser releases a new host image, hosts on the old one drain one at a time and are replaced. Tesser moves no box onto a host on the old image, including while the host that replaces it is still starting.
- Hosts are billed to your account at EC2’s rate for r8a.metal-24xl. The dashboard’s Usage tab estimates the org’s AWS spend from how long each host ran, instead of from each box’s hours.
- The dashboard’s Hosts tab appears only for orgs that run fleet hosts, and every member can open it. It lists each host with its state, how much of its memory is packed with microVMs, its hourly and today’s cost, and its CPU and memory for the last hour, minute by minute. Under each host are its boxes and their load. It updates live, like the Boxes tab.
Changing or leaving
tesser cloud connect --max-boxes <n>, --env <store>, --image <ami>,
--aws-profile <name>, and --no-crash-output (boxd keeps a crashed dev
server’s last lines on the box instead of sending them with its exit) work
at any time; the other flags default to the
stored config. Where boxes run cannot change while any exist. Remove them,
then connect again with the new ids, or tesser cloud disconnect to
return the org to Tesser’s cloud.
--fleet can be turned on while the org has boxes. Changing the host role or
profile, or --no-fleet, terminates the hosts first and is refused while the
org has any microVM; EC2 boxes do not block it. Disconnecting terminates the
hosts too.
Disconnecting deletes nothing in your account. pulumi destroy removes the
network, the roles, and any image copy when you no longer need them.