api.tesser.sh receives ids, ports, and states. It never receives a file or
a diff, and of program output only a dev server’s last few lines
(see below). It carries your app’s
traffic only for a share link on tesser.review, which
orgs in their own AWS account never get. For
an org that runs its boxes in its own AWS account, the code
exists in two places: your laptops and your account.
The only secrets in the system are each laptop’s SSH key, the values set
with tesser env set, and your API tokens. An org in its own AWS account
adds none: Tesser reaches the account with a short-lived token, not a key.
A snapshot is one copy that leaves the boxes: it
stores a builder box’s home directory, with the checkout in it, in an S3
bucket. For an org in Tesser’s cloud that bucket is Tesser’s; for an org in
its own AWS account it is a bucket in that account. In Tesser’s cloud, a box
that runs as a microVM leaves one more:
when it goes to sleep, its host stores a copy of its disk and memory in
Tesser’s S3 bucket (see below). With
fleet hosts in your own account, that
copy goes to your snapshots bucket instead.
Every hop
The control plane has no upload endpoint. A box talks to it over a
WebSocket whose upward messages are
hello, heartbeat, and a short list
of events, and over HTTPS to download boxd, fetch its snapshot plan, get
signed snapshot URLs, and report a finished snapshot’s sizes. None of them
carries a file.
The SSH connection
Each laptop has one ed25519 key pair, generated withssh-keygen the
first time you sign in and stored beside your credentials in
~/.config/tesser/. The private half never leaves that directory. The
public half goes to the control plane once, on your membership, and the
control plane decides which boxes it belongs on: your own boxes, every
box if you are an org admin, every shared instance, a box a CI job or
hosted agent launched on your behalf, and a teammate’s box while they
share it with you. A CI job or hosted agent that signs in through a grant
registers its own key, which lasts only as long as its token. boxd
receives that whole set over the box’s WebSocket and writes it as ~/.ssh/authorized_keys, replacing the file each time, so a
key that stops belonging there leaves it. sshd checks keys only at login,
so when a push removes a key, boxd waits two seconds and then ends every
open SSH connection that logged in with a key no longer in the file. It
finds each connection’s key in the box’s system journal, and leaves open a
connection whose login entry is no longer there. sshd on the box accepts nothing
else: no password, no cloud-init key, no key the control plane did not
place.
Every ssh and rsync the CLI runs uses IdentitiesOnly=yes with that
key, BatchMode=yes so an agent never hits a prompt, and a per-box
known_hosts file with StrictHostKeyChecking=accept-new: the first
connection records the box’s host key and every later one must match it.
Nothing touches your ~/.ssh. Through Session Manager the host key is
filed under the instance id instead of an IP.
What sync sends
sync, exec, and dev ship the files git would: tracked files plus
untracked files that are not ignored, as one explicit file list to rsync.
Ignored paths such as node_modules, .env.local, and build output stay
on the laptop unless the manifest’s env.files names them. Removals are
exact paths from the previous list rather than --delete, so the box’s own
build state survives.
The box’s .git is built from your clone. The CLI remembers which refs
the box holds, packs only the git objects the box lacks, and streams the
pack over SSH into git index-pack on the box. A fresh box gets your
branch’s commits since the default branch, plus the default branch’s tip;
afterwards the CLI sends up to 100 older commits in the background. The box
never fetches from GitHub and has no git
credential.
A shared instance works the same way. tesser make api --ensure-running <sha> sends that commit as a pack and checks it out on the box; only the
files the manifest’s env.files names ride rsync, from your worktree. The
control plane is told the sha, not the commit.
What the control plane receives
- Box metadata. Ids, class, service name, size, owner, power state, IP addresses, and the commit sha of a shared instance.
- Service manifests. The TOML text of each manifest, which holds ports
and the
setupanddevcommands and is committed to your repository. - Your public key, on the way to the box.
- boxd heartbeats and events. Whether the dev server is running, its
pid, exit code, which ports are listening, whether the health check
passes, and how many SSH sessions are active. Events add when
setupfinished, the status and time of the first page the server answered, the sizes ofnode_modulesand build caches, CPU, memory, and disk pressure, and the outcome of a boxd update. - Crash snapshots and the last log line. When a dev server exits, boxd
sends the last 40 lines of its output with the exit event so
tesser devcan say why. While a dev server runs, boxd also reports the last line of its output, cut to 300 characters, with how many times in a rowdevhas printed that same line, and when the app crashes under a file watcher or another process holds its port, the last 20 lines. These are the only places program output reaches the control plane. Until the dev server serves, boxd also reports the name of its busiest process: the program name, one script (a code file, abin/entry, or a-mmodule), and up to 2 lowercase subcommand words, such asnode medusa db:migrate. Flags, values, credentials, and connection strings in its arguments never leave the box. An org in its own AWS account turns all of these off withtesser cloud connect --no-crash-output: the control plane then keeps the exit code and the CPU use but none of the output lines, the repeat count, or the busiest process’s name, andtesser devpoints attesser logs, which reads the log file over SSH. - Snapshot records. Each snapshot’s id, repository, commit sha, sizes, state, and, for a failed build, a short reason. The setup log stays on the builder box, and the image goes to the bucket.
- Routing tables and wires. Which box each dependency port points at.
- Fleet host reports. Each host reports the state, disk use, ports, and errors of its microVMs, CPU, memory, and pressure for itself and each microVM, and hostd’s own events and log lines.
- Values set with
tesser env set, stored in the org and placed in the environment of the service’s boxes and of workbenches from the same repository. An org in its own AWS account can keep them in its Parameter Store instead: the control plane then writes each value once, keeps the key’s name, and never reads a value; the boxes read them themselves. Env files pushed to a dev box go over SSH and are not among them. - Usage. When each box was awake, with its size, service, and owner.
- API token hashes.
exec,
ssh, or logs, HTTP traffic to a dev server outside a tesser.review
share link, and the contents of any env file that was pushed or synced.
In a snapshot
A snapshot holds/home/ubuntu, /var/lib/docker, and
/var/lib/containerd from the builder box after every setup ran: the
checkout at the snapshot’s commit, node_modules and other installed
dependencies, and whatever setup wrote. If setup writes a value from
tesser env set to disk, the snapshot holds it too. Before the capture, the builder deletes
the personal files listed under Building one, such as
~/.ssh/authorized_keys, ~/.git-credentials, and ~/.aws. Tesser’s
bucket blocks public access, encrypts at rest with SSE-S3, and refuses
requests not made over TLS. In Tesser’s cloud, boxes have no S3
credentials: the control plane signs a URL that expires within an hour for
each object a box reads or writes. In your own AWS account
it is the reverse: boxes reach your bucket with the boxes’ role, and the
role Tesser assumes can only list the bucket and delete objects nothing
references, never read or write them.
On the box
The worktree lives at~/workspace and the dev server’s output at
~/.tesser/dev.log. boxd acts only when told to: start or stop the dev
server, rewrite its routing table, replace its authorized keys, set the
env for shells, mount a repo’s snapshot when a pool box is claimed, dial a
share gateway while the box is shared, delete old snapshots from the
bucket, and update itself. It has no message for reading
a file.
A start cannot name the command. A start message names only the service:
boxd reads setup and dev from the <service>.toml in ~/workspace,
the copy your own tesser sync or tesser dev put there, and runs those.
That guarantee covers starts and wakes, and nothing more:
- Env is the control plane’s to set (it holds what you gave
tesser env set). boxd drops keys that load a program, such asNODE_OPTIONS,LD_PRELOAD,BASH_ENV, andPROMPT_COMMAND, along with theGIT_,NPM_CONFIG_, and similar families. That list is a denylist, so it covers the known hooks rather than every program’s. Files the manifest’senv.loadnames are read from~/workspace, likesetupanddev, and are not filtered. - A boxd update carries a program. The control plane tells boxd to update, and boxd fetches a new bundle from the control plane and runs it in place of itself, but only a bundle that carries Tesser’s release signature; the first boot of a box checks the same signature. The control plane chooses which signed release runs, not what is in it.
authorized_keysis the control plane’s to write, which is how it decides who may ssh in.
tesser rm deletes the box’s copy. The host gives the
microVM the same user-data JSON, and the box has no route to its host or to
AWS’s metadata endpoint. In Tesser’s account the box has no AWS credentials.
On a host in your own account, the box gets credentials for the boxes’ role
you set with --fleet-box-role from its own metadata endpoint, in an STS
session of its own.
boxd proves which box it is with a token from AWS STS
GetWebIdentityToken: signed by AWS, valid for at most five minutes,
naming the instance and its account, and addressed to the control plane.
The control plane accepts a box’s WebSocket only with a token naming that
box’s instance. In Tesser’s account, minting that token is the only thing
the box’s instance role can do. Any process on the box can mint one, so
code running on a box can speak to the control plane as that box: it
receives that box’s routing table, env, and authorized keys, which are on
the box already, and nothing of any other box.
A microVM has no AWS identity. Its host vouches for it instead: boxd asks
the host over the microVM’s own vsock connection, and the host, which knows
the box from the socket the request arrived on, answers with a token naming
the box, signed with the host’s Ed25519 key and valid for 60 seconds. The
control plane accepts that token only when it is signed by the host the box
runs on.
In Tesser’s account, the boxes of one org sit in a security group that
admits port 22 from anywhere and, from the org’s own boxes, only UDP 51820
and TCP 14650. Boxes carry dependency traffic to each other over WireGuard
on UDP 51820, each with its own key and an address in 100.64.0.0/10.
TCP 14650 is the port boxd forwards dependencies over when either box runs
an older boxd. Once the org runs boxes
on fleet hosts, the group admits UDP 51820 and TCP 14650-14655 from its own
boxes and the fleet hosts; 14652-14655 carry a box moving between EC2 and a fleet
host. A box accepts a connection over WireGuard or on 14650 only to a port the
box’s manifest declares, so another box never reaches anything else your
app or a test listens on, and nothing reaches the group from another org.
In your own account there is no port 22: the group admits only UDP 51820 and
TCP 14650-14655, from the org’s own boxes. Box-to-box traffic stays on private VPC addresses.
A microVM keeps its private address when it moves to another host. SSH
reaches the box’s port 22 through a port on its host that is open to
anywhere. Port 14650 reaches
it only from the org’s own boxes: from microVMs on other hosts over WireGuard
between the hosts, and from the org’s EC2 boxes through a second port on the
host. The host drops everything else: traffic to or from another org’s boxes,
to the host itself, to AWS’s metadata endpoint or any other private address
except the VPC’s DNS resolver,
and outgoing mail on port 25. The box reaches the internet from its host’s
public address.