Deploying the daemon

One daemon process runs executions for whatever connects to it. A deployment sets where it listens, how much it runs at once, the bounds a caller meets, and how it stops. The threat model says which of the boundaries around a deployment this code enforces and which the operator adds. Manifests under deploy/ are one deployment with every one of them already set.

Every setting takes a command-line flag and an environment variable both, and a flag wins. codemode-executor --help prints every setting, with its default where it has one.

Commands and the image

codemode-executor                   # serve, reading the environment
codemode-executor check             # answer whether this machine can run sandboxed code
codemode-executor status            # ask a running daemon what it is doing
codemode-executor maintenance on    # turn away new callers, keep serving the open ones

check runs one real job through the whole path, isolate included, and names what is missing when the path is broken. check --json prints the same verdict as one object carrying ok, for a script to read. Either form exits on the four codes codemode-executor --help prints at the end of its output, and Ask the machine first lists them.

Exit 1 covers two failures at once, and a script polling status cannot tell a daemon that has stopped from a setting the command refused. The message separates them: an unreachable daemon names the address the command tried. Only check exits 5, so 5 is the code a deployment alerts on. By default the image runs the daemon, so a machine that cannot build an isolate starts cleanly and breaks when the first caller arrives. A deployment that runs check first turns that into a startup failure with a reason in it: as an init container, or as the command of a one-off Job, before the daemon’s own container takes traffic.

codemode-executor is the image’s entry point, so a bare docker run … starts the daemon and docker run … check checks. That same binary sits on the container’s path, so docker exec … codemode-executor status talks to a daemon that is already running. maintenance off reverses maintenance on, and status says whether maintenance is on. Deployments outside a container run the built entry, node apps/daemon/dist/bin.js, with the same flags. The daemon is not published to npm.

isolated-vm compiles against the Node that runs the install, so the daemon requires whichever Node line the workspace declares in its engines field. Most deployments install no Node at all and hand the image to docker run instead:

docker run --rm -p 8080:8080 ghcr.io/maksimyaromin/codemode-executor:latest

That image is ghcr.io/maksimyaromin/codemode-executor, tagged with each release version and with latest on the newest, and built for linux/amd64 and linux/arm64. Every stage takes the same base, the Debian slim image of that Node line and never Alpine. isolated-vm compiles from source in the build stage, and the binary has to meet at runtime the C library it was compiled against. Inside, the daemon runs as the unprivileged node user. One daemon runs one pool of child processes, one child runs one execution at a time by default, and each execution gets a V8 isolate that is disposed when the run finishes.

Two listeners and their addresses

Listener	Address	What runs on it
Protocol	HOST and PORT, or SOCKET_PATH in place of both. Default host: every interface	The executor protocol over a WebSocket, plus GET and HEAD on /healthz, /readyz and /metrics
Admin	ADMIN_HOST and ADMIN_PORT. Default: the loopback address. ADMIN=off opens no admin listener, and then status and maintenance have nothing to talk to	The process status, the settings in force and the maintenance switch

By itself the daemon runs whatever code arrives and authenticates nobody. So an address reachable past the callers a deployment intends is arbitrary code execution offered to whoever finds it. Kubernetes allows every connection to a Pod until a NetworkPolicy selects that Pod and closes it, so in a fresh cluster every workload can reach the daemon. That network is the operator’s to close. deploy/k8s/same-pod.yaml repeats that in its own header, because there the daemon has a port open on the Pod address with no Service in front of it. Probes and the scrape annotation in the manifests both point at the caller-facing port, so a Prometheus scrape reaches the daemon on the very port the callers use.

The admin listener authenticates nobody either, so it belongs where an operator can reach it and a caller cannot. status and maintenance are its clients, and any other HTTP client drives it just as well:

Route	What it returns
GET /__status__	service, version, runtime, pid, uptime, maintenance, the protocol address with its accepting flag, the admin address, and the pool’s counters
GET /__config__	Every setting in force, each with the value and where it came from
GET /__maintenance__	{ "maintenance": boolean }
PUT /__maintenance__	Turns maintenance on and returns the new state
DELETE /__maintenance__	Turns it off and returns the new state

Each route name is wrapped in dunders, so no path an ordinary reverse proxy forwards can collide with one. Anything else on this listener returns 404.

Sizing is a memory decision

Memory runs out before CPU does. Each execution is an isolate with a heap bound, so the memory a machine has for isolates decides how many run at once. MAX_WORKERS bounds how many child processes run at once. Default: one per CPU the machine reports. That default counts CPUs, so a deployment takes its number from the arithmetic below instead. EXECUTIONS_PER_WORKER bounds how many executions one child runs at a time. Default: 1, and at 1 a native crash costs exactly one caller. MEMORY_LIMIT_MB bounds one isolate’s heap.

A container limit covers every worker at once, each running its own Node runtime and an isolate that may overshoot its bound before the engine stops it. isolated-vm calls that limit a guideline, not a strict limit, and says a determined attacker could use two to three times it before the script is terminated. So the manifests size the container over three times the product of the worker count, the executions per worker and the isolate bound. One check in the daemon’s own suite fails if a manifest sizes the container under that. deploy/k8s/standalone.yaml works out like this:

Term	Value there
MAX_WORKERS	4
EXECUTIONS_PER_WORKER	1
MEMORY_LIMIT_MB	128 megabytes
Sandboxes at once, MAX_WORKERS times EXECUTIONS_PER_WORKER	4
Their heaps at three times MEMORY_LIMIT_MB each	1536 megabytes
resources.limits.memory	2Gi, or 2048 mebibytes

2Gi clears the 1536 megabytes with room left for the four Node processes the workers run in. But three of the pool’s own bounds are not daemon settings: the process memory a worker retires at, the delay before a hung child is terminated, and the idle bound on retained handles. Deployments take the pool’s numbers for those, and the published API lists them beside every other pool option.

The pool replaces a worker after RETIRE_AFTER_EXECUTIONS runs, and whatever that worker accumulated goes with it. The nightly soak measures that every worker retires on that count and not on the process memory bound behind it. A retained execution is one whose caller asked to keep handles, and it occupies its worker’s slot and its isolate until the handles are closed or the idle bound expires. Neither that bound nor the slot is an operator setting, so a deployment that grants handles counts those executions in its worker budget.

Bounds a caller meets

DEFAULT_TIMEOUT_MS bounds a run whose request names no timeout of its own. MAX_TIMEOUT_MS is the longest any caller may ask for. Default: unset, and the ceiling is then the largest delay a Node timer accepts, near enough twenty-five days, so one caller may occupy one worker for as long as it likes. GRACE_MS is how long an execution has to stop after its time is up. Past that the executor ends the run and reports the timeout text. The executor protocol specifies the run clock and what stops it.

MAX_QUEUE bounds how many executions may wait for a free worker. Past that bound execute rejects with a retryable QUEUE_OVERFLOW fault instead of taking work the machine cannot finish. Callers retry that fault, and an operator scales on it.

RESPONSE_LIMIT_BYTES caps what one execution may return. Default: unset, and the protocol’s own cap is then the only one left. But that cap belongs to the protocol and not to the operator: everything a run sends is bounded where it is built, so a program cannot end the worker running it by returning too much.

Network the code reaches

A caller decides per execution whether the sandbox reaches the network, and the executor protocol specifies the three states it chooses from. OUTBOUND_MODE vetoes one of them. native, the default, lets the executor perform a request itself, and null refuses that however the caller asked. Callers that supply their own function, or that disable outbound outright, are untouched either way. But native applies no URL policy of its own, and the sandbox reaches whatever the daemon’s network reaches. A deployment that keeps native closes the network around the daemon instead.

Health, readiness and metrics

Liveness returns 200 while the process is up. Readiness reports whether the daemon is accepting work. But neither probe attempts an execution, so a pool that accepts work it cannot finish still reads ready. executor.pool.queue.depth catches that. Kubernetes takes an unready Pod out of its Service endpoints without restarting it, so a drain is expressed as readiness and never as health.

Every measurement goes through OpenTelemetry, and the daemon registers a Prometheus reader for its own process, so a scrape of /metrics works with no configuration. OTEL_SDK_DISABLED turns measurement off entirely, and it takes /metrics with it: the path returns 404 while liveness and readiness keep working. Manifests under deploy/k8s set scrape annotations unconditionally, so a deployment that disables the SDK strips those annotations and does not leave Prometheus scraping a 404. Traces and metrics also leave through OTLP when OTEL_EXPORTER_OTLP_ENDPOINT names a collector. What the daemon publishes, what each metric records, and the Prometheus name each is scraped under are in Telemetry.

Logging

On stdout the daemon writes one JSON object per line, and a container platform collects that without help. LOG_LEVEL chooses how much it writes: trace, debug, info, warn, error, fatal or silent. Default: info. At info it reports its version, runtime and pid as it starts, the address each listener took, the maintenance switch moving either way, and the signal that stopped it. But no level logs executions one by one. A line per execution is a cost that scales with traffic, and the same facts are already metrics. Callers a connection policy turned away are logged beside their remote address, and so is a policy that threw.

Scaling on the queue, not on the CPU

An executor spends much of its time inside waits it does not own, so CPU utilization can read low on a replica whose callers are queuing. executor.pool.queue.depth says how many callers are waiting for a worker. Every execution publishes the depth it found, so a replica publishes the series from its first execution and not from its first queued one. With a metrics adapter in the cluster, a horizontal autoscaler reads it directly:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: codemode-executor
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: codemode-executor
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: executor_pool_queue_depth
        target:
          type: AverageValue
          averageValue: '2'

minReplicas: 2 keeps a second replica for the moment the first one restarts, because a replica that is restarting serves nothing. averageValue: '2' adds a Pod once the queue is longer than about two executions per Pod, and a deployment tunes that number against the run durations it sees.

Draining and stopping

maintenance on refuses new connections with 503 Service Unavailable and lets the connections already open finish their executions. /readyz returns 503 while it is on. Maintenance takes a replica out of rotation without stopping it: to watch it, to let it empty before a restart, or to keep it back while something upstream is repaired. Turning it on flips readiness and leaves liveness alone:

curl -sS -X PUT http://127.0.0.1:8081/__maintenance__     # {"maintenance":true}
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/readyz    # 503
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/healthz   # 200
curl -sS -X DELETE http://127.0.0.1:8081/__maintenance__  # {"maintenance":false}

/healthz returns 200 through all of it, because the process is up and a platform reading maintenance as ill health would restart a working daemon. A stop signal starts the other drain. SHUTDOWN_DELAY_MS is how long the daemon goes on taking callers after the signal, the window a load balancer needs to stop sending it work. /readyz returns 503 for that whole window, so a rolling update takes the Pod out of rotation before the daemon stops accepting. SHUTDOWN_GRACE_MS bounds the whole stop, and the process exits on whatever is still running when it expires. That grace pays for the delay, for the drain that follows it, and for the two seconds the stop keeps back: the pool’s kill spacing, and the admin app and the telemetry flush closing after the protocol listener. SHUTDOWN_DELAY_MS has to be smaller than SHUTDOWN_GRACE_MS, and the daemon refuses to start otherwise.

Phase	An upgrade request	/readyz	/healthz	Admin app
Serving	101 Switching Protocols	200	200	200
maintenance on	503 Service Unavailable	503	200	200
Inside SHUTDOWN_DELAY_MS	101 Switching Protocols	503	200	200
Inside the drain past the delay	503 Service Unavailable	503	200	200
Past the drain	no answer	no answer	no answer	no answer

Past the delay the daemon stops accepting and waits for the executions still running. That drain takes no setting of its own: the daemon derives it from the grace less the delay, less the second of kill spacing the pool ends on, less the second kept for what closes after the protocol listener, and never below zero. A run that finishes inside the drain returns its result to its caller over the connection it arrived on. The pool cuts one still running when the drain ends, and its caller’s execute rejects with a retryable CONNECTION_BROKEN fault reading Worker <n> was shut down during the execution. So a deployment that wants a run of a given length to finish sets SHUTDOWN_GRACE_MS to at least SHUTDOWN_DELAY_MS plus that length plus the two seconds the stop keeps for the pool’s kill spacing and for what closes after the listener, and the delay stays the routing window a load balancer needs. The admin app answers for the whole stop, so status still works while the protocol listener drains.

PING_INTERVAL_MS is how often an idle caller connection is asked whether it is still there, and a connection that has not answered by the next round is terminated. MAX_CONNECTION_AGE_MS closes a caller connection once it has been open that long. Default: unset, and then no connection is closed on age. Clients reconnect, so a deployment that has just scaled out sets it to spread its callers over the new replicas. Without it they stay on the ones they first connected to.

Every setting

Setting	What it decides
HOST, PORT, SOCKET_PATH	Where callers connect
ADMIN, ADMIN_HOST, ADMIN_PORT	Where an operator connects
MAX_WORKERS, EXECUTIONS_PER_WORKER, MEMORY_LIMIT_MB	How much memory the deployment needs
RETIRE_AFTER_EXECUTIONS	How long a worker lives before it is replaced
DEFAULT_TIMEOUT_MS, MAX_TIMEOUT_MS, GRACE_MS	How long one execution may run
MAX_QUEUE	When a caller is told to retry instead of waiting
RESPONSE_LIMIT_BYTES	How much one execution may return
OUTBOUND_MODE	Whether the executor may fetch for the sandbox
SHUTDOWN_DELAY_MS, SHUTDOWN_GRACE_MS	How a stop signal is handled
PING_INTERVAL_MS, MAX_CONNECTION_AGE_MS	How long a caller connection lives
LOG_LEVEL	How much the daemon says

Every pnpm verify compares this table against the daemon’s own declarations, in both directions, so a setting that appears or leaves fails a lane instead of leaving a stale page behind. No check compares a default, so the defaults stay in codemode-executor --help.

Deploying

deploy/compose.yaml runs the daemon on one machine beside whatever calls it, through Docker Compose, and its own comments say where its posture is weaker than the manifests below. deploy/k8s/standalone.yaml runs it as its own Deployment and Service. deploy/k8s/same-pod.yaml runs it as a second container beside the host that calls it. That keeps the traffic on the loopback address and needs no Service at all.

Both Kubernetes manifests run the daemon without root and without privilege escalation, and Kernel policy the manifests carry quotes the rest of that posture from the manifest itself. Operators who apply one inherit it whole. No manifest can decide what surrounds the process, and Controls the operator owns is that list in full.