Skip to content

Server and daemon

pool.execute takes a request and returns a result envelope. packages/executor/src/server owns everything between that call and a caller on another machine: the connection, the numbers a run is bounded by, and the probes a platform polls. apps/daemon owns the process around them. Node hosts that already run their own lifecycle take the server alone, through the ./server subpath of @supolka/codemode-executor, and keep their own configuration, logging and signals. The daemon is one composition of that server, with a command line, telemetry, an admin app and a stop handler around it.

The executor protocol states what crosses the wire, and the operator guide what a deployment decides. This chapter is the machinery under both, and it takes the pool as given.

createExecutorServer in packages/executor/src/server/listener.ts builds one http.Server, one ws server in noServer mode, and one WorkerPool from the pool options it was given. The HTTP server answers the probes. Every execution arrives through a WebSocket upgrade.

handleUpgrade reads the acceptance flag before anything else. Servers that have stopped accepting write 503 Service Unavailable on the socket and destroy it, and a socket destroyed with nothing more written on it reaches the caller as a network fault. stopAccepting sets that flag to false and resumeAccepting sets it back. Neither one touches an open connection or a running execution, and readiness reports the new value at once. But once close() has begun, resumeAccepting throws the INTERNAL fault A closed server cannot accept connections again.

Next comes the identity hook. onConnection receives a ConnectionHandshake: the request headers flattened into a string map, with repeated values joined by a comma and a space, and the remote address of the socket. It returns a ConnectionDecision: an identity string for a caller to admit, or reject: true with an optional reason string for one to turn away. Servers with no hook name every caller anonymous, and a daemon started from the command line is one of them, because no flag installs a hook. handleUpgrade catches whatever the hook throws, logs it and turns the connection away, so a policy that breaks admits nobody.

The listener writes 403 Forbidden with no reason in it, because a reason composed by the caller’s own policy would land inside a response header and open the response to header injection. It sends the reason to the logger beside the remote address instead, and moves executor.connection.rejections. The hook may be asynchronous and may resolve after close() began, so handleUpgrade reads the flag a second time before it completes the upgrade, and no session opens onto a pool that is shutting down. Controls the operator owns says what that identity is worth: it is attribution, and it scopes nothing.

Three refusals read back, from a raw upgrade with no Authorization header, from the same upgrade carrying the header the hook accepts once the server has stopped accepting, and from connect in this repository’s own client:

hook rejects: HTTP/1.1 403 Forbidden
not accepting: HTTP/1.1 503 Service Unavailable
the caller sees: CONNECTION_REJECTED Executor handshake failed for ws://127.0.0.1:8091

The second line is the acceptance flag on its own: that upgrade carries the header the hook accepts, and handleUpgrade returns before it ever calls the hook. Executor handshake failed is Session.open reporting that its version call failed, and both status lines reach a WebSocket client as a failed upgrade and never as a message, so the two refusals read alike from outside. Inside, only the 403 is written down: that branch logs the reason and moves the counter, while rejectUpgrade writes the 503 and touches neither.

An accepted socket joins the live set, takes its ping bookkeeping and its optional age timer, and then openRpcSession starts a capnweb session on it with a fresh ConnectionApi. That RpcTarget implements ExecutorApi and holds the bridge settings, the instrumentation and that connection’s identity for as long as the socket lives.

protocolVersion() returns the PROTOCOL_VERSION constant of the protocol package, and the caller does the comparing. Session.open in packages/cloudflare-codemode/src/session.ts calls it before anything else crosses the socket and compares with exact equality through assertProtocolCompatible. Disagreement is a PROTOCOL_MISMATCH fault raised on the caller’s side at connect time.

A caller’s dispatch function never leaves the caller’s process, and the guest that calls it runs two process boundaries away. Two independent capnweb sessions span that distance: the WebSocket session between the caller and the server, and the IPC session between the pool and one forked child. Their tables are separate, so no stub crosses from one to the other on its own. runBridgedExecution in packages/executor/src/server/bridge.ts copies each call from one session onto the other.

sequenceDiagram
    participant C as Caller
    participant S as Server
    participant P as Pool
    participant W as Child
    participant I as Isolate
    C->>S: execute with the dispatch and outbound stubs
    S->>S: validate the providers, duplicate the stubs, resolve the bounds
    S->>P: pool.execute with the resolved request
    P->>W: execute over the IPC session
    W->>I: fresh isolate, entry module
    I-->>W: __dispatch
    W-->>P: the same call over IPC
    P-->>S: the forwarding function
    S-->>C: the call over the WebSocket
    Note over S,C: a socket that died under this call gives the guest an error envelope and does not end the run
    C->>S: envelope JSON
    S->>P: the same string back
    P->>W: over IPC
    W->>I: the proxy resolves
    I->>W: the run settles
    W->>P: result envelope
    P->>S: the same envelope
    S->>S: measure it against the response limit
    S->>C: result, or the RESPONSE_TOO_LARGE fault
    Note over S: the guard flips with the return, so a tool call arriving later gets the completed envelope

capnweb delivers a function passed as a call parameter as an RPC stub, and the session disposes that stub when the call it arrived on returns. The bridge asserts that what arrived has a callable dup, raising The dispatch function did not arrive as an RPC stub. when it does not. It then works through a duplicate of its own, so it disposes what it took and leaves the session’s copy alone. That duplicate rides a DisposableStack that releases on every exit path, the pool rejecting included, and the child entry keeps the same discipline for what reaches it over IPC.

The forwarding function that reaches the pool closes over that duplicate, and it adds two things the caller’s own answer cannot carry. Calls arriving after the run settled get the pinned Execution has already completed. envelope and never reach a disposed stub. That guard flips before the duplicates die, because disposal runs in reverse order of registration. A transport failure under the call becomes an error envelope through describeFailure, so a dispatch whose connection died reaches the guest as a tool error instead of ending the run. Everything else passes through untouched: the arguments as one JSON string, the answer as the envelope the caller composed, and binary payloads still tagged. The protocol states that grammar, and the round trip of a tool call covers what the lower half of the path does with it.

Resolving the outbound function goes by the wire mode and not by its presence, because capnweb cannot tell an absent argument from an undefined one. stub duplicates the caller’s function and wraps it, disabled sends the pool null, and native sends undefined, or null when the operator’s veto says so. Anything else is a PROTOCOL_MISMATCH fault naming the mode, so an unknown mode can never decay into native by accident.

A duplicate outlives its call only for a retained execution. When the result carries an executionId and the request arrived in stub mode, the bridge stops releasing that duplicate with the call and hands it to releaseWithHandles. That subscribes to the pool’s handles.close event and disposes the duplicate when that execution’s handles end. It checks hasHandles immediately after the listener is placed, because the close can land between the result and the subscription. In the child, a matching duplicate stays in its own map for the same span. So a handler served by invokeHandle keeps fetching through the caller that started the execution, under the terms of execution handles.

runBridgedExecution fills in every number a pool request carries before it calls the pool, and the timeout is the only one a caller can move. A server configured with a 200 ms default and a 500 ms ceiling reports the number it resolved in the text of every run that hits the bound:

import { connect } from '@supolka/cloudflare-codemode-executor'
import { createExecutorServer } from '@supolka/codemode-executor/server'
const server = createExecutorServer({
host: '127.0.0.1',
port: 8092,
executor: { pool: { maxWorkers: 1 }, defaults: { timeout: 200, maxTimeout: 500 } },
})
await server.listen()
const executor = await connect('ws://127.0.0.1:8092')
const spin = 'async () => { for (;;) {} }'
for (const timeout of [undefined, 400, 5000, null] as const) {
const { error } = await executor.execute({ code: spin, timeout })
console.log(`asked for ${String(timeout)}:`, error?.split('.')[0])
}
await executor.dispose()
await server.close()
asked for undefined: Execution timed out at its 200 ms bound
asked for 400: Execution timed out at its 400 ms bound
asked for 5000: Execution timed out at its 500 ms bound
asked for null: Execution timed out at its 500 ms bound

maxTimeout is the ceiling in that run. A server that configures no default of its own falls to DEFAULT_TIMEOUT instead of the 200 above. MAX_SUPPORTED_TIMEOUT caps the ceiling itself, whatever an operator configured, because past the widest delay a 32-bit timer accepts Node fires a timer almost at once and a lifted bound would be no bound at all. Both constants are the protocol’s.

The grace and the isolate memory limit take no value from the request. They come from the server’s configured defaults, falling back to DEFAULT_GRACE and DEFAULT_MEMORY_LIMIT in packages/executor/src/executor/defaults.ts. Provider validation runs here as well, ahead of the pool: a descriptor list that cannot become the entry’s declarations comes back as an error inside the envelope, and no worker is leased, no stub is duplicated and no child is touched.

Five bounds belong to the server itself, above what the protocol fixes, and three of the five are options a host sets:

Bound Value What it does
responseLimit unset by default serializes the result envelope, measures it in UTF-8 bytes and raises the RESPONSE_TOO_LARGE fault above the bound, in place of a truncated answer
maxConnectionAge unset by default closes the connection with close code 1000 and the reason connection age limit, so the caller opens a new one and can reach a replica added after the first
pingInterval 30000 ms by default sends a ping to every socket that answered the previous round and terminates every socket that did not; zero or less builds no interval at all
keepAliveTimeout 72000 ms, a constant in listener.ts closes an idle HTTP keep-alive connection, above the sixty seconds an Application Load Balancer holds one by default
headersTimeout the keep-alive plus 2000 ms, the same constant file bounds how long the parser waits for the complete request headers

That measurement takes in the logs as well, because a run can outweigh its own result in console output. The run’s own value sits in the envelope as text, so its quotes and backslashes are escaped a second time and the count is larger than the value alone. Measuring precedes the send, because capnweb aborts the whole session when an incoming message passes the ceiling it puts on one message. That abort would cost every execution on the socket and not only the one that overran. A grant that dies at the limit closes its own handles first: the caller never learns the executionId, so nobody else could release the worker it holds. Handle invocations are measured against the same bound, their heads serialized as JSON and their bodies counted in bytes, since JSON.stringify cannot measure an ArrayBuffer.

The age timer and the heartbeat interval are both clamped to MAX_SUPPORTED_TIMEOUT and unreferenced, so neither one holds the process open. The keep-alive constant is the one an operator can outgrow: AWS takes that idle timeout as high as 4000 seconds, and a balancer set above 72 seconds needs listener.ts raised with it, or it reuses a connection this server is closing and answers its own caller with a gateway error.

packages/executor/src/server/probes.ts serves the probe paths over the same HTTP server the upgrades arrive on. It claims a request only for GET and HEAD, because a scrape can cost the host real work, and every other verb falls through to the listener’s own 404. Paths are read up to the query string, so a scraper that appends parameters is still served. One server answers these six polls, before and after stopAccepting:

/healthz 200 ok
/readyz 200 ready
/readyz?probe=k8s 200 ready
/metrics 404 not found
POST /healthz 404 not found
/readyz draining 503 not ready

not ready is the whole body of that 503. One predicate decides it, and the listener passes accepting && !draining. That predicate runs no execution and touches neither the workers nor the queue. Readiness gated on a live worker would never recover on a fresh replica, because spawning happens on demand and the traffic that creates the first worker is exactly what readiness withholds. Readiness gated on a full queue would take every replica out of the Service at once under an even load. That turns a typed and retryable fault into no address at all. A full queue rejects a caller with the QUEUE_OVERFLOW fault instead, and an operator watches the queue-depth gauge an autoscaler reads.

/metrics is claimed only while a metrics source is configured, so the unconfigured server above answers 404 and not an empty scrape. A render that throws is logged and answered 500 with metrics unavailable, so a collector that is down cannot leave the request hanging.

Executor packages measure themselves through @opentelemetry/api and register nothing. Without a registered SDK the API hands out no-op instruments, so an embedding host that wants no measurement pays almost nothing, and a host that wants it decides where the numbers go. That division is OpenTelemetry’s own guidance to library authors: a library depends on the API, and only the final application installs an SDK.

Scrapes follow the same rule. ServerOptions.metrics takes any object with a contentType string and a render that returns a promise for text, declared as MetricsSource in probes.ts. A source that renders one line is served end to end:

import { createExecutorServer } from '@supolka/codemode-executor/server'
const server = createExecutorServer({
host: '127.0.0.1',
port: 8094,
metrics: {
contentType: 'text/plain; version=0.0.4; charset=utf-8',
render: async () => 'executor_connection_active 1\n',
},
})
await server.listen()
const scrape = await fetch('http://127.0.0.1:8094/metrics')
console.log(scrape.status, scrape.headers.get('content-type'))
console.log(await scrape.text())
await server.close()
200 text/plain; version=0.0.4; charset=utf-8
executor_connection_active 1

contentType reaches the response header untouched, so the exposition version a Prometheus scrape reads is the source’s to declare. The server holds no exporter behind that seam, and the specs prove it against the real @opentelemetry/exporter-prometheus. That package is a development dependency here and not a shipped one. instrumented: false swaps in silentInstrumentation: its counters and its span do no work, and the pool carries the same switch.

Server measurements cover the part of a request the pool never sees. executor.connection.active rises and falls with the live set, and it falls only when the set actually held the socket, so a second close event cannot drive it below the truth. executor.connection.rejections counts what the hook turned away. Every execute runs inside an active executor.request span carrying executor.identity, so the pool’s own spans nest under it wherever the host registered a context manager. Telemetry names every instrument and what a Prometheus scrape renames it to.

createExecutorServer is the whole surface: one options object in, one handle out.

import { createExecutorServer } from '@supolka/codemode-executor/server'
const secret = process.env.SHARED_SECRET
const server = createExecutorServer({
host: '127.0.0.1',
port: 8080,
onConnection: ({ headers, remoteAddress }) =>
headers.authorization === secret
? { identity: remoteAddress }
: { reject: true, reason: 'the authorization header did not match' },
executor: {
pool: { maxWorkers: 4, maxQueue: 200 },
outboundMode: 'null',
defaults: { timeout: 30_000, maxTimeout: 120_000, memoryLimit: 256 },
},
responseLimit: 4 * 1024 * 1024,
shutdownDelay: 5000,
logger: console,
})
await server.listen()
console.log(server.address())
await server.close()
{ address: '127.0.0.1', family: 'IPv4', port: 8080 }

address() returns the HTTP server’s own address, so a host that asked for port 0 learns the port it got from there. listen() resolves when the port is open and rejects with whatever the HTTP server emitted, so an address already taken reaches the host as its own EADDRINUSE and not as a wrapped fault. socketPath replaces the host and the port with a unix socket. The handle also exposes the WorkerPool itself, so the events the pool publishes are available without a second subscription seam.

The server leaves the rest of a process to the host. It installs no signal handler, registers no OpenTelemetry provider, opens no second port and takes no setting from the environment. The environment reaches the pool in the default it hands a forked child and in the CODEMODE_TESTING switch the threat model accounts for, and nowhere else. Every other decision a deployment makes sits in the options object above, and startDaemon is one set of answers to those questions and not the only one.

commander owns the command line and zod owns the shape. Each setting is declared once in apps/daemon/src/cli/options.ts with its flags, its environment variable, its description and its default. Commander binds the variable to the flag, so a flag beats a variable and a variable beats the default. The daemon reads that provenance back out of commander, and the settings block of check prints it:

--max-workers is 1, from the flag.
--max-queue is 42, from the environment.

Only the settings away from their default are printed, and a daemon nobody has configured prints Every setting is at its default. in that block instead. zod then parses the strings a command line and an environment can only carry as text, and refuses the values that would leave the daemon unable to serve. Zero is a whole number, and it would give a daemon no worker, no execution slot, a queue that is full before the first caller, a child retired before its first execution, no heap, or a run that expires on arrival. So the schema refuses anything below 1 for the workers, the executions per worker, the queue, the retirement count, the memory limit and the timeout. Durations past the widest a timer accepts are refused for the reason the clamp above exists. So is a shutdown delay that is not smaller than the shutdown grace, because the delay is spent inside the grace and not added to it. Every problem is reported at once, each line naming both spellings of the setting it is about, in the shape --max-queue or MAX_QUEUE: Expected at least 1.

buildServerOptions in apps/daemon/src/config.ts is the only translation from that configuration to ServerOptions, and an absent metrics source turns instrumentation off in the pool and the server together. It sets maxWorkers, executionsPerWorker, maxQueue, retireAfterExecutions, the logger and whether the pool measures anything. The rest keep the pool’s own defaults, so the idle bound on retained handles, the worker memory bound behind retirement and the termination delay behind a hung child are not an operator’s to set.

startDaemon in apps/daemon/src/daemon.ts is the composition root. Telemetry is registered first, since the providers must exist before anything measures. Then the executor server listens, the pool counters subscribe to its events, and the admin app listens if it is turned on:

flowchart LR
    subgraph starting["taken in this order"]
        direction TB
        t1["telemetry providers"] --> s1["executor server listens"] --> c1["pool counters subscribe"] --> a1["admin app listens"]
    end
    subgraph stopping["released in this order"]
        direction TB
        s2["server drains, then closes"] --> a2["admin app closes"] --> t2["telemetry flushes"]
    end
    starting -. stop .-> stopping

Whatever the start took is held on one AsyncDisposableStack, and the daemon takes ownership of it through move() once the start succeeded. A step that fails disposes what came before it, so a failed start leaves nothing listening. But the protocol server is deferred last, both because it must be released first and because the window between its listen and its registration is covered explicitly in the failure path. Releasing runs in the inverse of the registration order, the same order a drain wants: the protocol port spends its delay draining while the admin app still answers, and the operator keeps status and maintenance for the whole shutdown.

countPool is a subscriber like any other, and it derives the worker, execution, queue and rejection counts from the pool’s own event emitter, so the status an operator reads is counted even in a process that registers no SDK and measures nothing.

The admin app is a hono app on a second listener, bound to the loopback address by default. It serves the process status at /__status__, the settings in force with the source of each at /__config__, and the maintenance switch as GET, PUT and DELETE on /__maintenance__. An unknown route answers a JSON 404 reading {"error":"No such route."}, and a failure inside the daemon answers a JSON 500 and not a stack. status and maintenance on the command line are a typed hono/client built from typeof app, so the two halves cannot drift. Every call that client makes carries a five second deadline, so a wedged daemon cannot hang the command. Maintenance drives the server’s acceptance switch, so readiness answers 503 while the executions already accepted run to their end.

startTelemetry in apps/daemon/src/telemetry.ts composes the pipeline explicitly, and not through the SDK’s own bootstrap. A Prometheus reader is registered whatever else is configured, so /metrics answers without an operator setting anything up. The daemon hands that reader to the listener through the MetricsSource seam and not through a second HTTP server. OTLP is added per signal when an endpoint variable names a collector, and registering the tracer provider also installs the context manager the executor’s spans nest under. stop shuts both providers down together and settles them even when one fails, so a collector that is down cannot restate a clean drain as a failed one. It disables the API globals afterwards, so the same process can start another pipeline. OTEL_SDK_DISABLED short-circuits all of it, and a daemon started that way answers /metrics with 404 while readiness still answers 200.

Three durations bound a stop: the shutdown delay and the drain behind it, both inside the server’s close(), and the shutdown grace around the daemon’s whole disposal. The first two run inside the last.

sequenceDiagram
    participant K as Platform and caller
    participant D as Daemon
    participant S as Server
    participant P as Pool
    K->>D: SIGTERM
    D->>S: close
    S->>S: draining, readyz answers 503
    K->>S: readyz, healthz, a new execution
    S->>K: 503, 200, and the execution runs
    Note over S: the shutdown delay ends here
    S->>P: shutdown, with the drain and the pool's own one second grace
    P->>P: wait up to the drain for the executions still running
    S->>K: the result of every run that finished inside it
    P->>P: SIGTERM to every worker, SIGKILL when the grace runs out
    S->>K: CONNECTION_BROKEN to whatever was still running
    S->>K: close 1001 on every socket
    D->>D: admin app closes, telemetry flushes
    Note over D: the shutdown grace bounds all of the above
    D->>K: exit

close() sets the draining flag first, so readiness answers 503 from the first moment while the listener keeps accepting for the whole delay. An endpoint controller needs that window, because it stops routing asynchronously and a caller it routed just before the flip would otherwise meet a closed port. When the delay ends, acceptance stops, the heartbeat interval is cleared, and pool.shutdown is called with the drain.

shutdown reads a ShutdownOptions, and close() fills its drainTimeout from shutdownDrain. A daemon derives that number in buildServerOptions: the shutdown grace less the shutdown delay, less the pool’s own second of kill spacing, less the second kept for the admin app and the telemetry flush that close after the server, and never below zero. The pool rejects its queued leases at once, waits up to the drain for every execution and every handle invocation still in flight, and signals the workers after that. So a running execution gets the delay and the drain together, and a caller whose execution outlives both reads this back:

CONNECTION_BROKEN true Worker 1 was shut down during the execution.

true there is retryable, so a caller with another replica to reach can send the same request again. Sizing those durations against the runs a deployment serves belongs to the operator guide. The --grace setting is a different number and never reaches this call: it is the margin added to a run’s own timeout. Once the pool is down, every open socket is closed with close code 1001 and the reason server shutting down, and the HTTP server closes. That promise is memoized, so a second close() cannot start a second drain.

Daemon.stop races the disposal of the whole stack against the shutdown grace and returns whether it finished. That answer travels on the stopped event. The command line installs close-with-grace with the same grace and calls stop from its handler, so a stop that finishes exits the process cleanly and a stop that outlives its grace is ended by close-with-grace instead of hanging.

apps/daemon/Dockerfile has a builder and a runtime stage, and both take the same ARG NODE_IMAGE. isolated-vm is compiled from source in the builder, and its binary has to meet at runtime the glibc it was compiled against. Stages naming their own base can produce a container that fails to load the engine, with only a linker error to read. Debian is the base this repository builds and supports, and it makes no claim about musl.

The builder installs corepack globally, since Node stopped shipping it in version 25. Corepack then reads the pnpm version out of the repository’s own packageManager field, not a second copy pinned in the Dockerfile. The builder also installs the toolchain that compilation needs, because the patch this repository carries changes isolated-vm’s C++ and the binaries the package ships were built without it. Copying the lockfile, the manifests and the patches before the sources lets the install layer survive every change to the code. tsc --build runs against the daemon’s own project references and not nx, because nx resolves its project graph across the whole workspace while .dockerignore leaves part of that workspace out of the build context. pnpm deploy --prod then writes the daemon, its production dependencies and the workspace packages it uses into one self-contained directory. The compiled engine is stripped of its debug sections, and its intermediate objects and foreign prebuilds are deleted, since node-gyp-build looks in build/Release first and would never load them. The last step of that stage runs the daemon’s own check inside the directory, so a binary the stripping damaged fails the image build and not every container started from it.

The runtime stage copies that directory owned by node and runs as the unprivileged node user. A symlink puts the entry point at /usr/local/bin/codemode-executor, so an operator can ask a running container for its status by name. Its HEALTHCHECK polls /healthz on the port the environment names, and docker ps and compose read that. Kubernetes polls the same path through its own probe.

Manifests under deploy/ are a deployment that has already made the operator’s decisions, and apps/daemon/src/deploy.spec.ts holds them to the daemon they run. Every variable they name is either a declared setting or an OTEL_ one, since a variable nobody reads is a setting an operator believes they have made. Both probes point at the port the container actually listens on, resolved through the named port and not a repeated number. Their termination grace period exceeds the shutdown grace, because SIGKILL arrives when it runs out. The memory limit covers at least three times the product of the workers, the executions per worker and the isolate limit: the upper end of the two to three times overshoot isolated-vm documents for a bound it calls a guideline. On the compose side the published port matches PORT, the stop grace period exceeds the shutdown grace, and the Dockerfile the compose file builds is the daemon’s own file byte for byte. A separate case starts a real daemon and asks it for every path the manifests and the image poll.

codemode-executor check reports whether a machine can serve at all. runCheck in apps/daemon/src/check.ts builds an IsolateExecutor with the configured timeout, memory limit, worker count and queue bound. One job then crosses the whole path an execution needs: a forked child, a fresh isolate, a timer on the host and an answer that arrives only after the isolate yields. It requires the result 42, so a sandbox that returns anything else fails the check like one that never started. Nothing escapes runCheck as a throw: a machine that cannot serve comes back as a result carrying a failure string. So --json prints one object a script can read, and the human form prints the reason beside the settings that are away from their default.

Exit codes are declared in apps/daemon/src/cli/index.ts, and codemode-executor --help prints all four under Exit codes, as does Ask the machine first. A clean run is 0. A taken address is 2, and exitCodeFor recognizes it by the EADDRINUSE code Node sets on the error and not by its text, because that is the one start failure an operator fixes somewhere other than the settings. Every other failure runCli catches is 1, a setting the schema refused among the rest. A container crash-looping on a machine that cannot run sandboxed code reports 5 while its command is check.

This socket has a client at its other end. The adapter is the client this repository ships, and the dialect it serves on top.