> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vpod.sh/llms.txt
> Use this file to discover all available pages before exploring further.
# Tracing
> Record what a sandbox did: which programs ran, which files changed, where it connected.
A sandbox will happily run a command an agent wrote, but the result only tells
you what it printed. Tracing records what it actually did: the programs it
started, the files it read and wrote, the hosts it reached. The record is plain
data you can print, diff, store or feed back to a model.
## Turning it on
Tracing is off unless you ask for it, and it is chosen when the sandbox is
created:
```ts theme={null}
import { Sandbox } from "vpod";
const sandbox = await Sandbox.create({ trace: true });
const result = await sandbox.commands.run("sh -c 'cd /app && make'");
for (const file of result.trace.files()) {
if (file.written) console.log(file.path);
}
```
Every result carries its own slice of the record, so `result.trace` is that
command and nothing else. `sandbox.trace.collect()` gives you everything the
sandbox has done so far.
```ts theme={null}
const first = await sandbox.commands.run("touch /tmp/first");
const second = await sandbox.commands.run("touch /tmp/second");
first.trace.files(); // /tmp/first
second.trace.files(); // /tmp/second
(await sandbox.trace.collect()).files(); // both
```
Once the record is in hand the views are synchronous. Only reaching into the
sandbox for new events is a promise, which is `collect()` and `clear()`.
`result.trace` on a sandbox created without `trace: true` throws, rather than
handing back an empty record that would read as "it did nothing".
## What ran
`processes()` returns the roots of a tree of programs:
```ts theme={null}
import type { ProcessNode } from "vpod";
function show(node: ProcessNode, depth = 0) {
console.log(" ".repeat(depth), node.pid, node.argv.join(" "), node.exitCode);
for (const child of node.children) show(child, depth + 1);
}
for (const root of result.trace.processes()) show(root);
```
```
563 sh -c cd /app && make null
563 make 2
564 cc -o build/out main.c 1
566 echo done 0
```
| Field | Type | Description |
| :---------- | :--------------- | :---------------------------------------------------------- |
| `pid` | `number \| null` | Process id in the guest, `null` if it could not be resolved |
| `path` | `string \| null` | The binary that was executed |
| `argv` | `string[]` | Arguments as the program received them |
| `exitCode` | `number \| null` | Exit code, `null` if the process had not exited yet |
| `startedAt` | `number` | Guest time in nanoseconds when the program started |
| `children` | `ProcessNode[]` | Programs that ran under this one |
A node is one successful exec, not one process. That distinction is what makes
the tree readable:
* A shell that execs its last command reuses its own pid, so `make` appears
nested under the `sh` that became it. Both nodes carry pid 563 above, and that
is correct: one process, two programs.
* A fork that never execs adds no node. Its children attach to the closest
program that did exec, so a pipeline's plumbing does not show up as empty
boxes.
* Threads are not programs. `process.fork` events with `thread: true` are
ignored here.
## What it touched
`files()` returns one entry per path, with everything that happened to it folded
together:
```ts theme={null}
for (const file of result.trace.files()) {
console.log(file.path, file.written ? "written" : "read");
}
```
| Field | Type | Description |
| :------------ | :--------------- | :-------------------------------------------------------- |
| `path` | `string` | Absolute path in the guest |
| `read` | `boolean` | Opened for reading, or read through a mount |
| `written` | `boolean` | Opened for writing, truncated, or written through a mount |
| `created` | `boolean` | The path did not exist before |
| `deleted` | `boolean` | Unlinked or removed |
| `renamedTo` | `string \| null` | Where this path moved |
| `renamedFrom` | `string \| null` | Where this path came from |
| `denied` | `boolean` | The guest tried and got `EACCES` or `EPERM` |
| `processes` | `number[]` | Pids that touched it |
Paths are recorded where the file actually is, not as the program typed it. A
program that does `cd /tmp/rel` and then writes `copy.txt` is recorded against
`/tmp/rel/copy.txt`, and a child inherits its parent's directory, so the
resolution survives child processes and shell pipelines.
`denied` is worth watching on its own. A failed open is not noise, it is the
most interesting line in the file:
```ts theme={null}
const blocked = result.trace.files().filter((file) => file.denied);
```
## Where it went
`network()` returns one entry per address and port:
```ts theme={null}
for (const host of result.trace.network()) {
console.log(host.host ?? host.address, host.port, host.bytesIn, host.failed ? "failed" : "");
for (const request of host.requests) console.log(" ", request.method, request.url);
}
```
| Field | Type | Description |
| :---------- | :--------------- | :------------------------------------------------------------------------------------------- |
| `host` | `string \| null` | Hostname, from the DNS answer that produced this address |
| `address` | `string` | IP address |
| `port` | `number` | Port |
| `protocol` | `string \| null` | `"tcp"` or `"udp"` |
| `requests` | `HttpRequest[]` | HTTP requests seen on this connection. See [what a request carried](#what-a-request-carried) |
| `bytesOut` | `number` | Bytes the guest sent |
| `bytesIn` | `number` | Bytes it received |
| `failed` | `boolean` | Nothing on this address and port succeeded |
| `processes` | `number[]` | Pids that reached it |
Request lines appear when vpod handles the connection itself, which covers plain
HTTP and the HTTPS path it proxies. A program that brings its own TLS end to end
still gets its connection, byte counts and hostname recorded, but not the
individual requests.
## What a request carried
A method and a URL say an agent called an API. They do not say which model it
picked or what it pasted into the prompt. Network tracing records what the guest
actually sent, so each request carries its headers and body:
```ts theme={null}
const sandbox = await Sandbox.create({ trace: { network: true } });
await sandbox.commands.run("python -c 'import app; app.sync()'");
for (const host of (await sandbox.trace.collect()).network()) {
for (const request of host.requests) {
console.log(request.method, request.url);
console.log(request.headers);
console.log(request.body);
}
}
```
| Field | Type | Description |
| :-------------- | :----------------------- | :------------------------------------------------------- |
| `headers` | `Record` | Every header sent, duplicates joined by `, ` |
| `bodyBytes` | `number \| null` | Length the sender declared, `null` when it declared none |
| `body` | `string \| null` | The body, or `null` when there was none to record |
| `bodyEncoding` | `string \| null` | `"utf8"`, or `"base64"` when the body was not text |
| `bodyTruncated` | `boolean` | True when the body ran past 64 KB and was cut |
Bodies are cut at 64 KB, so a large upload costs a bounded amount rather than
filling the buffer. Ten API calls with 300-byte bodies grow a trace by about
7 KiB, measured.
A credential set with [`secrets`](/sdk/typescript/sandbox#secrets) appears here as
its stand-in, never the real value, because the trace is taken before the gateway
swaps it in. A real key placed in `env` is recorded as-is.
Responses are not recorded. A body with no declared length, such as a chunked
upload, is not recorded either, and shows as `bodyBytes: null`.
## Following along
`watch()` is an async iterable of raw events, useful when a sandbox runs for a
long time and you want to react rather than wait:
```ts theme={null}
const sandbox = await Sandbox.create({ trace: true });
void (async () => {
for await (const event of sandbox.trace.watch()) {
if (event.kind === "net.connect") console.log("reached", event.host ?? event.address);
}
})();
await sandbox.commands.run("wget -q -O /tmp/page.html https://pypi.org/simple/");
```
The iteration ends when the sandbox closes, so the loop above finishes on its
own.
Events reach a watcher when the record is drained, which happens at the end of
every command and whenever you call `collect()`. A watcher sees a command's
events as that command finishes, not while it is still running.
`clear()` forgets what has been recorded so far, which keeps a long-lived
sandbox from accumulating a record you are never going to read:
```ts theme={null}
await sandbox.trace.clear();
```
## Was anything missed
`complete` answers the only question that matters before you trust a trace:
```ts theme={null}
if (!result.trace.complete) {
// ...
}
```
It is `false` in three cases, all of which appear in the raw events:
| Cause | What you see |
| :------------------------------------------ | :---------------------------------------- |
| The buffer filled and events were lost | a `trace.dropped` event carrying how many |
| The guest used a path the tracer cannot see | a `trace.blind` event carrying the reason |
| Process ids were never calibrated | events with `pid` set to `null` |
The third is the mild one. vpod learns where the guest kernel keeps process ids
by watching the guest tell it, which takes a couple of syscalls from two
different processes. Until that settles, events are still recorded, but without
pids and with relative paths left unresolved. In practice it settles during the
first command.
## Choosing what to record
`trace: true` records everything. Pass an object to record less:
```ts theme={null}
const sandbox = await Sandbox.create({ trace: { files: true, network: true } });
```
| Source | Records |
| :---------- | :-------------------------------------------------------------------------------- |
| `processes` | execs, exits and forks |
| `files` | opens, creates, renames, deletes, truncations in the guest filesystem |
| `network` | DNS, connects, listens, byte counts, HTTP requests with their headers and body |
| `mounts` | the same file activity on host directories you mounted, observed on the host side |
`bufferBytes` caps what the engine holds between drains, and defaults to 64 MB.
When it fills, the oldest events survive and new ones are counted into a
`trace.dropped` event, so the record tells you it is short rather than quietly
lying:
```ts theme={null}
const sandbox = await Sandbox.create({
trace: { files: true, bufferBytes: 4 * 1024 * 1024 },
});
```
## Noise and vpod's own plumbing
Two filters are on by default, both of which you can lift per call.
Noise is activity that is real but never what you asked about: `/proc`, `/sys`,
`/dev`, and shared libraries that were only read. vpod's own plumbing is the
handful of files and processes the SDK uses to run your command at all, marked
`internal` in the raw events.
```ts theme={null}
trace.files(); // what the program did
trace.files({ noise: true }); // plus libraries and /proc
trace.files({ internal: true, noise: true }); // plus vpod's own plumbing
```
`network()` and `processes()` take `internal` too.
## Raw events
Every view above is built from one flat list of events, and that list is public:
```ts theme={null}
for (const event of result.trace.events) {
console.log(event.kind, event.path ?? event.argv);
}
await writeFile("run.jsonl", result.trace.toJSONL());
```
Each event carries `v` (schema version), `seq`, `guest_ns` (guest time, which is
deterministic), `wall_ms` (host wall clock), and `kind`. Raw events keep the wire
spelling, so their fields stay snake\_case while the views above are camelCase.
Most carry `pid` and `task`, and vpod's own activity carries `internal: true`.
| Kind | Fields worth knowing |
| :----------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------ |
| `process.exec` | `path`, `argv`, `ppid`; a `result` field means the exec failed; `argv_truncated` when the arguments were long |
| `process.exit` | `code` |
| `process.fork` | `child_pid`, `thread` |
| `file.open` | `path`, `access`, `create`, `truncate`, `result` |
| `file.rename` | `from`, `to`, `result` |
| `file.delete` | `path`, `directory`, `result` |
| `dir.create` | `path`, `result` |
| `file.truncate` | `path`, `size`, `result` |
| `mount.open`, `mount.close`, `mount.create`, `mount.mkdir`, `mount.rename`, `mount.delete`, `mount.truncate` | the same on a mounted host directory; `mount.close` carries `bytes_read` and `bytes_written` |
| `net.dns` | `name`, `type`, `answers`, `error` |
| `net.connect` | `protocol`, `address`, `port`, `host`, `result` |
| `net.listen` | `protocol`, `address`, `port` |
| `net.flow` | `bytes_out`, `bytes_in`, `duration_ns`, `failed` |
| `net.udp` | `address`, `port`, `host` |
| `net.http` | `protocol`, `method`, `url`, `address`, `port` |
| `trace.dropped` | `count` |
| `trace.blind` | `reason` |
A path that could not be resolved to an absolute one is left as the program
wrote it and flagged with `path_unresolved: true`.
## What a guest can and cannot hide from
Network activity is observed outside the guest, in the emulator's own device
code, as is activity on a mounted host directory. Nothing running inside the
sandbox can perform that I/O without going through it, so those two sources are
not evadable.
Process and file activity is read from the guest's syscalls, and a program that
is trying to hide can get around that. `io_uring` is the practical route, so
vpod turns it off when tracing starts. Guest root can turn it back on, which is
why the engine also watches for it: the first ring that opens produces a
`trace.blind` event and `complete` turns `false`. Writing kernel memory directly
or loading a kernel module would evade it too.
So a trace is a faithful record of what a program did, and `complete` tells you
when it is not. It is not a security boundary. The boundary is the sandbox
itself, which holds whether or not anything is being traced.
## Cost
Tracing off is not a mode, it is one predictable branch per syscall, and
measures as no change.
On, measured on the wasm engine:
| Workload | Off | On | Change |
| :----------------------------- | :------ | :------ | :----- |
| 300 shell writes and deletes | 0.100 s | 0.105 s | +5.0% |
| 150 `/bin/true` execs | 0.589 s | 0.602 s | +2.2% |
| `python3 -c` with five imports | 0.139 s | 0.139 s | −0.3% |
| Trace start, once per sandbox | | | +20 ms |
CPU-bound work is unaffected, because nothing on the hot path changes. What
costs is syscall volume.
## Resumed sandboxes
Tracing is chosen per sandbox, including when you resume one:
```ts theme={null}
const resumed = await Sandbox.resume(instanceId, { trace: true });
```
A resumed sandbox starts a fresh record. The events from before the suspend are
not carried across, so `resumed.trace.collect()` covers what has happened since
it came back.