Tracing

A sandbox will happily run a command an agent wrote, but the result only tells you what it printed. Tracing records what it actually did: the programs it started, the files it read and wrote, the hosts it reached. The record is plain data you can print, diff, store or feed back to a model.

Turning it on

Tracing is off unless you ask for it, and it is chosen when the sandbox is created:

from vpod import Sandbox

with Sandbox.create(trace=True) as sandbox:
    result = sandbox.commands.run("sh -c 'cd /app && make'")

    for file in result.trace.files():
        if file.written:
            print(file.path)

Every result carries its own slice of the record, so result.trace is that command and nothing else. sandbox.trace.collect() gives you everything the sandbox has done so far.

first = sandbox.commands.run("touch /tmp/first")
second = sandbox.commands.run("touch /tmp/second")

first.trace.files()            # /tmp/first
second.trace.files()           # /tmp/second
sandbox.trace.collect().files()  # both

result.trace on a sandbox created without trace=True raises RuntimeError, rather than handing back an empty record that would read as “it did nothing”.

What ran

processes() returns the roots of a tree of programs:

def show(node, depth=0):
    print("  " * depth, node.pid, " ".join(node.argv), node.exit_code)
    for child in node.children:
        show(child, depth + 1)

for root in result.trace.processes():
    show(root)

sh -c cd /app && make None
make 2
cc -o build/out main.c 1
echo done 0

Field	Type	Description
`pid`	`Optional[int]`	Process id in the guest, `None` if it could not be resolved
`path`	`Optional[str]`	The binary that was executed
`argv`	`list[str]`	Arguments as the program received them
`exit_code`	`Optional[int]`	Exit code, `None` if the process had not exited yet
`started_at`	`int`	Guest time in nanoseconds when the program started
`children`	`list[ProcessNode]`	Programs that ran under this one

A node is one successful exec, not one process. That distinction is what makes the tree readable:

A shell that execs its last command reuses its own pid, so make appears nested under the sh that became it. Both nodes carry pid 563 above, and that is correct: one process, two programs.
A fork that never execs adds no node. Its children attach to the closest program that did exec, so a pipeline’s plumbing does not show up as empty boxes.
Threads are not programs. process.fork events with thread: true are ignored here.

What it touched

files() returns one entry per path, with everything that happened to it folded together:

for file in result.trace.files():
    print(file.path, "written" if file.written else "read")

Field	Type	Description
`path`	`str`	Absolute path in the guest
`read`	`bool`	Opened for reading, or read through a mount
`written`	`bool`	Opened for writing, truncated, or written through a mount
`created`	`bool`	The path did not exist before
`deleted`	`bool`	Unlinked or removed
`renamed_to`	`Optional[str]`	Where this path moved
`renamed_from`	`Optional[str]`	Where this path came from
`denied`	`bool`	The guest tried and got `EACCES` or `EPERM`
`processes`	`list[int]`	Pids that touched it

Paths are recorded where the file actually is, not as the program typed it. A program that does cd /tmp/rel and then writes copy.txt is recorded against /tmp/rel/copy.txt, and a child inherits its parent’s directory, so the resolution survives subprocess and shell pipelines. denied is worth watching on its own. A failed open is not noise, it is the most interesting line in the file:

blocked = [file.path for file in result.trace.files() if file.denied]

Where it went

network() returns one entry per address and port:

for host in result.trace.network():
    print(host.host or host.address, host.port, host.bytes_in, "failed" if host.failed else "")
    for request in host.requests:
        print("   ", request.method, request.url)

Field	Type	Description
`host`	`Optional[str]`	Hostname, from the DNS answer that produced this address
`address`	`str`	IP address
`port`	`int`	Port
`protocol`	`Optional[str]`	`"tcp"` or `"udp"`
`requests`	`list[HttpRequest]`	HTTP requests seen on this connection. See what a request carried
`bytes_out`	`int`	Bytes the guest sent
`bytes_in`	`int`	Bytes it received
`failed`	`bool`	Nothing on this address and port succeeded
`processes`	`list[int]`	Pids that reached it

Request lines appear when vpod handles the connection itself, which covers plain HTTP and the HTTPS path it proxies. A program that brings its own TLS end to end still gets its connection, byte counts and hostname recorded, but not the individual requests.

Following along

watch() hands you an iterator of raw events, useful when a sandbox runs for a long time and you want to react rather than wait:

import threading

with Sandbox.create(trace=True) as sandbox:
    def follow():
        for event in sandbox.trace.watch():
            if event["kind"] == "net.connect":
                print("reached", event.get("host") or event["address"])

    threading.Thread(target=follow, daemon=True).start()
    sandbox.commands.run("wget -q -O /tmp/page.html https://pypi.org/simple/")

The iterator ends when the sandbox closes, so the thread above finishes on its own.

Events reach a watcher when the record is drained, which happens at the end of every command and whenever you call collect(). A watcher sees a command’s events as that command finishes, not while it is still running.

clear() forgets what has been recorded so far, which keeps a long-lived sandbox from accumulating a record you are never going to read:

sandbox.trace.clear()

Was anything missed

complete answers the only question that matters before you trust a trace:

if not result.trace.complete:
    ...

It is False in three cases, all of which appear in the raw events:

Cause	What you see
The buffer filled and events were lost	a `trace.dropped` event carrying how many
The guest used a path the tracer cannot see	a `trace.blind` event carrying the reason
Process ids were never calibrated	events with `pid` set to `None`

The third is the mild one. vpod learns where the guest kernel keeps process ids by watching the guest tell it, which takes a couple of syscalls from two different processes. Until that settles, events are still recorded, but without pids and with relative paths left unresolved. In practice it settles during the first command.

Choosing what to record

trace=True records everything. Pass a dict to record less:

sandbox = Sandbox.create(trace={"files": True, "network": True})

Source	Records
`processes`	execs, exits and forks
`files`	opens, creates, renames, deletes, truncations in the guest filesystem
`network`	DNS, connects, listens, byte counts, HTTP requests with their headers and body
`mounts`	the same file activity on host directories you mounted, observed on the host side

buffer_bytes caps what the engine holds between drains, and defaults to 64 MB. When it fills, the oldest events survive and new ones are counted into a trace.dropped event, so the record tells you it is short rather than quietly lying:

sandbox = Sandbox.create(trace={"files": True, "buffer_bytes": 4 * 1024 * 1024})

Noise and vpod’s own plumbing

Two filters are on by default, both of which you can lift per call. Noise is activity that is real but never what you asked about: /proc, /sys, /dev, and shared libraries that were only read. vpod’s own plumbing is the handful of files and processes the SDK uses to run your command at all, marked internal in the raw events.

trace.files()                             # what the program did
trace.files(noise=True)                   # plus libraries and /proc
trace.files(internal=True, noise=True)    # plus vpod's own plumbing

network() and processes() take internal too.

What a request carried

A method and a URL say an agent called an API. They do not say which model it picked or what it pasted into the prompt. Network tracing records what the guest actually sent, so each request carries its headers and body:

with Sandbox.create(trace={"network": True}) as sandbox:
    sandbox.commands.run("python -c 'import app; app.sync()'")

    for host in sandbox.trace.collect().network():
        for request in host.requests:
            print(request.method, request.url)
            print(request.headers)
            print(request.body)

Field	Type	Description
`headers`	`dict[str, str]`	Every header sent, duplicates joined by `,`
`body_bytes`	`Optional[int]`	Length the sender declared, `None` when it declared none
`body`	`Optional[str]`	The body, or `None` when there was none to record
`body_encoding`	`Optional[str]`	`"utf8"`, or `"base64"` when the body was not text
`body_truncated`	`bool`	True when the body ran past 64 KB and was cut

Bodies are cut at 64 KB, so a large upload costs a bounded amount rather than filling the buffer. Ten API calls with 300-byte bodies grow a trace by about 7 KiB, measured.

A credential set with secrets appears here as its stand-in, never the real value, because the trace is taken before the gateway swaps it in. A real key placed in env is recorded as-is.

Responses are not recorded. A body with no declared length, such as a chunked upload, is not recorded either, and shows as body_bytes: None.

Raw events

Every view above is built from one flat list of events, and that list is public:

for event in result.trace.events:
    print(event["kind"], event.get("path") or event.get("argv"))

open("run.jsonl", "w").write(result.trace.to_jsonl())

Each event carries v (schema version), seq, guest_ns (guest time, which is deterministic), wall_ms (host wall clock), and kind. Most carry pid and task, and vpod’s own activity carries internal: true.

Kind	Fields worth knowing
`process.exec`	`path`, `argv`, `ppid`; a `result` field means the exec failed; `argv_truncated` when the arguments were long
`process.exit`	`code`
`process.fork`	`child_pid`, `thread`
`file.open`	`path`, `access`, `create`, `truncate`, `result`
`file.rename`	`from`, `to`, `result`
`file.delete`	`path`, `directory`, `result`
`dir.create`	`path`, `result`
`file.truncate`	`path`, `size`, `result`
`mount.open`, `mount.close`, `mount.create`, `mount.mkdir`, `mount.rename`, `mount.delete`, `mount.truncate`	the same on a mounted host directory; `mount.close` carries `bytes_read` and `bytes_written`
`net.dns`	`name`, `type`, `answers`, `error`
`net.connect`	`protocol`, `address`, `port`, `host`, `result`
`net.listen`	`protocol`, `address`, `port`
`net.flow`	`bytes_out`, `bytes_in`, `duration_ns`, `failed`
`net.udp`	`address`, `port`, `host`
`net.http`	`protocol`, `method`, `url`, `address`, `port`
`trace.dropped`	`count`
`trace.blind`	`reason`

A path that could not be resolved to an absolute one is left as the program wrote it and flagged with path_unresolved: true.

What a guest can and cannot hide from

Network and mount activity is observed outside the guest, in the emulator’s own device and host filesystem code. Nothing running inside the sandbox can perform that I/O without going through it, so those two sources are not evadable. Process and file activity is read from the guest’s syscalls, and a program that is trying to hide can get around that. io_uring is the practical route, so vpod turns it off when tracing starts. Guest root can turn it back on, which is why the engine also watches for it: the first ring that opens produces a trace.blind event and complete turns False. Writing kernel memory directly or loading a kernel module would evade it too. So a trace is a faithful record of what a program did, and complete tells you when it is not. It is not a security boundary. The boundary is the sandbox itself, which holds whether or not anything is being traced.

Cost

Tracing off is not a mode, it is one predictable branch per syscall, and measures as no change. On, measured on the wasm engine:

Workload	Off	On	Change
300 shell writes and deletes	0.100 s	0.105 s	+5.0%
150 `/bin/true` execs	0.589 s	0.602 s	+2.2%
`python3 -c` with five imports	0.139 s	0.139 s	−0.3%
Trace start, once per sandbox			+20 ms

CPU-bound work is unaffected, because nothing on the hot path changes. What costs is syscall volume.

Resumed sandboxes

Tracing is chosen per sandbox, including when you resume one:

resumed = Sandbox.resume(instance_id, trace=True)

A resumed sandbox starts a fresh record. The events from before the suspend are not carried across, so resumed.trace.collect() covers what has happened since it came back.

Get Started

TypeScript SDK

Python SDK

Embedded preview (In Iframe)

CLI

Resources

Turning it on

What ran

What it touched

Where it went

Following along

Was anything missed

Choosing what to record

Noise and vpod’s own plumbing

What a request carried

Raw events

What a guest can and cannot hide from

Cost

Resumed sandboxes

​Turning it on

​What ran

​What it touched

​Where it went

​Following along

​Was anything missed

​Choosing what to record

​Noise and vpod’s own plumbing

​What a request carried

​Raw events

​What a guest can and cannot hide from

​Cost

​Resumed sandboxes

Turning it on

What ran

What it touched

Where it went

Following along

Was anything missed

Choosing what to record

Noise and vpod’s own plumbing

What a request carried

Raw events

What a guest can and cannot hide from

Cost

Resumed sandboxes