> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vpod.sh/llms.txt
> Use this file to discover all available pages before exploring further.
# Tracing
> Record what a sandbox did: which programs ran, which files changed, where it connected.
A sandbox will happily run a command an agent wrote, but the result only tells you what it printed. Tracing records what it actually did: the programs it started, the files it read and wrote, the hosts it reached. The record is plain data you can print, diff, store or feed back to a model.
## Turning it on
Tracing is off unless you ask for it, and it is chosen when the sandbox is created:
```python theme={null}
from vpod import Sandbox
with Sandbox.create(trace=True) as sandbox:
result = sandbox.commands.run("sh -c 'cd /app && make'")
for file in result.trace.files():
if file.written:
print(file.path)
```
Every result carries its own slice of the record, so `result.trace` is that command and nothing else. `sandbox.trace.collect()` gives you everything the sandbox has done so far.
```python theme={null}
first = sandbox.commands.run("touch /tmp/first")
second = sandbox.commands.run("touch /tmp/second")
first.trace.files() # /tmp/first
second.trace.files() # /tmp/second
sandbox.trace.collect().files() # both
```
`result.trace` on a sandbox created without `trace=True` raises `RuntimeError`, rather than handing back an empty record that would read as "it did nothing".
## What ran
`processes()` returns the roots of a tree of programs:
```python theme={null}
def show(node, depth=0):
print(" " * depth, node.pid, " ".join(node.argv), node.exit_code)
for child in node.children:
show(child, depth + 1)
for root in result.trace.processes():
show(root)
```
```
563 sh -c cd /app && make None
563 make 2
564 cc -o build/out main.c 1
566 echo done 0
```
| Field | Type | Description |
| :----------- | :------------------ | :---------------------------------------------------------- |
| `pid` | `Optional[int]` | Process id in the guest, `None` if it could not be resolved |
| `path` | `Optional[str]` | The binary that was executed |
| `argv` | `list[str]` | Arguments as the program received them |
| `exit_code` | `Optional[int]` | Exit code, `None` if the process had not exited yet |
| `started_at` | `int` | Guest time in nanoseconds when the program started |
| `children` | `list[ProcessNode]` | Programs that ran under this one |
A node is one successful exec, not one process. That distinction is what makes the tree readable:
* A shell that execs its last command reuses its own pid, so `make` appears nested under the `sh` that became it. Both nodes carry pid 563 above, and that is correct: one process, two programs.
* A fork that never execs adds no node. Its children attach to the closest program that did exec, so a pipeline's plumbing does not show up as empty boxes.
* Threads are not programs. `process.fork` events with `thread: true` are ignored here.
## What it touched
`files()` returns one entry per path, with everything that happened to it folded together:
```python theme={null}
for file in result.trace.files():
print(file.path, "written" if file.written else "read")
```
| Field | Type | Description |
| :------------- | :-------------- | :-------------------------------------------------------- |
| `path` | `str` | Absolute path in the guest |
| `read` | `bool` | Opened for reading, or read through a mount |
| `written` | `bool` | Opened for writing, truncated, or written through a mount |
| `created` | `bool` | The path did not exist before |
| `deleted` | `bool` | Unlinked or removed |
| `renamed_to` | `Optional[str]` | Where this path moved |
| `renamed_from` | `Optional[str]` | Where this path came from |
| `denied` | `bool` | The guest tried and got `EACCES` or `EPERM` |
| `processes` | `list[int]` | Pids that touched it |
Paths are recorded where the file actually is, not as the program typed it. A program that does `cd /tmp/rel` and then writes `copy.txt` is recorded against `/tmp/rel/copy.txt`, and a child inherits its parent's directory, so the resolution survives `subprocess` and shell pipelines.
`denied` is worth watching on its own. A failed open is not noise, it is the most interesting line in the file:
```python theme={null}
blocked = [file.path for file in result.trace.files() if file.denied]
```
## Where it went
`network()` returns one entry per address and port:
```python theme={null}
for host in result.trace.network():
print(host.host or host.address, host.port, host.bytes_in, "failed" if host.failed else "")
for request in host.requests:
print(" ", request.method, request.url)
```
| Field | Type | Description |
| :---------- | :------------------ | :------------------------------------------------------------------------------------------- |
| `host` | `Optional[str]` | Hostname, from the DNS answer that produced this address |
| `address` | `str` | IP address |
| `port` | `int` | Port |
| `protocol` | `Optional[str]` | `"tcp"` or `"udp"` |
| `requests` | `list[HttpRequest]` | HTTP requests seen on this connection. See [what a request carried](#what-a-request-carried) |
| `bytes_out` | `int` | Bytes the guest sent |
| `bytes_in` | `int` | Bytes it received |
| `failed` | `bool` | Nothing on this address and port succeeded |
| `processes` | `list[int]` | Pids that reached it |
Request lines appear when vpod handles the connection itself, which covers plain HTTP and the HTTPS path it proxies. A program that brings its own TLS end to end still gets its connection, byte counts and hostname recorded, but not the individual requests.
## Following along
`watch()` hands you an iterator of raw events, useful when a sandbox runs for a long time and you want to react rather than wait:
```python theme={null}
import threading
with Sandbox.create(trace=True) as sandbox:
def follow():
for event in sandbox.trace.watch():
if event["kind"] == "net.connect":
print("reached", event.get("host") or event["address"])
threading.Thread(target=follow, daemon=True).start()
sandbox.commands.run("wget -q -O /tmp/page.html https://pypi.org/simple/")
```
The iterator ends when the sandbox closes, so the thread above finishes on its own.
Events reach a watcher when the record is drained, which happens at the end of every command and whenever you call `collect()`. A watcher sees a command's events as that command finishes, not while it is still running.
`clear()` forgets what has been recorded so far, which keeps a long-lived sandbox from accumulating a record you are never going to read:
```python theme={null}
sandbox.trace.clear()
```
## Was anything missed
`complete` answers the only question that matters before you trust a trace:
```python theme={null}
if not result.trace.complete:
...
```
It is `False` in three cases, all of which appear in the raw events:
| Cause | What you see |
| :------------------------------------------ | :---------------------------------------- |
| The buffer filled and events were lost | a `trace.dropped` event carrying how many |
| The guest used a path the tracer cannot see | a `trace.blind` event carrying the reason |
| Process ids were never calibrated | events with `pid` set to `None` |
The third is the mild one. vpod learns where the guest kernel keeps process ids by watching the guest tell it, which takes a couple of syscalls from two different processes. Until that settles, events are still recorded, but without pids and with relative paths left unresolved. In practice it settles during the first command.
## Choosing what to record
`trace=True` records everything. Pass a dict to record less:
```python theme={null}
sandbox = Sandbox.create(trace={"files": True, "network": True})
```
| Source | Records |
| :---------- | :-------------------------------------------------------------------------------- |
| `processes` | execs, exits and forks |
| `files` | opens, creates, renames, deletes, truncations in the guest filesystem |
| `network` | DNS, connects, listens, byte counts, HTTP requests with their headers and body |
| `mounts` | the same file activity on host directories you mounted, observed on the host side |
`buffer_bytes` caps what the engine holds between drains, and defaults to 64 MB. When it fills, the oldest events survive and new ones are counted into a `trace.dropped` event, so the record tells you it is short rather than quietly lying:
```python theme={null}
sandbox = Sandbox.create(trace={"files": True, "buffer_bytes": 4 * 1024 * 1024})
```
## Noise and vpod's own plumbing
Two filters are on by default, both of which you can lift per call.
Noise is activity that is real but never what you asked about: `/proc`, `/sys`, `/dev`, and shared libraries that were only read. vpod's own plumbing is the handful of files and processes the SDK uses to run your command at all, marked `internal` in the raw events.
```python theme={null}
trace.files() # what the program did
trace.files(noise=True) # plus libraries and /proc
trace.files(internal=True, noise=True) # plus vpod's own plumbing
```
`network()` and `processes()` take `internal` too.
## What a request carried
A method and a URL say an agent called an API. They do not say which model it
picked or what it pasted into the prompt. Network tracing records what the guest
actually sent, so each request carries its headers and body:
```python theme={null}
with Sandbox.create(trace={"network": True}) as sandbox:
sandbox.commands.run("python -c 'import app; app.sync()'")
for host in sandbox.trace.collect().network():
for request in host.requests:
print(request.method, request.url)
print(request.headers)
print(request.body)
```
| Field | Type | Description |
| :--------------- | :--------------- | :------------------------------------------------------- |
| `headers` | `dict[str, str]` | Every header sent, duplicates joined by `, ` |
| `body_bytes` | `Optional[int]` | Length the sender declared, `None` when it declared none |
| `body` | `Optional[str]` | The body, or `None` when there was none to record |
| `body_encoding` | `Optional[str]` | `"utf8"`, or `"base64"` when the body was not text |
| `body_truncated` | `bool` | True when the body ran past 64 KB and was cut |
Bodies are cut at 64 KB, so a large upload costs a bounded amount rather than
filling the buffer. Ten API calls with 300-byte bodies grow a trace by about
7 KiB, measured.
A credential set with [`secrets`](/sdk/python/sandbox#secrets) appears here as its
stand-in, never the real value, because the trace is taken before the gateway
swaps it in. A real key placed in `env` is recorded as-is.
Responses are not recorded. A body with no declared length, such as a chunked
upload, is not recorded either, and shows as `body_bytes: None`.
## Raw events
Every view above is built from one flat list of events, and that list is public:
```python theme={null}
for event in result.trace.events:
print(event["kind"], event.get("path") or event.get("argv"))
open("run.jsonl", "w").write(result.trace.to_jsonl())
```
Each event carries `v` (schema version), `seq`, `guest_ns` (guest time, which is deterministic), `wall_ms` (host wall clock), and `kind`. Most carry `pid` and `task`, and vpod's own activity carries `internal: true`.
| Kind | Fields worth knowing |
| :----------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------ |
| `process.exec` | `path`, `argv`, `ppid`; a `result` field means the exec failed; `argv_truncated` when the arguments were long |
| `process.exit` | `code` |
| `process.fork` | `child_pid`, `thread` |
| `file.open` | `path`, `access`, `create`, `truncate`, `result` |
| `file.rename` | `from`, `to`, `result` |
| `file.delete` | `path`, `directory`, `result` |
| `dir.create` | `path`, `result` |
| `file.truncate` | `path`, `size`, `result` |
| `mount.open`, `mount.close`, `mount.create`, `mount.mkdir`, `mount.rename`, `mount.delete`, `mount.truncate` | the same on a mounted host directory; `mount.close` carries `bytes_read` and `bytes_written` |
| `net.dns` | `name`, `type`, `answers`, `error` |
| `net.connect` | `protocol`, `address`, `port`, `host`, `result` |
| `net.listen` | `protocol`, `address`, `port` |
| `net.flow` | `bytes_out`, `bytes_in`, `duration_ns`, `failed` |
| `net.udp` | `address`, `port`, `host` |
| `net.http` | `protocol`, `method`, `url`, `address`, `port` |
| `trace.dropped` | `count` |
| `trace.blind` | `reason` |
A path that could not be resolved to an absolute one is left as the program wrote it and flagged with `path_unresolved: true`.
## What a guest can and cannot hide from
Network and mount activity is observed outside the guest, in the emulator's own device and host filesystem code. Nothing running inside the sandbox can perform that I/O without going through it, so those two sources are not evadable.
Process and file activity is read from the guest's syscalls, and a program that is trying to hide can get around that. `io_uring` is the practical route, so vpod turns it off when tracing starts. Guest root can turn it back on, which is why the engine also watches for it: the first ring that opens produces a `trace.blind` event and `complete` turns `False`. Writing kernel memory directly or loading a kernel module would evade it too.
So a trace is a faithful record of what a program did, and `complete` tells you when it is not. It is not a security boundary. The boundary is the sandbox itself, which holds whether or not anything is being traced.
## Cost
Tracing off is not a mode, it is one predictable branch per syscall, and measures as no change.
On, measured on the wasm engine:
| Workload | Off | On | Change |
| :----------------------------- | :------ | :------ | :----- |
| 300 shell writes and deletes | 0.100 s | 0.105 s | +5.0% |
| 150 `/bin/true` execs | 0.589 s | 0.602 s | +2.2% |
| `python3 -c` with five imports | 0.139 s | 0.139 s | −0.3% |
| Trace start, once per sandbox | | | +20 ms |
CPU-bound work is unaffected, because nothing on the hot path changes. What costs is syscall volume.
## Resumed sandboxes
Tracing is chosen per sandbox, including when you resume one:
```python theme={null}
resumed = Sandbox.resume(instance_id, trace=True)
```
A resumed sandbox starts a fresh record. The events from before the suspend are not carried across, so `resumed.trace.collect()` covers what has happened since it came back.