f22d0a5791
Source commit: 309831e1e021f962c118452336776fd9a94025f9 Public tree identity: sha256:6fa95c1745579bd6256dbeb3d476db0b07c2f24aa9213b0f7783ce1adfc8aca5
3559 lines
92 KiB
Markdown
3559 lines
92 KiB
Markdown
# Disasmer MVP Engineering Plan
|
|
|
|
**Status:** draft MVP execution plan
|
|
**Audience:** engineering / founder-engineer
|
|
**Goal:** build a public-launchable MVP that proves Disasmer can replace CI/build systems with a general distributed Wasm runtime whose flagship use case is distributed, local-first, source-debuggable builds.
|
|
|
|
---
|
|
|
|
## 0. Executive summary
|
|
|
|
Disasmer should launch as a **general distributed Wasm runtime** with a **build-system experience so strong that it is the obvious wedge**. The MVP is not a toy local scheduler and not merely a CI runner with nicer YAML. It must demonstrate the core thesis:
|
|
|
|
> A build can be a normal Rust program, running as one virtual process across multiple machines, with build steps represented as virtual threads/tasks, artifacts represented by a low-latency virtual filesystem, and debugging represented as one normal VS Code debug session.
|
|
|
|
The launchable MVP should include:
|
|
|
|
- Open-source local runtime, SDK, CLI, node runtime, protocol definitions, and VS Code extension.
|
|
- Hosted coordinator from day one, with persistent project/identity state in Postgres and private hosted Authentik/OIDC/community-policy modules; the community tier coordinates but does not provide arbitrary hosted compute.
|
|
- User-attached Linux and Windows nodes.
|
|
- Containerfile-based environment discovery for MVP user projects.
|
|
- Rootless Podman backend for Linux command execution.
|
|
- Windows command backend sufficient for executing commands unsandboxed, with the Windows sandboxing backend explicitly stubbed (to be implemented later) and isolated behind an interface.
|
|
- Wasmtime-based execution for Disasmer guest/orchestration code.
|
|
- Native command execution only through node runtimes, never through the hosted coordinator control plane.
|
|
- Small serializable task arguments and results.
|
|
- Large inputs/outputs passed through handles: `SourceSnapshot`, `Artifact`, `Blob`, and VFS manifests.
|
|
- Local-first VFS where `flush()` publishes metadata and `sync()` triggers explicit user-chosen export/transfer/durability behavior; artifacts are otherwise ephemeral node-local data.
|
|
- Git support as an optional source-provider module that is included by default, with non-Git providers pluggable and dependencies avoidable when unused.
|
|
- All-stop debugging with Debug Epochs as a mandatory MVP capability.
|
|
- A Disasmer DAP adapter that targets VS Code first while staying usable by other DAP clients.
|
|
- Task restart from filesystem/artifact checkpoint boundaries, not live stack migration or hot-swapping.
|
|
- Abuse controls designed into the object model and control plane, not bolted on later.
|
|
- Operator panel foundations in the protocol and runtime-state model, with the full immediate-mode panel as the second product milestone.
|
|
|
|
The flagship demo should make people immediately understand why this is not “yet another CI system”:
|
|
|
|
1. The repo has `envs/linux/Containerfile` and `envs/windows/Containerfile`.
|
|
2. The build is one Rust source file using `#[disasmer::main]`, `#[disasmer::task]`, `spawn::task`, `env!("linux")`, `env!("windows")`, `cmd!(...)`, `fs::flush()`, and `fs::sync() / artifact::export_to_node()`.
|
|
3. A Linux node and a Windows node can be attached to the hosted coordinator.
|
|
4. Pressing F5 in VS Code starts one virtual process.
|
|
5. VS Code shows `main`, `compile linux`, `compile windows`, and `package` as threads of one debug target.
|
|
6. A breakpoint in the Windows build task stops the whole virtual process using a Debug Epoch.
|
|
7. The user edits the build logic, restarts only the failed task from a clean VFS checkpoint, and continues.
|
|
8. Artifacts appear through the VFS without coordinator-routed bulk data.
|
|
|
|
---
|
|
|
|
## 1. Product stance and launch bar
|
|
|
|
### 1.1 Positioning
|
|
|
|
Disasmer should be positioned as:
|
|
|
|
> A distributed Wasm runtime that makes distributed execution feel like ordinary source-level-debuggable multithreaded programming, with build systems as the first killer use case.
|
|
|
|
It is not primarily:
|
|
|
|
- a hosted CI SaaS;
|
|
- a YAML syntax replacement;
|
|
- a remote shell wrapper;
|
|
- a Kubernetes abstraction;
|
|
- a general-purpose compute subsidy;
|
|
- a task queue with logs;
|
|
- or a build cache alone.
|
|
|
|
It should feel closer to:
|
|
|
|
> “What if my build system were an actual program, and the debugger saw the whole distributed build as one process?”
|
|
|
|
### 1.2 Public-launch promise
|
|
|
|
The launch should be built around a concrete promise:
|
|
|
|
```text
|
|
Add a Containerfile.
|
|
Reference it from Rust with env!("name").
|
|
Press F5.
|
|
Debug the distributed build like one normal program.
|
|
```
|
|
|
|
The public launch should be able to show a short video or GIF with these beats:
|
|
|
|
```text
|
|
1. VS Code open on a Rust build program.
|
|
2. env!("linux") and env!("windows") autocomplete from envs/**/Containerfile.
|
|
3. Press F5.
|
|
4. Hosted coordinator shows two attached nodes.
|
|
5. Linux and Windows build tasks start.
|
|
6. VS Code thread list shows virtual threads.
|
|
7. Breakpoint hits in the Windows task.
|
|
8. Both tasks stop.
|
|
9. User inspects args/locals/artifact handles.
|
|
10. User restarts only the failed task.
|
|
11. Final artifacts are explicitly exported.
|
|
```
|
|
|
|
### 1.3 What must be true before public launch
|
|
|
|
The MVP is launchable only if these are true:
|
|
|
|
- A user can use the hosted coordinator without you granting them compute.
|
|
- A user can attach their own Linux node and run a build task on it.
|
|
- A user can attach a Windows node and run a Windows command task on it.
|
|
- Source code references environments by logical name, not by runner label or machine name.
|
|
- Local-first Linux builds do not upload the repo, tar the checkout, route file reads through the coordinator, or automatically upload build outputs.
|
|
- `flush()` is cheap and metadata-first.
|
|
- `sync()` is explicit and may move bytes.
|
|
- The debugger shows one process and many virtual threads.
|
|
- A Debug Epoch stops all Wasm tasks and controlled native command tasks for the virtual process.
|
|
- Task restart works from task entrypoint or last published VFS checkpoint.
|
|
- Community hosted users cannot run arbitrary containers or native commands on your infrastructure.
|
|
- Every public object has tenant ownership, quota accounting, and authorization checks.
|
|
|
|
### 1.4 What can be rough at launch
|
|
|
|
These can be rough, provided the rough edges are honest and isolated:
|
|
|
|
- Windows sandboxing can be a stubbed backend with a clear `windows-command-dev` capability, as long as hosted managed Windows compute is not offered to untrusted users.
|
|
- Nix can be included as a default plugin but not required for the flagship Containerfile demo.
|
|
- The operator panel can be scaffolded and visible as a preview, with the full immediate-mode UI in the second milestone.
|
|
- Debugger variable inspection can prioritize task args, Disasmer handles, top-level locals around Disasmer API calls, stack frames, and command status before supporting every possible Rust compiler local perfectly.
|
|
- NAT traversal can be coordinator-aided without relay. If direct node-to-node connectivity fails, scheduling and artifact flows should avoid requiring peer transfer rather than silently relaying bulk data through the coordinator.
|
|
|
|
---
|
|
|
|
## 2. Hard MVP decisions from the project constraints
|
|
|
|
### 2.1 Product scope
|
|
|
|
The MVP includes both:
|
|
|
|
- **open-source local-first runtime**, because credibility depends on not hiding the core runtime; and
|
|
- **hosted coordinator**, because the GitHub Actions replacement story needs an easy entry point, public rendezvous, auth, node registry, debug sessions, and a web surface.
|
|
|
|
The hosted coordinator is a control plane, not a general-purpose compute platform.
|
|
|
|
### 2.2 First target user
|
|
|
|
The first target user is:
|
|
|
|
> A developer or small team that currently uses GitHub Actions, Forgejo Actions, or similar CI, and wants something more programmable, local-first, debuggable, and self-hostable.
|
|
|
|
The first user does not need managed compute from Disasmer. They need:
|
|
|
|
- local and self-hosted nodes;
|
|
- a hosted coordinator for convenience;
|
|
- VS Code debugging;
|
|
- artifact metadata, ephemeral artifact handles, and explicit output export;
|
|
- a better mental model than YAML pipelines;
|
|
- and a credible path to replacing CI.
|
|
|
|
### 2.3 Host platforms
|
|
|
|
MVP node platforms:
|
|
|
|
```text
|
|
Linux required
|
|
Windows required
|
|
macOS not required for launch, but kept in the model
|
|
BSD-like not required for launch, but kept in the model
|
|
```
|
|
|
|
Linux should be the most complete backend. Windows should be complete enough to run a real attached worker and prove cross-platform task placement. macOS remains a design-supported later node type.
|
|
|
|
### 2.4 Environment support
|
|
|
|
MVP user-facing environment support:
|
|
|
|
```text
|
|
Containerfile / Dockerfile discovery: required
|
|
Nix plugin: included by default, but not launch-critical for all flows
|
|
Windows sandbox backend: interface + stub, not trusted managed compute
|
|
```
|
|
|
|
The MVP should discover:
|
|
|
|
```text
|
|
envs/<name>/Containerfile
|
|
envs/<name>/Dockerfile
|
|
```
|
|
|
|
It may also discover:
|
|
|
|
```text
|
|
envs/<name>/flake.nix
|
|
envs/<name>/flake.lock
|
|
```
|
|
|
|
But the flagship launch demo should not depend on Nix unless it is already stable in the implementation.
|
|
|
|
### 2.5 Execution support
|
|
|
|
The MVP must run both:
|
|
|
|
- Wasm task entrypoints, using Wasmtime on node runtimes; and
|
|
- native commands, through `cmd!(...)`, executed only by node runtimes inside declared environments.
|
|
|
|
The coordinator must not run native commands or containers. The hosted control loop may run zero-capability Wasm, but only with strict fuel, memory, log, state, and wall-clock limits.
|
|
|
|
### 2.6 Wasm model
|
|
|
|
Use the simplest Wasmtime model that is secure and functional:
|
|
|
|
```text
|
|
Core Wasm module + explicit host imports first.
|
|
Component Model support later or behind an internal abstraction.
|
|
```
|
|
|
|
Reasoning:
|
|
|
|
- Core Wasm embedding is enough for task entrypoints, host imports, memory access, and debug probes.
|
|
- The MVP does not need to expose a stable cross-language component ABI yet.
|
|
- The runtime should keep an internal `GuestAbi` trait so Component Model support can replace or augment this later.
|
|
|
|
### 2.7 Data model
|
|
|
|
MVP task boundaries allow only:
|
|
|
|
```text
|
|
small serializable values
|
|
immutable content-addressed objects
|
|
Disasmer runtime handles
|
|
```
|
|
|
|
Do not allow:
|
|
|
|
```text
|
|
raw pointers
|
|
borrowed references
|
|
local file handles
|
|
sockets
|
|
process handles
|
|
MutexGuards
|
|
unbounded byte vectors
|
|
native closures
|
|
live stack migration
|
|
```
|
|
|
|
Large data moves through:
|
|
|
|
```text
|
|
SourceSnapshot
|
|
Blob
|
|
Artifact
|
|
VirtualFile
|
|
VfsManifest
|
|
```
|
|
|
|
### 2.8 Checkpoint model
|
|
|
|
MVP checkpointing means:
|
|
|
|
```text
|
|
filesystem/artifact checkpoints for task restart
|
|
```
|
|
|
|
It does not mean:
|
|
|
|
```text
|
|
live Wasm stack checkpointing
|
|
native process checkpointing
|
|
socket checkpointing
|
|
transparent migration of running processes
|
|
hot-swapping code into existing stack frames
|
|
```
|
|
|
|
A restartable task must be recreated from:
|
|
|
|
```text
|
|
task name
|
|
new module hash
|
|
original serialized args
|
|
same environment handle
|
|
same source snapshot or selected dirty overlay
|
|
latest allowed VFS checkpoint
|
|
```
|
|
|
|
### 2.9 Debugging model
|
|
|
|
All-stop debugging is mandatory.
|
|
|
|
The debugger must present:
|
|
|
|
```text
|
|
one virtual process
|
|
many virtual threads/tasks
|
|
normal breakpoints in Disasmer Rust task code
|
|
normal thread list
|
|
stacks
|
|
selected locals / task args / handle inspection
|
|
stdout/stderr per task
|
|
artifacts and VFS state in optional views
|
|
restart selected task
|
|
```
|
|
|
|
The MVP should provide a Disasmer-owned DAP adapter. VS Code is the first supported client, but the adapter should speak ordinary DAP so other IDEs or tools can integrate later without Disasmer providing those integrations. LLDB/lldb-dap can be used internally later for richer DWARF support, but the virtual-process illusion should remain owned by Disasmer.
|
|
|
|
### 2.10 Transport
|
|
|
|
Use Quinn for MVP QUIC transport, behind a transport abstraction:
|
|
|
|
```text
|
|
node-to-node data/control links: QUIC over Quinn
|
|
node-to-coordinator persistent session: QUIC preferred, HTTPS/WebSocket allowed for auth/bootstrap
|
|
bulk transfer: node-to-node only, no coordinator bulk relay
|
|
```
|
|
|
|
Quinn is the MVP default because it is Rust-native and async-friendly, which keeps packaging and dogfooding simpler than an external C runtime dependency. Keep the `Transport` trait strict enough that quiche, TQUIC, or another implementation can replace it later if benchmarks, platform packaging, or protocol features justify a change.
|
|
|
|
The coordinator can help with rendezvous and endpoint candidate exchange. It should not relay bulk traffic. MVP can avoid cross-node bulk transfer if direct connectivity fails by scheduling tasks where the data already is or requiring explicit export to a receiver node or user-configured storage integration.
|
|
|
|
### 2.11 Storage
|
|
|
|
Use an abstract coordinator store with a strict split between persistent product state and ephemeral virtual-process state.
|
|
|
|
Persistent state should use Postgres in the hosted deployment and should be limited to things that must survive coordinator restart:
|
|
|
|
```text
|
|
users / tenants / memberships where applicable
|
|
projects
|
|
node identities and enrollment records
|
|
node public keys / credentials / revocation state
|
|
bundle registry and environment metadata worth reusing
|
|
source-provider configuration metadata
|
|
hosted/community policy configuration
|
|
billing/plan flags later
|
|
longer-lived audit events where needed
|
|
```
|
|
|
|
Ephemeral runtime state should live in coordinator memory for the MVP:
|
|
|
|
```text
|
|
active virtual processes
|
|
virtual threads and task placement
|
|
Debug Epochs
|
|
pending spawn queues
|
|
scheduler decisions
|
|
live node sessions and heartbeats
|
|
recent logs/events
|
|
VFS manifest cache for active processes
|
|
artifact metadata for active processes
|
|
quota counters for active processes, except hosted/community abuse ledgers that the private service wants to persist
|
|
```
|
|
|
|
If the coordinator restarts during the MVP, active virtual processes may be lost and should restart cleanly. That is acceptable. Do not overbuild durable process recovery before the runtime/debugging thesis is proven.
|
|
|
|
The store abstraction still matters because later the coordinator may need durable process recovery, Raft, sharding, or a different store. The MVP should not spend engineering effort on distributed consensus before the product is proven.
|
|
|
|
### 2.12 Auth
|
|
|
|
Support two auth modes from the start:
|
|
|
|
```text
|
|
hosted/community service auth: Authentik through OAuth2/OIDC, implemented in a private hosted module
|
|
CLI human auth: browser callback against the hosted service
|
|
CLI/agent auth: public-key identity with explicit enrollment/authorization
|
|
node attach: short-lived enrollment token -> long-lived node key/certificate
|
|
service-to-service: scoped service tokens where needed
|
|
open-source/private coordinator auth: simple local/team trust model with pluggable auth hooks, not the hosted OIDC stack by default
|
|
```
|
|
|
|
User OAuth tokens must not be passed to nodes. Nodes authenticate with node identity, and tasks receive only explicit, scoped, short-lived task tokens or secrets.
|
|
|
|
The CLI should be comfortable for both humans and agents. Browser login is fine for a developer at a workstation; agents should be able to use a generated keypair, an enrollment challenge, and a scoped project/node credential without opening a browser.
|
|
|
|
### 2.13 Community hosted abuse stance
|
|
|
|
The community hosted tier should prove the platform, not subsidize builds. The word “community” is intentional: it should be useful for serious users, while enterprise/team use cases should naturally need more than one active process, longer retention, stronger identity, or managed capacity.
|
|
|
|
Community hosted mode supports:
|
|
|
|
```text
|
|
sign in
|
|
create project (may be implicitly created, max 1)
|
|
create virtual process (may be implicitly created, max 1)
|
|
open coordinator UI
|
|
attach own node (max 4 nodes)
|
|
spawn tasks onto own node (max 8 tasks in flight)
|
|
debug from VS Code
|
|
see logs/artifact metadata for active processes (logs can't be bigger than 16KiB)
|
|
```
|
|
|
|
Community hosted mode does not support:
|
|
|
|
```text
|
|
hosted containers
|
|
hosted native commands
|
|
default hosted artifact storage
|
|
public artifact hosting
|
|
network access from hosted Wasm
|
|
long-running compute
|
|
relay bandwidth
|
|
```
|
|
|
|
### 2.14 Operator panel
|
|
|
|
The operator panel is second milestone, but the MVP should design toward it:
|
|
|
|
- runtime-state structures for panel state and events;
|
|
- protocol namespace for typed UI updates;
|
|
- control-plane UI slot for a panel;
|
|
- built-in widgets only;
|
|
- no custom HTML/JS;
|
|
- read-only behavior when the virtual process is stopped in a debugger.
|
|
- The panel has very tight size/state constraints for community mode
|
|
|
|
For public launch, a fixed coordinator dashboard is enough if it clearly hints at the immediate-mode panel API.
|
|
|
|
---
|
|
|
|
## 3. MVP architecture
|
|
|
|
### 3.1 High-level architecture
|
|
|
|
```text
|
|
VS Code
|
|
├─ Disasmer VS Code extension
|
|
└─ Disasmer DAP adapter
|
|
│
|
|
▼
|
|
Hosted or local coordinator
|
|
├─ Auth / tenants / projects
|
|
├─ node registry and rendezvous
|
|
├─ virtual process registry
|
|
├─ scheduler
|
|
├─ debug epoch coordinator
|
|
├─ VFS/artifact metadata for active processes
|
|
├─ hosted/community policy hooks
|
|
├─ operator panel state/events scaffold
|
|
├─ in-memory active runtime state
|
|
└─ Postgres persistent project/identity store
|
|
│
|
|
├────────────── QUIC / control session ───────────────┐
|
|
▼ ▼
|
|
Linux node runtime Windows node runtime
|
|
├─ Wasmtime task runner ├─ Wasmtime task runner
|
|
├─ command runner ├─ command runner
|
|
├─ Podman environment backend ├─ Windows command backend
|
|
├─ Nix plugin backend ├─ Windows sandbox stub backend
|
|
├─ VFS overlay store ├─ VFS overlay store
|
|
├─ ephemeral artifact/blob cache ├─ ephemeral artifact/blob cache
|
|
├─ debug agent ├─ debug agent
|
|
├─ node cache ├─ node cache
|
|
└─ Quinn peer transport └─ Quinn peer transport
|
|
```
|
|
|
|
### 3.2 Control plane vs data plane
|
|
|
|
The coordinator owns:
|
|
|
|
```text
|
|
identity
|
|
membership
|
|
node capabilities
|
|
bundle metadata
|
|
virtual process metadata
|
|
task placement decisions
|
|
debug epochs
|
|
VFS manifest metadata for active processes
|
|
artifact metadata for active processes
|
|
quota accounting
|
|
operator panel metadata
|
|
rendezvous
|
|
```
|
|
|
|
All of these need constraints so that the community service can't be abused.
|
|
|
|
The nodes own:
|
|
|
|
```text
|
|
Wasm execution
|
|
native command execution
|
|
environment materialization
|
|
source checkout access
|
|
build caches
|
|
VFS bytes
|
|
artifact bytes
|
|
large blob transfer
|
|
debug stop/resume implementation
|
|
```
|
|
|
|
User-attached nodes are the users' responsibility and can run broad local capabilities. The hosted coordinator must still validate what capabilities are accepted for a hosted/community process, but the open-source/private coordinator should not block normal local-node capabilities by default.
|
|
|
|
The coordinator must not become:
|
|
|
|
```text
|
|
a network filesystem
|
|
a blob proxy
|
|
a remote compiler filesystem
|
|
a hidden build runner
|
|
a relay by default
|
|
```
|
|
|
|
### 3.3 Recommended repository layout
|
|
|
|
```text
|
|
disasmer/
|
|
crates/
|
|
disasmer-sdk/ # guest-facing Rust API
|
|
disasmer-macros/ # #[disasmer::main], #[disasmer::task], env!()
|
|
disasmer-abi/ # guest/host ABI structs and serialization
|
|
disasmer-cli/ # user CLI
|
|
disasmer-coordinator/ # hosted/local coordinator service
|
|
disasmer-node/ # node runtime daemon
|
|
disasmer-dap/ # custom Debug Adapter Protocol server
|
|
disasmer-vscode-protocol/ # extension <-> CLI/DAP shared types
|
|
disasmer-transport/ # Quinn-backed QUIC wrapper and protocol framing
|
|
disasmer-store/ # store traits + Postgres persistent-store impl
|
|
disasmer-runtime-state/ # in-memory active process/thread/debug state
|
|
disasmer-vfs/ # VFS manifests, overlays, blob/chunk references
|
|
disasmer-env/ # env discovery, normalization, caching
|
|
disasmer-source/ # source-provider traits and manifests
|
|
disasmer-git/ # default optional Git source provider module
|
|
disasmer-debug-model/ # breakpoints, epochs, stack/locals schema
|
|
disasmer-ui-model/ # immediate-mode UI schema scaffold
|
|
disasmer-auth/ # node enrollment, pubkey identities, pluggable auth hooks
|
|
disasmer-protocol/ # coordinator/node/DAP wire messages
|
|
private/
|
|
hosted-oidc/ # Authentik/OIDC integration for the public service
|
|
hosted-policy/ # community quotas, abuse controls, zero-cap hosted-loop rules
|
|
hosted-admin/ # service admin tools, kill switches, internal dashboards
|
|
vscode/
|
|
extension/ # TypeScript extension
|
|
examples/
|
|
launch-build/ # flagship Linux + Windows build demo
|
|
local-one-node/ # minimal spawn/join example
|
|
operator-panel-preview/ # second-milestone preview
|
|
docs/
|
|
MVP.md
|
|
architecture.md
|
|
protocol.md
|
|
security.md
|
|
debugging.md
|
|
```
|
|
|
|
The private folders are normal repository folders during development, but must be easy to exclude from the public source release. The public crates should expose policy/auth interfaces; the private modules implement hosted-service-specific OIDC, community quota enforcement, hosted zero-capability Wasm limits, abuse tooling, and admin affordances.
|
|
|
|
### 3.4 Process model
|
|
|
|
MVP process types:
|
|
|
|
```text
|
|
disasmer coordinator
|
|
Hosted or local control-plane service.
|
|
|
|
disasmer node
|
|
Long-running node runtime attached to a coordinator.
|
|
|
|
disasmer dap
|
|
Debug Adapter Protocol process launched by VS Code.
|
|
|
|
disasmer cli
|
|
User/agent CLI for login, init, build, run, node attach, artifact export.
|
|
```
|
|
|
|
### 3.5 Primary user commands
|
|
|
|
```bash
|
|
# Authenticate a human against the hosted coordinator.
|
|
disasmer login https://disasmer.example.com
|
|
|
|
# Create or enroll a non-browser CLI identity for an agent.
|
|
disasmer auth keygen --name forgejo-worker
|
|
|
|
disasmer auth enroll-key --coordinator https://disasmer.example.com --project my-project --public-key ~/.config/disasmer/keys/forgejo-worker.pub
|
|
|
|
# Initialize project metadata; optional because defaults should work.
|
|
disasmer init
|
|
|
|
# Show discovered environments, source providers, and snapshot policy.
|
|
disasmer inspect
|
|
|
|
# Build the virtual binary/bundle.
|
|
disasmer build
|
|
|
|
# Attach a node. OS, architecture, container backend, Git availability, and most
|
|
# capabilities are auto-detected; --cap remains available for explicit overrides.
|
|
disasmer node attach --coordinator https://disasmer.example.com --token dis_node_enroll_...
|
|
|
|
# Override or add an auto-detected capability only when needed.
|
|
disasmer node attach --coordinator https://disasmer.example.com --token dis_node_enroll_... --cap env.windows-command-dev=true
|
|
|
|
# Run without VS Code. Hosted coordinator is the default when logged in;
|
|
# project defaults to the current directory. Entry point is optional.
|
|
disasmer run
|
|
|
|
disasmer run build-linux
|
|
|
|
disasmer run --project ../other-repo package-release
|
|
|
|
disasmer run --local smoke-test
|
|
|
|
# Export final artifacts by asking an attached/local receiver node to pull them.
|
|
# There is no default coordinator artifact store.
|
|
disasmer artifact export --process <id> --to ./dist
|
|
```
|
|
|
|
`disasmer run` should accept an optional entry point. This is a good idea because one Disasmer project may contain several useful virtual-process entrypoints: `build`, `test`, `package`, `release`, `watch`, `operator`, or project-specific workflows. The default remains the `#[disasmer::main]` entrypoint when no entry is supplied.
|
|
|
|
VS Code should hide most of this behind:
|
|
|
|
```text
|
|
Disasmer: Login
|
|
Disasmer: Attach Local Node
|
|
Disasmer: Build Bundle
|
|
Disasmer: Run
|
|
Disasmer: Debug
|
|
Disasmer: Restart Failed Task
|
|
Disasmer: Export Artifacts
|
|
```
|
|
|
|
---
|
|
|
|
## 4. The flagship launch example
|
|
|
|
### 4.1 Project layout
|
|
|
|
```text
|
|
launch-build-demo/
|
|
Cargo.toml
|
|
Cargo.lock
|
|
src/
|
|
main.rs
|
|
envs/
|
|
linux/
|
|
Containerfile
|
|
windows/
|
|
Containerfile
|
|
inputs/
|
|
build-config.json
|
|
disasmer.toml # optional; only used for launch polish/overrides
|
|
```
|
|
|
|
### 4.2 Example `disasmer.toml`
|
|
|
|
The file should be optional, but the demo can include it to make platform constraints obvious.
|
|
|
|
```toml
|
|
[project]
|
|
name = "launch-build-demo"
|
|
|
|
[envs.linux]
|
|
path = "envs/linux/Containerfile"
|
|
kind = "container"
|
|
requires = { os = "linux", arch = "x86_64" }
|
|
|
|
[envs.windows]
|
|
path = "envs/windows/Containerfile"
|
|
kind = "container"
|
|
requires = { os = "windows", arch = "x86_64" }
|
|
backend_hint = "windows-command-dev"
|
|
|
|
[vfs]
|
|
include = [
|
|
"src/**",
|
|
"Cargo.toml",
|
|
"Cargo.lock",
|
|
"inputs/**"
|
|
]
|
|
exclude = [
|
|
"target/**",
|
|
".git/**"
|
|
]
|
|
|
|
[git]
|
|
enabled = true
|
|
dirty_overlay = true
|
|
submodules = true
|
|
```
|
|
|
|
### 4.3 Example Rust build program
|
|
|
|
```rust
|
|
use disasmer::prelude::*;
|
|
|
|
#[derive(Serialize, Deserialize, Clone, Debug, DisasmerArg)]
|
|
enum Target {
|
|
LinuxX64,
|
|
WindowsX64,
|
|
}
|
|
|
|
impl Target {
|
|
fn env(&self) -> Env {
|
|
match self {
|
|
Target::LinuxX64 => env!("linux"),
|
|
Target::WindowsX64 => env!("windows"),
|
|
}
|
|
}
|
|
|
|
fn name(&self) -> &'static str {
|
|
match self {
|
|
Target::LinuxX64 => "linux-x64",
|
|
Target::WindowsX64 => "windows-x64",
|
|
}
|
|
}
|
|
|
|
fn command(&self) -> CommandSpec {
|
|
match self {
|
|
Target::LinuxX64 => cmd!("cargo", "build", "--release"),
|
|
Target::WindowsX64 => cmd!("cargo", "build", "--release"),
|
|
}
|
|
}
|
|
}
|
|
|
|
#[disasmer::main]
|
|
async fn main() -> Result<()> {
|
|
// The hosted coordinator control loop is intentionally capless. Source access
|
|
// happens on a user-attached node through the default Git source-provider
|
|
// module, not inside the coordinator process.
|
|
let src = spawn::task(prepare_source)
|
|
.placement(placement::any_node().with_cap("source.git"))
|
|
.queue_until_node_available()
|
|
.start()
|
|
.await?
|
|
.join()
|
|
.await?;
|
|
|
|
let completed = DAtomicU32::new(0).await?;
|
|
|
|
let linux = spawn::task(compile)
|
|
.arg(BuildArgs::new(Target::LinuxX64, src.clone(), completed.clone()))
|
|
.env(env!("linux"))
|
|
.start()
|
|
.await?;
|
|
|
|
let windows = spawn::task(compile)
|
|
.arg(BuildArgs::new(Target::WindowsX64, src.clone(), completed.clone()))
|
|
.env(env!("windows"))
|
|
.start()
|
|
.await?;
|
|
|
|
let linux_artifact = linux.join().await?;
|
|
let windows_artifact = windows.join().await?;
|
|
|
|
let dist = package(vec![linux_artifact, windows_artifact]).await?;
|
|
|
|
assert_eq!(completed.load(Ordering::SeqCst).await?, 2);
|
|
|
|
// Export is explicit code. The coordinator does not provide a default
|
|
// artifact store; this asks an attached receiver/local node to pull bytes.
|
|
artifact::export_to_node(dist, receiver::local("dist")).await?;
|
|
Ok(())
|
|
}
|
|
|
|
#[disasmer::task]
|
|
async fn prepare_source() -> Result<SourceSnapshot> {
|
|
source::git::current_repo().snapshot().await
|
|
}
|
|
|
|
#[derive(Serialize, Deserialize, Clone, Debug, DisasmerArg)]
|
|
struct BuildArgs {
|
|
target: Target,
|
|
src: SourceSnapshot,
|
|
completed: DAtomicU32,
|
|
}
|
|
|
|
impl BuildArgs {
|
|
fn new(target: Target, src: SourceSnapshot, completed: DAtomicU32) -> Self {
|
|
Self { target, src, completed }
|
|
}
|
|
}
|
|
|
|
#[disasmer::task]
|
|
async fn compile(args: BuildArgs) -> Result<Artifact> {
|
|
fs::mount("/src", args.src).await?;
|
|
|
|
args.target.command()
|
|
.cwd("/src")
|
|
.env("DISASMER_TARGET", args.target.name())
|
|
.run()
|
|
.await?;
|
|
|
|
let output_path = match args.target {
|
|
Target::LinuxX64 => "/src/target/release/app",
|
|
Target::WindowsX64 => "/src/target/release/app.exe",
|
|
};
|
|
|
|
let artifact_path = format!("/vfs/artifacts/{}/{}", args.target.name(), basename(output_path));
|
|
fs::copy(output_path, &artifact_path).await?;
|
|
|
|
fs::flush().await?;
|
|
args.completed.fetch_add(1, Ordering::SeqCst).await?;
|
|
|
|
Artifact::from_path(artifact_path).await
|
|
}
|
|
|
|
#[disasmer::task]
|
|
async fn package(outputs: Vec<Artifact>) -> Result<Artifact> {
|
|
fs::create_dir_all("/vfs/artifacts/dist").await?;
|
|
|
|
for output in outputs {
|
|
fs::copy(output.path(), "/vfs/artifacts/dist/").await?;
|
|
}
|
|
|
|
fs::flush().await?;
|
|
Artifact::from_path("/vfs/artifacts/dist").await
|
|
}
|
|
```
|
|
|
|
### 4.4 What the demo proves
|
|
|
|
The demo proves:
|
|
|
|
- The build graph is source code, not YAML.
|
|
- Environment resources live in the repo.
|
|
- Source code references `env!("linux")` and `env!("windows")`, not runner labels.
|
|
- The coordinator schedules by logical environment and node capability.
|
|
- Linux and Windows tasks can run in the same virtual process.
|
|
- `cmd!(...)` runs on nodes, not on the hosted coordinator.
|
|
- Outputs become artifact handles through the VFS.
|
|
- `flush()` publishes artifacts for downstream tasks.
|
|
- Final artifact export is explicit code and does not imply coordinator artifact storage.
|
|
- VS Code sees one virtual process and multiple virtual threads.
|
|
- A Debug Epoch stops the whole virtual process.
|
|
- Restarting a failed task after editing source is part of the normal loop.
|
|
|
|
---
|
|
|
|
## 5. Rust SDK and task API
|
|
|
|
### 5.1 MVP API surface
|
|
|
|
The MVP Rust SDK should include:
|
|
|
|
```rust
|
|
#[disasmer::main]
|
|
#[disasmer::task]
|
|
env!("name")
|
|
spawn::task(task_fn)
|
|
TaskHandle<T>
|
|
Artifact
|
|
Blob
|
|
SourceSnapshot
|
|
VirtualFile
|
|
DAtomicU32
|
|
DAtomicU64
|
|
DChannel<T>
|
|
fs::mount()
|
|
fs::copy()
|
|
fs::flush()
|
|
fs::sync() / artifact::export_to_node()
|
|
cmd!(...)
|
|
source::git::current_repo().snapshot()
|
|
ui::* scaffold only
|
|
```
|
|
|
|
Do not include every distributed data structure in the first API. The launch demo needs small data, atomics/counters, task handles, source snapshots, artifacts, and VFS operations more than it needs a complete distributed memory library.
|
|
|
|
### 5.2 Task registration
|
|
|
|
`#[disasmer::task]` should generate:
|
|
|
|
- a stable task symbol;
|
|
- a task ABI descriptor;
|
|
- argument/result serialization metadata;
|
|
- debug metadata for task entry, source spans, and selected local/argument values;
|
|
- capability declarations inferred from APIs used where possible;
|
|
- task restart compatibility hash;
|
|
- a registration entry in the bundle manifest.
|
|
|
|
Example generated manifest entry:
|
|
|
|
```json
|
|
{
|
|
"task_id": "compile@sha256:...",
|
|
"symbol": "compile",
|
|
"source": "src/main.rs",
|
|
"arg_type": "BuildArgs@sha256:...",
|
|
"result_type": "Artifact@sha256:...",
|
|
"requires": ["vfs", "cmd", "artifact.publish"],
|
|
"debug": {
|
|
"spans": "debug/compile.spans.json",
|
|
"locals_schema": "debug/compile.locals.json"
|
|
}
|
|
}
|
|
```
|
|
|
|
### 5.3 Argument rules
|
|
|
|
Task arguments and results must implement `DisasmerArg`.
|
|
|
|
Allowed:
|
|
|
|
```text
|
|
u32, u64, bool, String
|
|
small structs/enums
|
|
SourceSnapshot
|
|
Artifact
|
|
Blob
|
|
Env
|
|
DAtomicU32 / DAtomicU64
|
|
DChannel<T>
|
|
VirtualFile
|
|
small Vec<T> with configured size limit
|
|
```
|
|
|
|
Rejected:
|
|
|
|
```text
|
|
&T / &mut T
|
|
raw pointers
|
|
Rc<T>
|
|
RefCell<T>
|
|
std::fs::File
|
|
TcpStream
|
|
MutexGuard
|
|
native thread/process handles
|
|
unbounded Vec<u8>
|
|
closures with implicit captures
|
|
```
|
|
|
|
The compiler error should explain the distributed boundary. For example:
|
|
|
|
```text
|
|
BuildArgs cannot be passed to a Disasmer virtual thread because it contains `&Path`.
|
|
Use String, SourceSnapshot, VirtualFile, Blob, or Artifact instead.
|
|
```
|
|
|
|
### 5.4 Size limits
|
|
|
|
Set a default task argument size limit such as:
|
|
|
|
```text
|
|
soft warning: 256 KiB
|
|
hard default: 4 MiB
|
|
project override: allowed but noisy
|
|
```
|
|
|
|
The hosted/community coordinator control loop should use much tighter limits, for example a 4 KiB warning and 16 KiB hard limit. User-attached nodes can keep the normal node defaults unless project policy overrides them.
|
|
A user accidentally passing an 82 MB `Vec<u8>` should see:
|
|
|
|
```text
|
|
Task argument is 82 MB and would be copied to the spawned virtual thread.
|
|
Use Blob::from_file, SourceSnapshot, Artifact, or VirtualFile instead.
|
|
```
|
|
|
|
### 5.5 Host imports
|
|
|
|
The MVP guest ABI should be explicit and small.
|
|
|
|
Core imports:
|
|
|
|
```text
|
|
disasmer.spawn.start
|
|
disasmer.spawn.join
|
|
disasmer.task.yield_safepoint
|
|
disasmer.task.current
|
|
disasmer.fs.mount
|
|
disasmer.fs.copy
|
|
disasmer.fs.flush
|
|
disasmer.fs.sync
|
|
disasmer.artifact.from_path
|
|
disasmer.artifact.open
|
|
disasmer.blob.open
|
|
disasmer.cmd.start
|
|
disasmer.cmd.poll
|
|
disasmer.cmd.kill
|
|
disasmer.atomic.new_u32
|
|
disasmer.atomic.fetch_add_u32
|
|
disasmer.atomic.load_u32
|
|
disasmer.debug.probe
|
|
disasmer.log.write
|
|
disasmer.ui.emit_snapshot # scaffold only
|
|
```
|
|
|
|
### 5.6 Native command API
|
|
|
|
`cmd!(...)` should build a `CommandSpec` rather than immediately running a process.
|
|
|
|
```rust
|
|
cmd!("cargo", "build", "--release")
|
|
.cwd("/src")
|
|
.env("RUSTFLAGS", "...")
|
|
.timeout(Duration::from_secs(600))
|
|
.run()
|
|
.await?;
|
|
```
|
|
|
|
The host checks:
|
|
|
|
- current virtual thread has `cmd.run` capability;
|
|
- selected environment supports native commands;
|
|
- the task is running on a node, not the hosted coordinator;
|
|
- process limits are configured;
|
|
- filesystem mounts are inside the VFS/environment policy;
|
|
- network policy is explicit;
|
|
- logs are capped and streamed with backpressure.
|
|
|
|
The hosted coordinator control plane never has this capability. An open-source/private coordinator may run with an embedded local node for trusted deployments; that node can have normal local command capabilities by default. Keep this as a compact policy/runtime module boundary rather than a separate node implementation, so dogfooding still uses the same node runtime code.
|
|
|
|
### 5.7 Compile-time environment references
|
|
|
|
`env!("linux")` should validate against generated environment metadata.
|
|
|
|
Implementation:
|
|
|
|
1. `disasmer build` scans `envs/**` and optional `disasmer.toml`.
|
|
2. It writes generated metadata to `target/disasmer/generated/envs.rs` and a JSON manifest.
|
|
3. The macro consults generated metadata during build.
|
|
4. VS Code reads the same metadata for autocomplete and diagnostics.
|
|
|
|
Missing env error:
|
|
|
|
```text
|
|
env!("windows") does not exist.
|
|
Discovered environments: linux.
|
|
Create envs/windows/Containerfile or add [envs.windows] to disasmer.toml.
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Bundle and build pipeline
|
|
|
|
### 6.1 Bundle concept
|
|
|
|
A Disasmer bundle is a content-addressed virtual binary.
|
|
|
|
It includes:
|
|
|
|
```text
|
|
Wasm module(s)
|
|
Disasmer manifest
|
|
registered tasks
|
|
debug metadata
|
|
environment definitions
|
|
selected environment context files
|
|
source snapshot metadata
|
|
selected VFS seed files
|
|
capability declarations
|
|
content hashes
|
|
```
|
|
|
|
It should not embed full container images.
|
|
|
|
### 6.2 Bundle layout
|
|
|
|
```text
|
|
app.dis/
|
|
manifest/disasmer.json
|
|
modules/main.wasm
|
|
debug/main.dwarf
|
|
debug/disasmer-debug.json
|
|
envs/linux/Containerfile
|
|
envs/windows/Containerfile
|
|
envs/manifest.json
|
|
source/source-snapshot.json
|
|
source/dirty-overlay-manifest.json
|
|
inputs/build-config.json
|
|
vfs/seed-manifest.json
|
|
capabilities.json
|
|
```
|
|
|
|
The on-disk `.dis` can initially be a directory during development and later become a zip/tar-like archive with content-addressed chunks.
|
|
|
|
### 6.3 Build command pipeline
|
|
|
|
```text
|
|
disasmer build
|
|
1. discover project root
|
|
2. load disasmer.toml if present
|
|
3. discover envs/**/Containerfile and envs/**/Dockerfile
|
|
4. discover Nix flakes if plugin enabled
|
|
5. resolve source snapshot through enabled source-provider module, Git by default
|
|
6. compute dirty overlay manifest
|
|
7. generate env metadata for env!()
|
|
8. compile Rust guest to Wasm
|
|
9. run macro/task registration collection
|
|
10. collect debug metadata
|
|
11. build VFS seed manifest
|
|
12. compute bundle digest
|
|
13. write app.dis
|
|
```
|
|
|
|
### 6.4 Digest strategy
|
|
|
|
Bundle digest should include:
|
|
|
|
```text
|
|
Wasm bytes
|
|
task ABI descriptors
|
|
environment recipe files
|
|
environment context file hashes
|
|
source snapshot provider identity and revision, such as a Git commit id
|
|
dirty overlay file hashes
|
|
selected input file hashes
|
|
capability declarations
|
|
debug metadata hash
|
|
SDK ABI version
|
|
```
|
|
|
|
It should not include:
|
|
|
|
```text
|
|
node-local cache paths
|
|
absolute host checkout path
|
|
local build timestamps
|
|
unselected .git internals
|
|
unselected target directory contents
|
|
```
|
|
|
|
### 6.5 Compatibility hashes
|
|
|
|
Task restart requires compatibility checks. Generate hashes for:
|
|
|
|
```text
|
|
task public signature
|
|
argument schema
|
|
result schema
|
|
capability declaration
|
|
environment reference set
|
|
VFS mount contract
|
|
Disasmer ABI version
|
|
```
|
|
|
|
If only implementation changed and public schema is compatible, allow selected task restart. If schema changed, restart the whole virtual process.
|
|
|
|
---
|
|
|
|
## 7. Node runtime
|
|
|
|
### 7.1 Node responsibilities
|
|
|
|
The node runtime is where real work happens.
|
|
|
|
It must support:
|
|
|
|
```text
|
|
registration with coordinator
|
|
capability reporting
|
|
heartbeat and health reporting
|
|
bundle fetch or local bundle lookup
|
|
Wasmtime task instantiation
|
|
native command execution
|
|
environment materialization/caching
|
|
VFS overlay storage
|
|
ephemeral artifact/blob cache
|
|
source snapshot providers
|
|
debug epoch handling
|
|
logs/events streaming
|
|
quota enforcement
|
|
Quinn peer links
|
|
```
|
|
|
|
### 7.2 Node state directories
|
|
|
|
Linux default:
|
|
|
|
```text
|
|
~/.local/share/disasmer/node/
|
|
config.toml
|
|
identity/
|
|
bundles/
|
|
env-cache/
|
|
vfs/
|
|
blobs/
|
|
artifacts/
|
|
logs/
|
|
tmp/
|
|
```
|
|
|
|
Windows default:
|
|
|
|
```text
|
|
%LOCALAPPDATA%\Disasmer\node\
|
|
config.toml
|
|
identity\
|
|
bundles\
|
|
env-cache\
|
|
vfs\
|
|
blobs\
|
|
artifacts\
|
|
logs\
|
|
tmp\
|
|
```
|
|
|
|
### 7.3 Node capability model
|
|
|
|
Node capabilities should be explicit and signed by node identity plus coordinator policy.
|
|
|
|
Example:
|
|
|
|
```json
|
|
{
|
|
"node_id": "node_123",
|
|
"tenant_id": "tenant_abc",
|
|
"os": "linux",
|
|
"arch": "x86_64",
|
|
"capabilities": {
|
|
"wasm": true,
|
|
"cmd": true,
|
|
"container.podman.rootless": true,
|
|
"nix": true,
|
|
"git.source_provider": true,
|
|
"debug.suspend_process_tree": true
|
|
},
|
|
"limits": {
|
|
"max_tasks": 4,
|
|
"max_log_bytes_per_task": 10485760,
|
|
"max_arg_bytes": 4194304
|
|
}
|
|
}
|
|
```
|
|
|
|
Windows dev backend example:
|
|
|
|
```json
|
|
{
|
|
"node_id": "node_win_123",
|
|
"tenant_id": "tenant_abc",
|
|
"os": "windows",
|
|
"arch": "x86_64",
|
|
"capabilities": {
|
|
"wasm": true,
|
|
"cmd": true,
|
|
"windows.command_dev": true,
|
|
"windows.sandbox_stub": true,
|
|
"debug.job_object_suspend": true
|
|
},
|
|
"security_note": "not a managed-untrusted sandbox backend"
|
|
}
|
|
```
|
|
|
|
### 7.4 Wasmtime task runner
|
|
|
|
Each running virtual thread/task maps to:
|
|
|
|
```text
|
|
Wasmtime Engine
|
|
compiled module cache
|
|
Store<ThreadHostState>
|
|
Linker with Disasmer imports
|
|
task entrypoint dispatcher
|
|
fuel/epoch/resource limits
|
|
debug probe table
|
|
VFS overlay handle
|
|
handle table
|
|
```
|
|
|
|
`ThreadHostState` should contain:
|
|
|
|
```rust
|
|
struct ThreadHostState {
|
|
tenant_id: TenantId,
|
|
project_id: ProjectId,
|
|
virtual_process_id: VirtualProcessId,
|
|
virtual_thread_id: VirtualThreadId,
|
|
task_id: TaskId,
|
|
bundle_digest: Digest,
|
|
debug_epoch: Option<DebugEpochId>,
|
|
vfs_overlay: VfsOverlayHandle,
|
|
handles: HandleTable,
|
|
limits: ResourceLimits,
|
|
coordinator: CoordinatorClient,
|
|
node_store: NodeStore,
|
|
command_runner: Arc<dyn CommandRunner>,
|
|
}
|
|
```
|
|
|
|
### 7.5 Command runner
|
|
|
|
The command runner must abstract platform differences.
|
|
|
|
```rust
|
|
trait CommandRunner {
|
|
async fn start(&self, spec: CommandSpec, ctx: CommandContext) -> Result<CommandHandle>;
|
|
async fn poll(&self, handle: CommandHandle) -> Result<CommandStatus>;
|
|
async fn suspend_for_debug(&self, handle: CommandHandle, epoch: DebugEpochId) -> Result<()>;
|
|
async fn resume_from_debug(&self, handle: CommandHandle, epoch: DebugEpochId) -> Result<()>;
|
|
async fn terminate(&self, handle: CommandHandle) -> Result<()>;
|
|
}
|
|
```
|
|
|
|
Linux implementations:
|
|
|
|
```text
|
|
PodmanRootlessCommandRunner
|
|
LocalProcessGroupCommandRunner for development
|
|
```
|
|
|
|
Windows implementations:
|
|
|
|
```text
|
|
WindowsCommandDevRunner
|
|
WindowsSandboxRunnerStub
|
|
```
|
|
|
|
Do not offer `WindowsCommandDevRunner` as secure managed compute for untrusted public workloads.
|
|
|
|
### 7.6 Linux Podman backend
|
|
|
|
Linux MVP should use rootless Podman.
|
|
|
|
Responsibilities:
|
|
|
|
```text
|
|
build image from Containerfile/Dockerfile
|
|
cache image by environment digest
|
|
mount /src, /work, /cache, /out or VFS paths
|
|
run command with resource limits where practical
|
|
capture stdout/stderr
|
|
pause/unpause or suspend process tree for debug epoch
|
|
publish outputs into VFS overlay
|
|
```
|
|
|
|
Default mounts:
|
|
|
|
```text
|
|
/src source snapshot or bind mount
|
|
/work task-local writable overlay
|
|
/cache node-local dependency cache keyed by env/project
|
|
/out artifact staging path
|
|
```
|
|
|
|
### 7.7 Windows backend
|
|
|
|
For the public-launch MVP, Windows should support an attached node that can run build commands in a controlled workspace.
|
|
|
|
Minimum Windows backend:
|
|
|
|
```text
|
|
create per-task temp workspace
|
|
materialize source snapshot into workspace or bind local checkout when safe
|
|
set environment variables from normalized env
|
|
run command as non-admin service user where possible
|
|
capture stdout/stderr
|
|
track process tree with Job Object-like lifecycle management
|
|
stage artifacts into VFS overlay
|
|
support best-effort debug suspend/resume
|
|
```
|
|
|
|
Windows sandboxing should be behind an interface but not claimed as production isolation until implemented and tested.
|
|
|
|
Potential future backend options:
|
|
|
|
```text
|
|
Windows Sandbox CLI backend
|
|
AppContainer / Less-Privileged AppContainer backend
|
|
Hyper-V isolated runner backend
|
|
Windows container backend, if project needs it
|
|
```
|
|
|
|
### 7.8 Logs
|
|
|
|
Logs are both a UX feature and an abuse vector.
|
|
|
|
Node should:
|
|
|
|
- stream recent log chunks to coordinator;
|
|
- keep larger logs node-local unless `sync()` or retention policy requires upload;
|
|
- redact configured secrets before sending;
|
|
- enforce per-task and per-process log byte caps;
|
|
- include task id, virtual thread id, sequence number, timestamp, and stream name.
|
|
|
|
Log event:
|
|
|
|
```json
|
|
{
|
|
"tenant_id": "tenant_abc",
|
|
"process_id": "vp_123",
|
|
"thread_id": "vt_456",
|
|
"task_id": "task_789",
|
|
"stream": "stderr",
|
|
"seq": 42,
|
|
"bytes": "...",
|
|
"truncated": false
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 8. Coordinator
|
|
|
|
### 8.1 Coordinator responsibilities
|
|
|
|
The coordinator should provide:
|
|
|
|
```text
|
|
auth/session handling; hosted OIDC through private module
|
|
project registry
|
|
node enrollment and identity
|
|
node heartbeats
|
|
node capability registry
|
|
virtual process lifecycle
|
|
task placement
|
|
bundle registry
|
|
debug session registry
|
|
Debug Epoch coordination
|
|
VFS manifest metadata for active processes
|
|
artifact metadata for active processes
|
|
hosted/community policy hooks and optional private quota ledger
|
|
operator panel state/events scaffold
|
|
rendezvous for node-to-node QUIC
|
|
web dashboard
|
|
CLI API
|
|
VS Code/DAP API
|
|
```
|
|
|
|
It should not run arbitrary user commands.
|
|
|
|
### 8.2 Coordinator state split
|
|
|
|
The coordinator needs two state layers in the MVP.
|
|
|
|
Persistent Postgres state:
|
|
|
|
```text
|
|
users / tenants / memberships if hosted module is enabled
|
|
projects
|
|
node enrollment records
|
|
node public keys / credentials / revocations
|
|
project-scoped source-provider configuration
|
|
bundle registry entries worth reusing
|
|
persistent environment metadata/cache hints
|
|
hosted/community plan or policy records in the private module
|
|
longer-lived audit records where needed
|
|
```
|
|
|
|
In-memory runtime state:
|
|
|
|
```text
|
|
active virtual processes
|
|
virtual threads
|
|
pending task queues
|
|
live node sessions and heartbeats
|
|
scheduler placements
|
|
Debug Epochs
|
|
recent logs/events
|
|
VFS manifest metadata for active processes
|
|
artifact metadata and locations for active processes
|
|
operator panel snapshots/events for active processes
|
|
```
|
|
|
|
For the MVP, if the coordinator restarts, active virtual processes restart or fail cleanly. Do not persist every process/thread/log/VFS row merely to survive coordinator restarts. Persistence is for product identity and project continuity, not durable runtime recovery.
|
|
|
|
### 8.3 Store interfaces and indicative persistent tables
|
|
|
|
Define a persistent store trait for things that must survive restart:
|
|
|
|
```rust
|
|
trait CoordinatorPersistentStore {
|
|
async fn create_tenant(...);
|
|
async fn create_project(...);
|
|
async fn register_node_identity(...);
|
|
async fn revoke_node_credential(...);
|
|
async fn create_bundle(...);
|
|
async fn save_environment_metadata(...);
|
|
async fn save_source_provider_config(...);
|
|
async fn authorize_persistent_object(...);
|
|
}
|
|
```
|
|
|
|
Define a separate runtime-state trait that is in-memory for the MVP:
|
|
|
|
```rust
|
|
trait CoordinatorRuntimeState {
|
|
async fn create_virtual_process(...);
|
|
async fn create_virtual_thread(...);
|
|
async fn update_thread_state(...);
|
|
async fn create_debug_epoch(...);
|
|
async fn publish_vfs_manifest(...);
|
|
async fn publish_artifact_metadata(...);
|
|
async fn append_recent_log_event(...);
|
|
async fn reserve_runtime_quota(...);
|
|
}
|
|
```
|
|
|
|
Indicative persistent Postgres tables:
|
|
|
|
```sql
|
|
users(
|
|
id,
|
|
authentik_subject nullable,
|
|
email nullable,
|
|
created_at,
|
|
suspended_at
|
|
)
|
|
|
|
tenants(
|
|
id,
|
|
slug,
|
|
plan,
|
|
created_at
|
|
)
|
|
|
|
tenant_members(
|
|
tenant_id,
|
|
user_id,
|
|
role
|
|
)
|
|
|
|
projects(
|
|
id,
|
|
tenant_id,
|
|
name,
|
|
visibility,
|
|
created_at
|
|
)
|
|
|
|
nodes(
|
|
id,
|
|
tenant_id,
|
|
project_id nullable,
|
|
display_name,
|
|
os,
|
|
arch,
|
|
trust_label,
|
|
created_at,
|
|
revoked_at
|
|
)
|
|
|
|
node_capabilities_last_seen(
|
|
node_id,
|
|
capability,
|
|
value,
|
|
reported_at
|
|
)
|
|
|
|
node_credentials(
|
|
node_id,
|
|
public_key,
|
|
cert_fingerprint nullable,
|
|
expires_at nullable,
|
|
revoked_at
|
|
)
|
|
|
|
bundles(
|
|
id,
|
|
tenant_id,
|
|
project_id,
|
|
digest,
|
|
manifest_json,
|
|
created_at
|
|
)
|
|
|
|
environments(
|
|
id,
|
|
tenant_id,
|
|
project_id,
|
|
bundle_id nullable,
|
|
name,
|
|
kind,
|
|
digest,
|
|
requirements_json,
|
|
recipe_manifest_json,
|
|
created_at
|
|
)
|
|
|
|
source_provider_configs(
|
|
id,
|
|
tenant_id,
|
|
project_id,
|
|
provider,
|
|
config_json,
|
|
created_at
|
|
)
|
|
|
|
audit_events(
|
|
id,
|
|
tenant_id,
|
|
project_id nullable,
|
|
actor_id nullable,
|
|
event_kind,
|
|
object_id nullable,
|
|
event_json,
|
|
created_at
|
|
)
|
|
```
|
|
|
|
Runtime-only structures may look like the old tables conceptually, but they should be ordinary in-memory maps/queues first: `virtual_processes`, `virtual_threads`, `debug_epochs`, active `vfs_manifests`, artifact locations, recent logs, and operator panel snapshots.
|
|
|
|
### 8.4 Tenant safety invariant
|
|
|
|
Every row that belongs to user content should include `tenant_id`. Every API should authorize by tenant and object relationship.
|
|
|
|
Required tests:
|
|
|
|
```text
|
|
user A cannot list user B nodes
|
|
user A cannot attach to user B process
|
|
user A cannot read user B logs
|
|
user A cannot read user B artifact metadata for active processes
|
|
user A cannot request debug memory from user B process
|
|
node A cannot claim tenant B process
|
|
node A cannot publish artifact for tenant B
|
|
```
|
|
|
|
### 8.5 Node enrollment
|
|
|
|
Flow:
|
|
|
|
```text
|
|
1. User opens coordinator UI or CLI.
|
|
2. User creates a node enrollment token for a project or tenant.
|
|
3. Token has short TTL and may include allowed capability constraints; node OS, architecture, and most capabilities are auto-detected at attach time.
|
|
4. Node CLI starts with token.
|
|
5. Node generates keypair.
|
|
6. Node calls enrollment endpoint.
|
|
7. Coordinator validates token and policy.
|
|
8. Coordinator creates node identity/certificate.
|
|
9. Node stores credential locally.
|
|
10. Node opens persistent session to coordinator.
|
|
```
|
|
|
|
Enrollment token example:
|
|
|
|
```json
|
|
{
|
|
"tenant_id": "tenant_abc",
|
|
"project_id": "proj_123",
|
|
"expires_at": "...",
|
|
"allowed_capabilities": [
|
|
"wasm",
|
|
"cmd",
|
|
"container.podman.rootless"
|
|
],
|
|
"auto_detect": ["os", "arch", "env", "source_providers"],
|
|
"overrides_allowed": true
|
|
}
|
|
```
|
|
|
|
### 8.6 Scheduler
|
|
|
|
The MVP scheduler should be simple, deterministic, and locality-aware.
|
|
It should also be an easily replacible module in case the user wants different scheduling, or perhaps they have limitless resources (e.g. cloud backed)
|
|
|
|
Inputs:
|
|
|
|
```text
|
|
task environment requirements
|
|
node capabilities
|
|
node health
|
|
source snapshot locations
|
|
environment cache locations
|
|
artifact locations
|
|
node current load
|
|
tenant/project policy
|
|
quota availability
|
|
debug affinity / restart affinity
|
|
```
|
|
|
|
Scoring:
|
|
|
|
```text
|
|
score(node) =
|
|
+ environment compatible
|
|
+ source snapshot already local
|
|
+ environment digest cached
|
|
+ dependency cache likely warm
|
|
+ needed artifacts local
|
|
+ same node as previous failed attempt when restart is desired
|
|
- current load
|
|
- expected bytes to transfer
|
|
- recent failures
|
|
- policy warnings
|
|
```
|
|
|
|
Placement should prefer the local node if it can run the task and already has the repo.
|
|
|
|
### 8.7 State machine
|
|
|
|
Virtual process states:
|
|
|
|
```text
|
|
CREATED
|
|
STARTING
|
|
RUNNING
|
|
FREEZE_REQUESTED
|
|
FROZEN
|
|
RESUMING
|
|
COMPLETED
|
|
FAILED
|
|
CANCELLED
|
|
```
|
|
|
|
Virtual thread states:
|
|
|
|
```text
|
|
PENDING
|
|
PLACED
|
|
STARTING
|
|
RUNNING
|
|
WAITING
|
|
COMMAND_RUNNING
|
|
FREEZE_REQUESTED
|
|
FROZEN
|
|
COMPLETED
|
|
FAILED
|
|
CANCELLED
|
|
RESTARTING
|
|
```
|
|
|
|
Debug epoch states:
|
|
|
|
```text
|
|
REQUESTED
|
|
FREEZING
|
|
FROZEN
|
|
RESUMING
|
|
RESUMED
|
|
FAILED
|
|
```
|
|
|
|
---
|
|
|
|
## 9. Transport and networking
|
|
|
|
### 9.1 Transport choices
|
|
|
|
Use Quinn for QUIC in the MVP, behind a Rust abstraction.
|
|
|
|
```rust
|
|
trait Transport {
|
|
async fn connect_node(&self, endpoint: Endpoint, identity: NodeIdentity) -> Result<Connection>;
|
|
async fn accept_node(&self) -> Result<Connection>;
|
|
async fn open_stream(&self, kind: StreamKind) -> Result<Stream>;
|
|
async fn datagram(&self, msg: Datagram) -> Result<()>;
|
|
}
|
|
```
|
|
|
|
The abstraction is important because Rust bindings and platform packaging may be rough. Keep QUIC-specific details in `disasmer-transport`.
|
|
|
|
### 9.2 Coordinator sessions
|
|
|
|
The coordinator session carries:
|
|
|
|
```text
|
|
node heartbeat
|
|
capability updates
|
|
task start/stop messages
|
|
debug epoch messages
|
|
VFS manifest publication
|
|
artifact metadata for active processes publication
|
|
log event stream
|
|
rendezvous endpoint exchange
|
|
quota updates
|
|
```
|
|
|
|
For development, it is acceptable to bootstrap auth over HTTPS and then use QUIC for persistent sessions.
|
|
|
|
### 9.3 Node-to-node sessions
|
|
|
|
Node-to-node sessions carry:
|
|
|
|
```text
|
|
blob chunk requests
|
|
artifact reads
|
|
source snapshot delta transfer
|
|
VFS chunk transfer
|
|
debug lazy memory/page reads where appropriate
|
|
```
|
|
|
|
They should not carry unauthenticated traffic. Every stream must include:
|
|
|
|
```text
|
|
tenant id
|
|
project id
|
|
process id when applicable
|
|
authorized object id
|
|
fencing token / epoch where applicable
|
|
request id
|
|
quota accounting key
|
|
```
|
|
|
|
### 9.4 Rendezvous without relay
|
|
|
|
Coordinator-aided connection:
|
|
|
|
```text
|
|
1. Node A needs artifact from Node B.
|
|
2. A asks coordinator for artifact locations.
|
|
3. Coordinator authorizes A and returns candidate endpoints for B.
|
|
4. Coordinator notifies B to expect A.
|
|
5. A and B attempt direct QUIC connection.
|
|
6. If connection succeeds, data moves node-to-node.
|
|
7. If connection fails, scheduler avoids this transfer or asks user to sync/export through configured durable storage.
|
|
```
|
|
|
|
No silent bulk relay through the coordinator.
|
|
|
|
### 9.5 Failure handling
|
|
|
|
If direct node-to-node connection fails:
|
|
|
|
- do not fall back to coordinator bulk transfer by default;
|
|
- mark the object location as unreachable from the requesting node;
|
|
- choose a different placement if possible;
|
|
- if impossible, fail with a clear action:
|
|
|
|
```text
|
|
Artifact is on node windows-1, but linux-1 cannot establish direct QUIC.
|
|
Run an artifact receiver node, configure an explicit external storage/export task, or run the dependent task on windows-1.
|
|
```
|
|
|
|
---
|
|
|
|
## 10. Environment system
|
|
|
|
### 10.1 Discovery
|
|
|
|
Scan default paths:
|
|
|
|
```text
|
|
envs/<name>/Containerfile
|
|
envs/<name>/Dockerfile
|
|
envs/<name>/flake.nix
|
|
envs/<name>/flake.lock
|
|
```
|
|
|
|
Normalize to:
|
|
|
|
```json
|
|
{
|
|
"name": "linux",
|
|
"kind": "container",
|
|
"digest": "sha256:...",
|
|
"recipe": "envs/linux/Containerfile",
|
|
"context": "envs/linux",
|
|
"requires": {
|
|
"os": "linux",
|
|
"arch": "x86_64"
|
|
},
|
|
"capabilities": {
|
|
"cmd": true,
|
|
"network": false
|
|
}
|
|
}
|
|
```
|
|
|
|
### 10.2 Containerfile MVP
|
|
|
|
Containerfile is the MVP path.
|
|
|
|
Linux:
|
|
|
|
```text
|
|
Containerfile -> Podman build -> local image id -> run task command
|
|
```
|
|
|
|
Windows:
|
|
|
|
```text
|
|
Containerfile discovered and hashed
|
|
Windows backend may treat it as a declared environment contract/stub
|
|
actual command execution uses WindowsCommandDevRunner for attached user nodes
|
|
```
|
|
|
|
This is intentionally not marketed as secure Windows container isolation.
|
|
|
|
### 10.3 Nix plugin
|
|
|
|
Nix should be present as a plugin because the architecture already expects pluggable environment resources, and Nix is useful for development reproducibility.
|
|
|
|
MVP Nix support can be:
|
|
|
|
```text
|
|
discovery of flake.nix and flake.lock
|
|
hashing into environment digest
|
|
optional local materialization on nodes with Nix installed
|
|
cache location reporting
|
|
not required for the flagship launch demo
|
|
```
|
|
|
|
### 10.4 Environment materialization cache
|
|
|
|
Cache key:
|
|
|
|
```text
|
|
env digest + backend kind + platform + Disasmer env ABI version
|
|
```
|
|
|
|
Cache record:
|
|
|
|
```json
|
|
{
|
|
"env_digest": "sha256:...",
|
|
"node_id": "node_123",
|
|
"backend": "podman-rootless",
|
|
"materialized_ref": "localhost/disasmer/env@sha256:...",
|
|
"status": "ready",
|
|
"last_used_at": "..."
|
|
}
|
|
```
|
|
|
|
### 10.5 Environment security policy
|
|
|
|
Default task environment policy:
|
|
|
|
```text
|
|
network disabled unless explicitly granted
|
|
host filesystem inaccessible except declared VFS mounts
|
|
secrets absent unless explicitly granted
|
|
no privileged containers
|
|
no Docker/Podman socket mount
|
|
read-only source mount when possible
|
|
write access only to /work and artifact staging
|
|
logs capped
|
|
process lifetime capped by policy
|
|
```
|
|
|
|
For user-attached nodes, Disasmer should still enforce its policy, but documentation must be honest that the node owner controls the machine and can bypass local restrictions.
|
|
|
|
---
|
|
|
|
## 11. VFS, artifacts, and latency
|
|
|
|
### 11.1 Core rule
|
|
|
|
```text
|
|
Move metadata eagerly.
|
|
Move bytes lazily.
|
|
Run work where the bytes already are.
|
|
```
|
|
|
|
This should be treated as an engineering invariant, not a performance optimization later.
|
|
|
|
### 11.2 Mount model
|
|
|
|
Default virtual filesystem:
|
|
|
|
```text
|
|
/src or /vfs/project
|
|
read-only source/input snapshot
|
|
|
|
/work or /vfs/work
|
|
task-local writable overlay
|
|
|
|
/vfs/artifacts
|
|
published artifact namespace
|
|
|
|
/vfs/shared
|
|
optional explicit shared namespace, probably not launch-critical
|
|
```
|
|
|
|
### 11.3 Source snapshots and source-provider modules
|
|
|
|
Source support should be pluggable. Git is the default included source-provider module, not a hard dependency of the VFS core.
|
|
|
|
A source-provider module must be able to produce a `SourceSnapshot` manifest such as:
|
|
|
|
```text
|
|
provider name and version
|
|
source identity
|
|
base revision or content root
|
|
sub-resource state if applicable
|
|
changed-file / dirty overlay manifest
|
|
included untracked or generated inputs
|
|
ignored-file policy
|
|
```
|
|
|
|
For Git specifically, the default module should model a source snapshot as:
|
|
|
|
```text
|
|
repo identity
|
|
commit hash
|
|
submodule state
|
|
dirty overlay manifest
|
|
included untracked files
|
|
ignored-file policy
|
|
```
|
|
|
|
Local node behavior:
|
|
|
|
```text
|
|
if task runs on node with checkout:
|
|
bind mount checkout read-only where safe
|
|
materialize dirty overlay if needed
|
|
else:
|
|
fetch missing content through the enabled source provider or by content hash
|
|
```
|
|
|
|
Do not tar and upload the entire repo by default. Do not require users who use another source system to carry Git dependencies in their runtime.
|
|
|
|
### 11.4 Dirty overlay
|
|
|
|
Dirty overlay manifest:
|
|
|
|
```json
|
|
{
|
|
"base_commit": "abc123",
|
|
"files": [
|
|
{
|
|
"path": "src/main.rs",
|
|
"mode": "100644",
|
|
"digest": "sha256:...",
|
|
"size": 12345
|
|
}
|
|
],
|
|
"deleted": ["old/file.rs"]
|
|
}
|
|
```
|
|
|
|
Dirty bytes move only if the chosen node does not already have them.
|
|
|
|
### 11.5 VFS overlay commits
|
|
|
|
Each task has a writable overlay. `flush()` creates a manifest delta:
|
|
|
|
```json
|
|
{
|
|
"process_id": "vp_123",
|
|
"thread_id": "vt_456",
|
|
"parent_epoch": 4,
|
|
"epoch": 5,
|
|
"created": [
|
|
{
|
|
"path": "/vfs/artifacts/linux-x64/app",
|
|
"digest": "sha256:...",
|
|
"size": 9912321,
|
|
"locations": ["node_linux_1"]
|
|
}
|
|
],
|
|
"modified": [],
|
|
"deleted": []
|
|
}
|
|
```
|
|
|
|
The coordinator stores the manifest and locations. It does not receive the bytes unless a configured durability/export policy requires it.
|
|
|
|
### 11.6 `flush()` semantics
|
|
|
|
`flush()` means:
|
|
|
|
```text
|
|
close current task write epoch
|
|
hash changed files/chunks
|
|
publish VFS manifest delta
|
|
publish artifact metadata for /vfs/artifacts changes
|
|
make changes visible to downstream tasks according to VFS consistency rules
|
|
return after metadata is accepted by coordinator
|
|
```
|
|
|
|
`flush()` should not:
|
|
|
|
```text
|
|
upload large artifacts by default
|
|
replicate to other nodes by default
|
|
export to local dist by default
|
|
route bytes through coordinator
|
|
```
|
|
|
|
### 11.7 `sync()` and explicit artifact movement semantics
|
|
|
|
`sync()` means:
|
|
|
|
```text
|
|
apply an explicit export/transfer/durability policy chosen by the program or CLI
|
|
verify referenced bytes still exist on at least one node or explicit store
|
|
move bytes directly between nodes or to a user-configured external destination
|
|
update artifact metadata for active processes with the result
|
|
```
|
|
|
|
`sync()` should not imply a default coordinator artifact store. By default, artifacts and other large data are nearly ephemeral: they live on nodes for as long as a task, downstream consumer, export, restart, or configured cache/GC policy needs them. They may linger long enough for practical task restart, but unsynced node-local artifacts are not a durability guarantee.
|
|
|
|
Example policies:
|
|
|
|
```rust
|
|
// Ask an attached/local receiver node to pull bytes into ./dist.
|
|
artifact::export_to_node(dist, receiver::local("dist")).await?;
|
|
|
|
// Send bytes to a user-provided storage integration implemented by project code.
|
|
artifact::copy_to(outputs, storage::s3("release-bucket")).await?;
|
|
|
|
// Keep artifacts available on the producing node long enough for likely restarts.
|
|
artifact::retain(outputs, RetainPolicy::BestEffort { min_ttl: Duration::from_hours(24) }).await?;
|
|
```
|
|
|
|
MVP should implement at least:
|
|
|
|
```text
|
|
metadata-only flush
|
|
node-local ephemeral artifact/blob cache
|
|
explicit export to an attached/local receiver node
|
|
best-effort retention/GC knobs
|
|
clear errors when bytes have been garbage-collected or are unreachable
|
|
```
|
|
|
|
Users who want artifacts on their own machine should either run a receiver node there or write/use an explicit storage/export integration. The coordinator should keep artifact metadata for active processes, not act as the default artifact data store.
|
|
|
|
### 11.8 Artifact lifecycle
|
|
|
|
Artifact states:
|
|
|
|
```text
|
|
PUBLISHED_METADATA
|
|
AVAILABLE_ON_NODE
|
|
RETAINED_BEST_EFFORT
|
|
EXPORTED
|
|
DURABLE_EXTERNAL
|
|
EXPIRED
|
|
DELETED
|
|
```
|
|
|
|
To allow producing artifacts without much effort from the user, the control pane could provide a download button through the immediate-mode UI, which will stream the (best-effort retained) artefact from the node that retains it. "Best effort" and usage tier limits apply as normal, but it should be an error to create a download link to an artifact that can't be downloaded within the limits in the first place. Download links should be secure and not just allow anyone that guesses the correct link to download any artifact.
|
|
|
|
Artifact metadata:
|
|
|
|
```json
|
|
{
|
|
"artifact_id": "art_123",
|
|
"path": "/vfs/artifacts/linux-x64/app",
|
|
"digest": "sha256:...",
|
|
"size": 9912321,
|
|
"producer_thread_id": "vt_linux",
|
|
"locations": [
|
|
{ "kind": "node", "node_id": "node_linux_1" }
|
|
],
|
|
"durability_state": "PUBLISHED_METADATA",
|
|
"retention": "best_effort_node_gc"
|
|
}
|
|
```
|
|
|
|
### 11.9 Local-first acceptance test
|
|
|
|
For a local Linux build from an existing Git checkout:
|
|
|
|
- no full-repo tarball is created;
|
|
- no source bytes are uploaded to coordinator;
|
|
- compiler file reads are local;
|
|
- build cache reads are local;
|
|
- artifact bytes remain local after `flush()`;
|
|
- coordinator receives only metadata, logs within caps, and debug/session events;
|
|
- explicit export/transfer code is the first operation allowed to move final artifact bytes for durability or user-visible export.
|
|
|
|
---
|
|
|
|
## 12. Task execution lifecycle
|
|
|
|
### 12.1 Spawn flow
|
|
|
|
```text
|
|
1. Guest calls spawn::task(compile).arg(...).env(env!("linux")).start().await.
|
|
2. Wasm guest invokes disasmer.spawn.start host import.
|
|
3. Node host validates current task capability.
|
|
4. Host serializes TaskSpec with task id, env id, args, handles, VFS base epoch.
|
|
5. Coordinator creates virtual_thread row in PENDING state.
|
|
6. Scheduler chooses node.
|
|
7. Coordinator sends StartTask to chosen node.
|
|
8. Node fetches or locates bundle.
|
|
9. Node materializes environment if needed.
|
|
10. Node creates VFS overlay.
|
|
11. Node instantiates Wasmtime module or command wrapper.
|
|
12. Node calls named task entrypoint.
|
|
13. Thread becomes visible in debugger and coordinator UI.
|
|
```
|
|
|
|
### 12.2 Join flow
|
|
|
|
```text
|
|
1. Parent awaits TaskHandle.join().
|
|
2. Parent task parks or yields.
|
|
3. Child completes with serialized result or error.
|
|
4. Result may contain handles such as Artifact.
|
|
5. Coordinator records completion.
|
|
6. Parent is woken.
|
|
7. Parent receives decoded result.
|
|
```
|
|
|
|
### 12.3 Command flow inside task
|
|
|
|
```text
|
|
1. Guest task calls cmd!(...).run().await.
|
|
2. Host import validates command capability.
|
|
3. Node command runner starts process in task environment.
|
|
4. Task enters COMMAND_RUNNING state.
|
|
5. Logs stream to node and coordinator under caps.
|
|
6. Debug Epoch requests suspend controlled process tree.
|
|
7. Command exit status is returned to guest.
|
|
8. Guest decides whether to publish artifacts.
|
|
```
|
|
|
|
### 12.4 Failure behavior
|
|
|
|
MVP rules:
|
|
|
|
```text
|
|
Task trap -> task FAILED, process may enter debug break depending policy.
|
|
Command nonzero exit -> task receives error, can handle or fail.
|
|
Node lost with non-replicated running state -> virtual process FAILED.
|
|
Node lost after flushed but unsynced artifact -> artifact may be unavailable.
|
|
Node lost after synced artifact -> artifact survives according to sync policy.
|
|
```
|
|
|
|
Do not pretend non-replicated state is recoverable.
|
|
|
|
### 12.5 Restart flow
|
|
|
|
```text
|
|
1. User edits source.
|
|
2. VS Code triggers disasmer build.
|
|
3. New bundle is produced.
|
|
4. User selects Restart Failed Task or Restart Selected Virtual Thread.
|
|
5. DAP/coordinator checks task compatibility hash.
|
|
6. Runtime stops old task if needed.
|
|
7. Unflushed overlay changes are discarded unless explicitly preserved.
|
|
8. New task starts with original args, same env, chosen VFS checkpoint.
|
|
9. Breakpoints are rebound.
|
|
10. Other tasks remain alive only if compatibility checks pass.
|
|
```
|
|
|
|
If compatibility fails:
|
|
|
|
```text
|
|
Task signature changed; restarting whole virtual process is required.
|
|
```
|
|
|
|
This capability is important to advertise in any demo(s).
|
|
|
|
---
|
|
|
|
## 13. Debugging
|
|
|
|
### 13.1 Design goal
|
|
|
|
The user should experience Disasmer as one ordinary debug target:
|
|
|
|
```text
|
|
Process: launch-build-demo
|
|
Threads:
|
|
1 main
|
|
2 compile linux-x64
|
|
3 compile windows-x64
|
|
4 package
|
|
```
|
|
|
|
The debugger should not force the user to understand node IDs, QUIC links, VFS manifests, or environment cache internals unless they open the Disasmer inspector.
|
|
|
|
### 13.2 Use a custom DAP adapter
|
|
|
|
Use `disasmer-dap` as the VS Code debug adapter.
|
|
|
|
Why:
|
|
|
|
- DAP is the right integration point for VS Code.
|
|
- The target is not one native process; it is a virtual process spanning nodes.
|
|
- Disasmer must control the thread list, stop reasons, restart commands, artifact views, and all-stop semantics.
|
|
- LLDB can be a future internal helper, but it should not own the UX.
|
|
|
|
### 13.3 Debug components
|
|
|
|
```text
|
|
VS Code extension
|
|
launches disasmer-dap and contributes commands/views
|
|
|
|
disasmer-dap
|
|
speaks DAP to VS Code
|
|
speaks Disasmer debug protocol to coordinator
|
|
maps DAP process/thread/stack/variables to virtual model
|
|
|
|
coordinator debug service
|
|
owns debug sessions and Debug Epochs
|
|
broadcasts freeze/resume
|
|
aggregates thread state
|
|
|
|
node debug agent
|
|
sets breakpoints/probes
|
|
freezes/resumes Wasmtime tasks
|
|
suspends/resumes controlled command processes
|
|
captures stacks/locals/handle state
|
|
```
|
|
|
|
### 13.4 Debug Epoch protocol
|
|
|
|
When a breakpoint is hit:
|
|
|
|
```text
|
|
1. Node detects breakpoint/trap/probe in virtual thread T.
|
|
2. Node reports BreakHit(process, thread, location) to coordinator.
|
|
3. Coordinator creates DebugEpoch E.
|
|
4. Coordinator marks process FREEZE_REQUESTED.
|
|
5. Coordinator sends Freeze(E) to all nodes hosting threads for the process.
|
|
6. Wasm tasks stop at current probe/safepoint or next safepoint.
|
|
7. Native command tasks are suspended using backend-specific process-tree control.
|
|
8. VFS overlay manifests are pinned.
|
|
9. Memory/handle snapshots are pinned copy-on-write where applicable.
|
|
10. Nodes report Frozen(E, thread states).
|
|
11. Coordinator marks epoch FROZEN.
|
|
12. DAP emits stopped event for all threads.
|
|
```
|
|
|
|
Resume:
|
|
|
|
```text
|
|
1. User presses Continue.
|
|
2. DAP sends Continue(E) to coordinator.
|
|
3. Coordinator sends Resume(E) to nodes.
|
|
4. Nodes unpin debug snapshots.
|
|
5. Wasm tasks continue.
|
|
6. Native command tasks resume.
|
|
7. Coordinator marks epoch RESUMED.
|
|
```
|
|
|
|
### 13.5 Safepoints and breakpoints
|
|
|
|
MVP should use layered breakpoint support:
|
|
|
|
#### Layer 1: explicit debug probes
|
|
|
|
`#[disasmer::task]` and `#[disasmer::main]` generate probes at:
|
|
|
|
```text
|
|
task entry
|
|
before/after spawn
|
|
before/after join
|
|
before/after cmd run
|
|
before/after flush/sync
|
|
await boundaries
|
|
explicit disasmer::debug::probe!() calls
|
|
```
|
|
|
|
This provides reliable all-stop behavior early.
|
|
|
|
#### Layer 2: source-line breakpoints
|
|
|
|
The build pipeline maps source line breakpoints to probe points and, where possible, Wasm code offsets using debug metadata.
|
|
|
|
Launch acceptance should include:
|
|
|
|
```text
|
|
breakpoint in main()
|
|
breakpoint in compile()
|
|
breakpoint before/after cmd!().run()
|
|
breakpoint before artifact publish/flush
|
|
breakpoint in package()
|
|
```
|
|
|
|
#### Layer 3: richer DWARF/LLDB integration
|
|
|
|
After the DAP skeleton works, improve locals/stacks by reading Rust/Wasm debug metadata. This is important, but it should not block the all-stop architecture.
|
|
|
|
### 13.6 Variables and locals
|
|
|
|
Launchable debugger variable support should include:
|
|
|
|
```text
|
|
task arguments
|
|
return values
|
|
Target enums and simple structs
|
|
Artifact handles
|
|
SourceSnapshot handles
|
|
Blob handles
|
|
DAtomic values
|
|
DChannel status
|
|
current VFS mounts
|
|
current command spec/status
|
|
recent stdout/stderr tail
|
|
```
|
|
|
|
For arbitrary Rust locals, use best-effort DWARF support where available. If a local cannot be inspected, the debugger should say so clearly and still show task args/handles.
|
|
|
|
### 13.7 Native command tasks in debugger
|
|
|
|
A native command is debugger-visible as part of the virtual thread, but MVP does not need to debug inside `cargo` or the compiler itself.
|
|
|
|
The command frame should show:
|
|
|
|
```text
|
|
command argv
|
|
working directory
|
|
environment name
|
|
node id in inspector only
|
|
start time
|
|
status
|
|
exit code when done
|
|
recent stdout/stderr
|
|
resource limits
|
|
```
|
|
|
|
When the virtual process is frozen, controlled native command processes should be suspended. If the backend cannot safely suspend a command, the node must report failure and the debugger must show a clear error rather than claiming all-stop succeeded.
|
|
|
|
### 13.8 DAP features required for launch
|
|
|
|
Required:
|
|
|
|
```text
|
|
initialize
|
|
launch / attach
|
|
setBreakpoints
|
|
configurationDone
|
|
threads
|
|
stackTrace
|
|
scopes
|
|
variables
|
|
continue
|
|
pause
|
|
next / step-over at probe granularity
|
|
disconnect
|
|
stopped events
|
|
continued events
|
|
output events
|
|
custom restart-task request
|
|
```
|
|
|
|
Not required for launch:
|
|
|
|
```text
|
|
watchpoints
|
|
reverse debugging
|
|
non-stop debugging
|
|
hot code replace
|
|
debugging arbitrary child process internals
|
|
```
|
|
|
|
### 13.9 Disasmer inspector
|
|
|
|
Default debugger view remains normal. Add an optional inspector tree:
|
|
|
|
```text
|
|
Virtual Process
|
|
Threads
|
|
compile linux-x64 -> node linux-1 -> env linux -> running/frozen
|
|
compile windows-x64 -> node windows-1 -> env windows -> running/frozen
|
|
VFS
|
|
epoch 4
|
|
artifacts
|
|
Debug Epoch
|
|
id E
|
|
frozen nodes
|
|
Environments
|
|
linux digest cached on linux-1
|
|
windows digest cached/stub on windows-1
|
|
Transport
|
|
node links status
|
|
```
|
|
|
|
This helps explain the system without polluting the normal debug UX.
|
|
|
|
---
|
|
|
|
## 14. VS Code extension
|
|
|
|
### 14.1 Extension responsibilities
|
|
|
|
The extension should:
|
|
|
|
```text
|
|
watch envs/**/Containerfile
|
|
watch envs/**/Dockerfile
|
|
watch envs/**/flake.nix
|
|
watch envs/**/flake.lock
|
|
watch inputs/**
|
|
watch src/**
|
|
run disasmer build before debugging
|
|
launch/attach DAP session
|
|
show virtual threads in normal debug UI
|
|
show per-task logs
|
|
show artifact tree
|
|
show node connection status
|
|
provide env!("...") autocomplete
|
|
provide disasmer.toml diagnostics
|
|
provide restart selected/failed task commands
|
|
```
|
|
|
|
### 14.2 Launch configuration
|
|
|
|
Example `.vscode/launch.json`:
|
|
|
|
```json
|
|
{
|
|
"version": "0.2.0",
|
|
"configurations": [
|
|
{
|
|
"name": "Debug Disasmer Build",
|
|
"type": "disasmer",
|
|
"request": "launch",
|
|
"coordinator": "https://disasmer.example.com",
|
|
"project": "${workspaceFolder}",
|
|
"preLaunchTask": "disasmer: build",
|
|
"localFirst": true
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### 14.3 Editor diagnostics
|
|
|
|
Diagnostics:
|
|
|
|
```text
|
|
env!("name") missing
|
|
env discovered but incompatible with available nodes
|
|
Containerfile changed; bundle rebuild needed
|
|
large task argument warning
|
|
cmd! used in task without cmd capability
|
|
fs::sync missing for final artifact export warning
|
|
Windows backend is dev/stub isolation warning
|
|
```
|
|
|
|
### 14.4 VS Code views
|
|
|
|
MVP views:
|
|
|
|
```text
|
|
Disasmer: Nodes
|
|
Disasmer: Virtual Processes
|
|
Disasmer: Artifacts
|
|
Disasmer: Logs
|
|
Disasmer: Inspector
|
|
```
|
|
|
|
Operator panel preview can appear as:
|
|
|
|
```text
|
|
Disasmer: Operator Panel Preview
|
|
```
|
|
|
|
But the full immediate-mode panel is second milestone.
|
|
|
|
---
|
|
|
|
## 15. Hosted coordinator and product UX
|
|
|
|
### 15.1 Hosted coordinator MVP
|
|
|
|
Hosted coordinator pages:
|
|
|
|
```text
|
|
Sign in
|
|
Projects
|
|
Project detail
|
|
Node attach instructions
|
|
Node list
|
|
Virtual process list
|
|
Virtual process detail
|
|
Logs
|
|
Artifacts
|
|
Debug session status
|
|
Operator panel placeholder/preview
|
|
Settings / quotas
|
|
```
|
|
|
|
### 15.2 Hosted Authentik/OIDC integration
|
|
|
|
Use Authentik as the identity provider through OIDC in the private hosted module. The open-source/private coordinator should expose auth hooks and simpler local/team auth, but should not require the hosted OIDC stack.
|
|
|
|
Implementation requirements:
|
|
|
|
```text
|
|
authorization code + PKCE for web login
|
|
strict redirect URI allowlist
|
|
minimal scopes
|
|
short session lifetime
|
|
refresh handling if needed
|
|
logout/back-channel support later
|
|
CLI token issuance after web login
|
|
node enrollment tokens separate from user sessions
|
|
```
|
|
|
|
### 15.3 Community tier
|
|
|
|
Community tier defaults:
|
|
|
|
```text
|
|
1 active virtual process
|
|
tiny hosted Wasm control loop limit
|
|
no hosted native commands
|
|
no hosted containers
|
|
no hosted network from tasks
|
|
small logs
|
|
small metadata
|
|
small artifact retention
|
|
4 user-attached nodes allowed
|
|
operator panel access
|
|
```
|
|
|
|
The hosted control loop is useful for trying Disasmer and coordinating attached nodes, not for mining, scraping, compiling huge repos, or hosting files.
|
|
|
|
### 15.4 Abuse controls as architecture
|
|
|
|
Every resource-consuming action must pass through a hosted/community quota policy module before work starts.
|
|
|
|
Quota dimensions:
|
|
|
|
```text
|
|
active virtual processes
|
|
hosted Wasm fuel
|
|
hosted Wasm memory
|
|
hosted process wall-clock
|
|
API calls
|
|
node heartbeats
|
|
spawn attempts
|
|
log bytes
|
|
metadata bytes
|
|
artifact bytes
|
|
artifact downloads
|
|
debug memory reads
|
|
operator panel updates
|
|
rendezvous attempts
|
|
```
|
|
|
|
Pattern:
|
|
|
|
```rust
|
|
quota.reserve(tenant, Resource::SpawnAttempt, 1).await?;
|
|
// perform operation
|
|
quota.commit(reservation).await?;
|
|
```
|
|
|
|
For streams, use incremental accounting and backpressure.
|
|
|
|
### 15.5 Public input threat model
|
|
|
|
Treat as hostile:
|
|
|
|
```text
|
|
users
|
|
OAuth identities
|
|
nodes
|
|
programs
|
|
build files
|
|
Containerfiles
|
|
Git repos
|
|
logs
|
|
artifacts
|
|
operator panel text/events
|
|
debug requests
|
|
node capabilities
|
|
cache claims
|
|
```
|
|
|
|
### 15.6 Public launch security baseline
|
|
|
|
Required before public launch:
|
|
|
|
- Tenant ID on every object.
|
|
- Authorization check on every API endpoint.
|
|
- Negative cross-tenant tests.
|
|
- No hosted containers on community tier.
|
|
- No hosted native commands on community tier.
|
|
- No arbitrary network access from hosted Wasm.
|
|
- Rate limits on auth, APIs, rendezvous, logs, spawn, and UI events.
|
|
- Log and artifact size caps.
|
|
- Private artifacts by default.
|
|
- No public file hosting by default.
|
|
- Node identity signing/revocation.
|
|
- Task capabilities explicit.
|
|
- Debug attach requires project permission.
|
|
- Debug memory reads quota-limited and audited.
|
|
- Operator panel uses built-in widgets only.
|
|
- No custom HTML or JS in panels.
|
|
- No password fields in user-defined panels.
|
|
- No OAuth flows inside operator panels.
|
|
|
|
---
|
|
|
|
## 16. Operator panel scaffold and second milestone
|
|
|
|
### 16.1 MVP scaffold
|
|
|
|
Even if the full panel is second milestone, the MVP should include:
|
|
|
|
```text
|
|
ui model crate
|
|
panel snapshot table
|
|
panel event table
|
|
coordinator API namespace
|
|
web placeholder area
|
|
DAP/debug interaction policy
|
|
security restrictions
|
|
```
|
|
|
|
The first visible version can be a fixed coordinator dashboard showing:
|
|
|
|
```text
|
|
process state
|
|
tasks
|
|
progress
|
|
logs
|
|
artifacts
|
|
Debug / Restart / Cancel buttons as control-plane actions
|
|
```
|
|
|
|
### 16.2 Immediate-mode second milestone
|
|
|
|
The second milestone adds:
|
|
|
|
```rust
|
|
let panel = ui::panel("Repo builder").await?;
|
|
|
|
loop {
|
|
let action = panel.draw(|ui| {
|
|
ui.heading("Repo builder");
|
|
ui.text("Build status");
|
|
ui.task_table("Tasks", tasks.snapshot());
|
|
ui.log_tail("Recent output", logs.tail());
|
|
if ui.button("Restart failed").clicked() {
|
|
ui.emit(Action::RestartFailed);
|
|
}
|
|
}).await?;
|
|
|
|
if let Some(action) = action {
|
|
handle(action).await?;
|
|
}
|
|
}
|
|
```
|
|
|
|
### 16.3 Panel security model
|
|
|
|
MVP/second milestone rules:
|
|
|
|
```text
|
|
built-in widgets only
|
|
no custom HTML
|
|
no custom JavaScript
|
|
all text escaped
|
|
external links restricted
|
|
no OAuth/password widgets
|
|
panel labeled as user-program-provided
|
|
rate-limited updates
|
|
state size capped
|
|
events typed and schema-checked
|
|
read-only while debugger has process frozen
|
|
control-plane actions may remain available while frozen
|
|
```
|
|
|
|
---
|
|
|
|
## 17. Security and capability model
|
|
|
|
### 17.1 Capability-first execution
|
|
|
|
Every task has a capability set.
|
|
|
|
Example:
|
|
|
|
```json
|
|
{
|
|
"task": "compile",
|
|
"capabilities": {
|
|
"vfs.read": ["/src"],
|
|
"vfs.write": ["/work", "/vfs/artifacts/linux-x64"],
|
|
"cmd.run": true,
|
|
"network.outbound": false,
|
|
"secrets": []
|
|
}
|
|
}
|
|
```
|
|
|
|
### 17.2 Coordinator authz checks
|
|
|
|
Every coordinator API checks:
|
|
|
|
```text
|
|
authenticated principal
|
|
principal tenant membership
|
|
object tenant id
|
|
project membership/role
|
|
operation permission
|
|
quota availability
|
|
object state compatibility
|
|
```
|
|
|
|
### 17.3 Node trust model
|
|
|
|
Default rule:
|
|
|
|
```text
|
|
A user's work runs only on that user's nodes, managed paid nodes, or explicitly shared team nodes.
|
|
```
|
|
|
|
For MVP:
|
|
|
|
- do not mix tenants on user-provided nodes;
|
|
- do not trust user-provided nodes to produce truthful artifacts for other tenants;
|
|
- label artifacts with node provenance;
|
|
- expose trust label in inspector;
|
|
- eventually support reproducible repeated builds for higher trust.
|
|
|
|
### 17.4 Bundle identity
|
|
|
|
Bundles should be content-addressed and integrity-checked.
|
|
|
|
Later signing can be added, but MVP should at least ensure:
|
|
|
|
```text
|
|
coordinator records bundle digest
|
|
node verifies bundle digest before running
|
|
manifest records environment/resource digests
|
|
task specs reference bundle digest
|
|
artifact provenance references bundle/task/node
|
|
```
|
|
|
|
### 17.5 Secrets
|
|
|
|
MVP secrets support should be minimal.
|
|
|
|
Recommended launch stance:
|
|
|
|
```text
|
|
no secrets in hosted/community tasks
|
|
secrets not shown in logs/UI/debug by default
|
|
explicit grant required per task/environment
|
|
short-lived scoped secret materialization
|
|
secret access audited
|
|
secret redaction in logs
|
|
```
|
|
|
|
For the first launch demo, avoid secrets entirely.
|
|
|
|
### 17.6 Container/runtime hardening
|
|
|
|
Linux Podman backend should default to:
|
|
|
|
```text
|
|
rootless
|
|
no privileged containers
|
|
no host Docker/Podman socket mount
|
|
read-only source mount
|
|
writable task workdir only
|
|
limited network by default
|
|
seccomp/AppArmor/SELinux where available
|
|
resource limits where practical
|
|
```
|
|
|
|
Windows dev backend should be clearly scoped as user-attached-node execution, not public managed secure isolation.
|
|
|
|
---
|
|
|
|
## 18. Implementation milestones
|
|
|
|
This sequence avoids building a beautiful UI on top of a runtime that cannot yet prove the thesis.
|
|
|
|
### Milestone 0: Repository and protocol spine
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
monorepo layout
|
|
shared IDs and protocol types
|
|
CoordinatorStore trait
|
|
Postgres persistent-store migrations skeleton
|
|
CLI skeleton
|
|
node skeleton
|
|
SDK skeleton
|
|
VS Code extension skeleton
|
|
DAP skeleton
|
|
example project skeleton
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
cargo workspace builds
|
|
CLI can print discovered project info
|
|
coordinator starts locally with Postgres for persistent project/node state and in-memory runtime state
|
|
node starts and registers fake heartbeat locally
|
|
VS Code extension can run a dummy DAP session
|
|
```
|
|
|
|
Key considerations:
|
|
|
|
- Do not overdesign protocol versioning, but add a protocol version field immediately.
|
|
- Use stable IDs everywhere.
|
|
- Add tenant_id to data models from the first migration.
|
|
|
|
### Milestone 1: Local single-node Wasm task execution
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
#[disasmer::main]
|
|
#[disasmer::task]
|
|
DisasmerArg derive
|
|
named task registry
|
|
Wasmtime runner
|
|
spawn/join on same node
|
|
small serialized args/results
|
|
basic logs
|
|
basic DAtomicU32 or equivalent demo handle
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
example local-one-node runs main task
|
|
main spawns child Wasm task
|
|
child returns serializable result
|
|
parent joins result
|
|
logs show per virtual thread
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- Keep the first task ABI tiny.
|
|
- Ban closures for MVP.
|
|
- Fail early on invalid args.
|
|
|
|
### Milestone 2: Environment discovery and bundle generation
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
envs/** discovery
|
|
env!("name") compile/editor validation
|
|
disasmer build creates app.dis directory
|
|
bundle manifest
|
|
task manifest
|
|
environment manifest
|
|
source snapshot manifest placeholder
|
|
VFS seed manifest
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
adding envs/linux/Containerfile makes env!("linux") valid
|
|
missing env gives useful error
|
|
bundle digest changes when Containerfile changes
|
|
bundle digest changes when task ABI changes
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- Keep `disasmer.toml` optional.
|
|
- Make generated metadata visible and inspectable.
|
|
|
|
### Milestone 3: VFS and artifacts
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
per-task VFS overlay
|
|
/vfs/project or /src mount
|
|
/vfs/work
|
|
/vfs/artifacts
|
|
fs::copy
|
|
fs::flush metadata publication
|
|
Artifact::from_path
|
|
fs::sync export-to-local-directory
|
|
content-addressed blob/chunk store
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
task writes artifact
|
|
flush publishes artifact metadata for active processes
|
|
parent joins Artifact handle
|
|
package task reads artifact handle
|
|
sync exports final artifact to dist/
|
|
large artifact bytes do not go through coordinator in local mode
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- Build the metadata/byte split now.
|
|
- Ensure artifact path namespace is stable.
|
|
- Add tests proving `flush()` is metadata-only.
|
|
|
|
### Milestone 4: Linux Podman command backend
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
Podman rootless env materialization
|
|
Containerfile build/cache by digest
|
|
cmd!(...).run()
|
|
stdout/stderr streaming
|
|
process limits/log caps
|
|
artifact staging from command output
|
|
basic process suspend/resume for debug epoch
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
Linux task runs cargo build in envs/linux/Containerfile
|
|
logs stream to CLI/coordinator
|
|
artifact is published through VFS
|
|
local source checkout is bind-mounted when task runs locally
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- Keep the first Containerfile expectations simple.
|
|
- Do not support arbitrary privileged containers.
|
|
- Do not mount the container engine socket.
|
|
|
|
### Milestone 5: Coordinator/node split with hosted-ready persistent store
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
coordinator service
|
|
Postgres-backed persistent store
|
|
node enrollment token
|
|
node identity
|
|
node heartbeat
|
|
bundle registration
|
|
task placement
|
|
StartTask command over persistent session
|
|
logs/events over stream
|
|
local coordinator mode still works
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
coordinator runs separately from node
|
|
node attaches with token
|
|
disasmer run schedules task onto attached node
|
|
node executes task and reports completion
|
|
Postgres contains persistent project/node/bundle metadata; active process/thread/artifact metadata lives in memory
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- Use same code path for local and hosted as much as possible.
|
|
- Do not create a special “demo scheduler” that will be thrown away.
|
|
|
|
### Milestone 6: Quinn transport and rendezvous
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
Quinn transport wrapper
|
|
node-to-coordinator persistent session if practical
|
|
node-to-node direct connection
|
|
rendezvous endpoint exchange
|
|
authenticated stream framing
|
|
blob/artifact chunk transfer protocol
|
|
no coordinator bulk relay
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
two nodes establish direct QUIC link
|
|
node A can request authorized artifact chunk from node B
|
|
failed direct connection produces clear error
|
|
coordinator never receives bulk artifact bytes by default
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- Keep HTTPS bootstrap if it makes auth simpler.
|
|
- Put transport behind traits for testability.
|
|
- Add trace logs for failed connectivity.
|
|
|
|
### Milestone 7: Windows attached node backend
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
Windows node build/test
|
|
WindowsCommandDevRunner
|
|
per-task workspace
|
|
process/log capture
|
|
artifact staging
|
|
best-effort process tree lifecycle
|
|
Windows sandbox backend trait + stub
|
|
Forgejo Windows worker setup docs
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
Windows node attaches to hosted/local coordinator
|
|
scheduler places env!("windows") task on Windows node
|
|
Windows task runs cargo build or a simple command
|
|
artifact is published and joined by parent
|
|
VS Code/coordinator shows Windows virtual thread
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- Be explicit that this is not production-grade untrusted Windows sandboxing.
|
|
- Do not offer managed public Windows compute until isolation is real.
|
|
|
|
### Milestone 8: All-stop debugging and DAP
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
custom DAP adapter
|
|
launch/attach
|
|
thread list
|
|
breakpoint setting
|
|
source/probe mapping
|
|
Debug Epoch coordinator
|
|
node freeze/resume
|
|
Wasm safepoints/probes
|
|
command process suspend/resume integration
|
|
stack/scopes/variables for task args and handles
|
|
stdout/stderr output events
|
|
restart selected/failed task command
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
VS Code F5 starts a virtual process
|
|
breakpoint in main stops all virtual threads
|
|
breakpoint in Linux compile task stops Windows task too
|
|
breakpoint in Windows task stops Linux task too
|
|
thread list is coherent
|
|
user can inspect Target arg and Artifact handle
|
|
continue resumes all tasks
|
|
task restart works after source edit
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- This is the riskiest milestone. Start early.
|
|
- Use probe-granularity stepping first.
|
|
- Make all-stop correctness more important than perfect DWARF locals.
|
|
|
|
### Milestone 9: Local-first Git/source snapshots
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
git source provider module
|
|
commit/submodule/dirty overlay manifest
|
|
local checkout detection
|
|
bind-mount local checkout for local tasks
|
|
remote missing-content transfer by hash
|
|
scheduler locality scoring
|
|
latency regression tests
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
local Linux build from local checkout does not upload repo
|
|
remote Windows build receives only required source content
|
|
changing one file transfers only that file/overlay where possible
|
|
scheduler prefers node with source checkout and env cache
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- Keep source provider modular.
|
|
- Do not hardwire Git into the VFS core.
|
|
|
|
### Milestone 10: Hosted coordinator launch hardening
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
private hosted Authentik/OIDC login
|
|
tenant/project UI
|
|
node attach UI
|
|
hosted/community quota policy module
|
|
authz middleware and private hosted OIDC integration
|
|
cross-tenant negative tests
|
|
community tier limits
|
|
rate limiting
|
|
log/artifact caps
|
|
admin kill switch
|
|
basic plan/policy flags, even if only community exists
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
public user can sign in
|
|
public user can create project
|
|
public user can attach own node
|
|
public user cannot run hosted native command/container
|
|
public user can run task on own node
|
|
cross-tenant tests pass
|
|
quota tests pass
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- Do not launch without abuse controls.
|
|
- Design quotas as first-class resource accounting.
|
|
|
|
### Milestone 11: Launch demo polish
|
|
|
|
Deliverables:
|
|
|
|
```text
|
|
launch-build example
|
|
README quickstart
|
|
short screencast script
|
|
hosted landing page
|
|
install script/packages
|
|
VS Code extension packaging
|
|
error message pass
|
|
first-run diagnostics
|
|
artifact export UX
|
|
fixed coordinator dashboard
|
|
operator panel preview scaffold
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
```text
|
|
fresh developer can follow quickstart
|
|
Linux node attach works
|
|
Windows node attach works
|
|
F5 demo works
|
|
breakpoint demo works
|
|
restart failed task demo works
|
|
artifact export works
|
|
README explains community tier honestly
|
|
```
|
|
|
|
Considerations:
|
|
|
|
- Public launch users will try weird setups immediately.
|
|
- Error messages matter as much as features.
|
|
- Make the first demo resilient.
|
|
|
|
---
|
|
|
|
## 19. Testing strategy
|
|
|
|
### 19.1 Unit tests
|
|
|
|
Required areas:
|
|
|
|
```text
|
|
DisasmerArg derive allowed/rejected types
|
|
environment discovery
|
|
bundle digest stability
|
|
manifest generation
|
|
VFS path normalization
|
|
flush manifest generation
|
|
sync policy selection
|
|
hosted/community quota policy module
|
|
scheduler scoring
|
|
authz checks
|
|
protocol encoding/decoding
|
|
```
|
|
|
|
### 19.2 Integration tests
|
|
|
|
Local integration tests:
|
|
|
|
```text
|
|
single-node spawn/join
|
|
two Wasm tasks with DAtomic
|
|
task writes artifact and parent consumes it
|
|
cmd runner fake backend
|
|
Debug Epoch with two Wasm tasks
|
|
restart failed task with compatible bundle
|
|
restart fails with incompatible schema
|
|
```
|
|
|
|
Linux integration tests:
|
|
|
|
```text
|
|
Podman build env
|
|
Podman run command
|
|
local checkout bind mount
|
|
artifact publish/export
|
|
process suspend/resume if supported
|
|
```
|
|
|
|
Windows integration tests:
|
|
|
|
```text
|
|
node attach
|
|
command execution
|
|
log capture
|
|
artifact staging
|
|
process lifecycle cleanup
|
|
```
|
|
|
|
Network integration tests:
|
|
|
|
```text
|
|
node-to-node QUIC connect
|
|
artifact chunk transfer
|
|
connection failure clear error
|
|
no coordinator bulk relay
|
|
```
|
|
|
|
### 19.3 End-to-end tests
|
|
|
|
E2E test scenarios:
|
|
|
|
```text
|
|
F5 local single node
|
|
hosted coordinator + attached Linux node
|
|
hosted coordinator + Linux + Windows nodes
|
|
breakpoint in Linux task all-stops process
|
|
breakpoint in Windows task all-stops process
|
|
restart failed Windows task
|
|
local-first no-upload test
|
|
community tier quota enforcement
|
|
cross-tenant forbidden access
|
|
```
|
|
|
|
### 19.4 Latency regression tests
|
|
|
|
Add tests that fail if:
|
|
|
|
```text
|
|
local source bytes are uploaded during local build
|
|
artifact bytes go to coordinator during flush
|
|
full repo tarball is created by default
|
|
large artifact is transferred before consumer/export/sync needs it
|
|
scheduler chooses remote node despite local compatible warm node without reason
|
|
```
|
|
|
|
### 19.5 Security tests
|
|
|
|
Required negative tests:
|
|
|
|
```text
|
|
user cannot read another tenant's process
|
|
user cannot attach debugger to another tenant's process
|
|
node cannot claim another tenant's task
|
|
node cannot publish artifact for unauthorized process
|
|
community hosted task cannot request cmd.run
|
|
community hosted task cannot request network
|
|
operator panel cannot submit custom HTML/JS
|
|
log cap truncates noisy task
|
|
quota rejects spawn storm
|
|
```
|
|
|
|
---
|
|
|
|
## 20. Documentation required for MVP
|
|
|
|
### 20.1 User docs
|
|
|
|
```text
|
|
README quickstart
|
|
What is Disasmer?
|
|
Install CLI
|
|
Install VS Code extension
|
|
Login to hosted coordinator
|
|
Attach a Linux node
|
|
Attach a Windows node
|
|
Create envs/<name>/Containerfile
|
|
Write first build program
|
|
Debug in VS Code
|
|
Publish artifacts with flush()
|
|
Export artifacts through an attached receiver node or explicit storage integration
|
|
Community tier limits
|
|
Security model for user-attached nodes
|
|
```
|
|
|
|
### 20.2 Engineering docs
|
|
|
|
```text
|
|
architecture.md
|
|
protocol.md
|
|
task-abi.md
|
|
vfs.md
|
|
debugging.md
|
|
transport.md
|
|
security.md
|
|
operator-panel.md
|
|
windows-backend.md
|
|
```
|
|
|
|
### 20.3 Error messages as docs
|
|
|
|
Invest heavily in errors:
|
|
|
|
```text
|
|
missing env
|
|
node unavailable
|
|
environment incompatible
|
|
large task arg
|
|
host-only type in DisasmerArg
|
|
cmd used on coordinator
|
|
cmd capability missing
|
|
artifact bytes unavailable
|
|
QUIC direct connection failed
|
|
sync required for export
|
|
Debug Epoch freeze failed
|
|
restart incompatible after source change
|
|
community tier quota exceeded
|
|
```
|
|
|
|
---
|
|
|
|
## 21. Key risks and mitigations
|
|
|
|
### 21.1 Risk: debugger is the hardest part
|
|
|
|
Mitigation:
|
|
|
|
- Build DAP skeleton early.
|
|
- Implement Debug Epochs before fancy UI.
|
|
- Use generated debug probes for reliable breakpoints first.
|
|
- Add DWARF/local richness incrementally.
|
|
- Define launch acceptance around real breakpoints in the demo code, thread list, task args, handles, stacks, all-stop, continue, and restart.
|
|
|
|
Experiment result: `experiments/debugger-gate` has derisked the core VS Code debugging UX. It proves a custom DAP adapter can present a virtual process with `main` and task threads, bind ordinary VS Code gutter breakpoints to Wasmtime-backed Rust/Wasm source locations, hit breakpoints in multiple virtual tasks, support basic stepping/watch/variables, and keep the backend boundary shaped like a replaceable virtual-process runtime. Treat that experiment as a proof and reference, not as implementation foundation: it is intentionally hacky prototype code with hardcoded example-local variable mapping and a simplified local Wasmtime scheduler.
|
|
|
|
### 21.2 Risk: Windows isolation is not real yet
|
|
|
|
Mitigation:
|
|
|
|
- Keep Windows as user-attached node execution only.
|
|
- Label backend as `windows-command-dev`.
|
|
- Do not offer public managed Windows compute in MVP.
|
|
- Keep sandbox backend interface ready.
|
|
- Investigate Windows Sandbox/AppContainer after public launch or before managed nodes.
|
|
|
|
### 21.3 Risk: hosted community tier becomes compute abuse target
|
|
|
|
Mitigation:
|
|
|
|
- No community hosted containers.
|
|
- No community hosted native commands.
|
|
- Zero-cap hosted Wasm control loop only.
|
|
- Fuel, wall-clock, memory, logs, and metadata limits.
|
|
- Rate-limit every API and event stream.
|
|
- Require attached user node for real work.
|
|
|
|
### 21.4 Risk: QUIC/NAT becomes launch blocker
|
|
|
|
Mitigation:
|
|
|
|
- Use coordinator control session for scheduling and metadata.
|
|
- Require direct node-to-node only when bytes must move.
|
|
- Prefer scheduling where bytes already are.
|
|
- Fail clearly when direct transfer is impossible.
|
|
- For launch demo, place Linux and Windows nodes in a connectivity setup you control.
|
|
|
|
### 21.5 Risk: local-first goal gets lost
|
|
|
|
Mitigation:
|
|
|
|
- Add tests that fail on unwanted uploads.
|
|
- Instrument bytes moved per build.
|
|
- Show “bytes moved” in logs/dashboard.
|
|
- Make `flush()` metadata-only by default from day one.
|
|
|
|
### 21.6 Risk: product sounds too ambitious
|
|
|
|
Mitigation:
|
|
|
|
- Demo one narrow workflow extremely well.
|
|
- Keep the public README focused on build systems.
|
|
- Keep architecture docs honest about non-goals.
|
|
- Do not claim managed secure compute until it exists.
|
|
- Emphasize open-source local-first runtime.
|
|
|
|
---
|
|
|
|
## 22. Definition of done for the public-launch MVP
|
|
|
|
The public-launch MVP is done when a clean machine can:
|
|
|
|
```text
|
|
1. Install Disasmer CLI.
|
|
2. Install the VS Code extension.
|
|
3. Sign in to hosted coordinator through Authentik/OIDC.
|
|
4. Create or select a project.
|
|
5. Attach a Linux node.
|
|
6. Attach a Windows node.
|
|
7. Clone the launch-build demo repo.
|
|
8. Open it in VS Code.
|
|
9. See env!("linux") and env!("windows") recognized.
|
|
10. Press F5.
|
|
11. See one virtual process start.
|
|
12. See Linux and Windows virtual threads in VS Code.
|
|
13. Hit a breakpoint in a spawned task.
|
|
14. Observe all-stop Debug Epoch behavior.
|
|
15. Inspect task args, command status, logs, and artifact handles.
|
|
16. Edit source and restart only the failed/selected task.
|
|
17. Finish the build.
|
|
18. Export artifacts through an attached/local receiver node or explicit storage integration.
|
|
19. Confirm local-first behavior in logs/metrics.
|
|
20. Confirm hosted coordinator did not run arbitrary native commands.
|
|
```
|
|
|
|
---
|
|
|
|
## 23. Suggested implementation order inside each milestone
|
|
|
|
When in doubt, build vertical slices rather than broad infrastructure.
|
|
|
|
Recommended pattern:
|
|
|
|
```text
|
|
1. Fake backend, real API.
|
|
2. Real backend, local only.
|
|
3. Real backend, coordinator split.
|
|
4. Real backend, hosted auth/quotas.
|
|
5. Polish only after the vertical slice proves the thesis.
|
|
```
|
|
|
|
Example for debugging:
|
|
|
|
```text
|
|
1. DAP shows fake virtual process with fake threads.
|
|
2. DAP attaches to local one-node Wasm process.
|
|
3. Breakpoint probe freezes one Wasm task.
|
|
4. Debug Epoch freezes two Wasm tasks.
|
|
5. Debug Epoch freezes Wasm + command task.
|
|
6. Same flow across two nodes.
|
|
7. VS Code restart selected task.
|
|
8. Richer locals/stacks.
|
|
```
|
|
|
|
Example for VFS:
|
|
|
|
```text
|
|
1. In-memory manifest.
|
|
2. Local disk overlay.
|
|
3. Artifact handle.
|
|
4. Metadata-only flush.
|
|
5. Export sync.
|
|
6. Node-to-node chunk fetch.
|
|
7. Explicit receiver-node export.
|
|
```
|
|
|
|
---
|
|
|
|
## 24. External technical references to keep nearby
|
|
|
|
These are useful implementation references, not product dependencies:
|
|
|
|
- Wasmtime Rust embedding API: https://docs.wasmtime.dev/api/wasmtime/
|
|
- Wasmtime Component Model API: https://docs.wasmtime.dev/api/wasmtime/component/index.html
|
|
- Quinn repository and docs: https://github.com/quinn-rs/quinn
|
|
- Debug Adapter Protocol: https://microsoft.github.io/debug-adapter-protocol/
|
|
- VS Code debugger extension guide: https://code.visualstudio.com/api/extension-guides/debugger-extension
|
|
- Authentik OAuth2/OIDC provider docs: https://docs.goauthentik.io/add-secure-apps/providers/oauth2/
|
|
- Podman: https://podman.io/
|
|
- Nix flakes: https://nix.dev/concepts/flakes.html
|
|
- Windows Sandbox CLI: https://learn.microsoft.com/en-us/windows/security/application-security/application-isolation/windows-sandbox/windows-sandbox-cli
|
|
- Windows AppContainer isolation: https://learn.microsoft.com/en-us/windows/win32/secauthz/appcontainer-isolation
|
|
- OWASP API Security Top 10 2023, unrestricted resource consumption: https://owasp.org/API-Security/editions/2023/en/0xa4-unrestricted-resource-consumption/
|
|
- OAuth 2.0 Security Best Current Practice, RFC 9700: https://datatracker.ietf.org/doc/rfc9700/
|
|
- NIST SP 800-190, Application Container Security Guide: https://csrc.nist.gov/pubs/sp/800/190/final
|
|
|
|
---
|
|
|
|
## 25. Final guidance
|
|
|
|
Do not shrink the ambition of the demo, but aggressively constrain the implementation semantics.
|
|
|
|
The MVP should be ambitious in what the user sees:
|
|
|
|
```text
|
|
one Rust build program
|
|
hosted coordinator
|
|
Linux + Windows nodes
|
|
source-level VS Code debugging
|
|
all-stop Debug Epochs
|
|
task restart
|
|
local-first artifacts
|
|
```
|
|
|
|
It should be conservative in what the runtime promises internally:
|
|
|
|
```text
|
|
no transparent raw remote pointers
|
|
no live stack migration
|
|
no coordinator bulk data plane
|
|
no community hosted native compute
|
|
no fake Windows sandbox claims
|
|
no arbitrary child-process source debugging
|
|
no automatic survival of non-replicated node failure
|
|
```
|
|
|
|
That combination is what can make the launch credible: the visible experience is striking, while the implementation model remains honest enough to actually build.
|