clusterflux-public/latency_notes.md

Disasmer Latency Notes

Low latency is a core design goal. Disasmer should avoid copying or sending data unless it is required.

Core Rule

Move metadata eagerly.
Move bytes lazily.
Run work where the bytes already are.

The coordinator should track locations, hashes, ownership, and versions. It should not sit in the middle of large file transfers.

Local-First Execution

For a simple build, such as building a Linux binary from a Git repo that already exists locally, Disasmer should behave almost like a local build:

/src    -> bind mount existing checkout
/work   -> local writable scratch space
/cache  -> local persistent dependency cache
/out    -> local artifact/output directory

No repo upload, no tarballing, no coordinator-routed file transfer, and no automatic artifact upload.

Placement Policy

Task placement should prioritize data locality:

prefer node with source checkout
prefer node with container/Nix environment cached
prefer node with dependency/build cache
prefer node with previous outputs needed by this task
avoid expected network transfer

If the local machine can run the task and already has the repo, it should usually win placement.

VFS Semantics

The virtual filesystem should be copy-on-write and metadata-driven.

flush()
  Publishes filesystem metadata and artifact references.
  Makes outputs visible to Disasmer.
  Does not necessarily upload bytes.

sync()
  Makes selected outputs durable or replicated.
  May upload or transfer bytes.

This distinction is important: flush() should be cheap; sync() may be expensive.

Git Inputs

A Git repo should be treated as a source provider, not as a directory to blindly copy.

A source snapshot is:

commit hash
submodule state
dirty-file overlay
included input files
ignored-file policy

If the task runs locally, /src can be a direct bind mount. If it runs remotely, only missing or changed content should be sent.

Artifacts

Publishing a large output should register metadata first:

artifact id
path
node
size
digest/version

The actual bytes move only when needed:

another node consumes the artifact
user exports/downloads it
sync() requires durability
replication policy requires it

Same-node reuse should use bind mounts, hardlinks, reflinks, or copy-on-write where possible.

Environments

Containerfiles, Dockerfiles, and Nix flakes should be part of the Disasmer bundle as environment definitions.

Resolved container layers, built images, Nix store paths, and dependency caches should live in node-local caches and be reused by digest.

Coordinator Responsibility

The coordinator should answer questions like:

which node has source snapshot X?
which node has artifact Y?
which node has environment digest Z cached?
where should this task run?

It should not handle normal compiler file reads, linker writes, cache reads, or large artifact transfers.

MVP Defaults

Disasmer should default to:

local-first task placement
lazy byte transfer
metadata-only flush
explicit sync for durability
node-to-node bulk transfer
content-addressed storage
copy-on-write VFS overlays
no coordinator-routed bulk data

Summary

For low latency, Disasmer should make the common local case stay local.

A local Linux build from a local Git repo should add only small runtime overhead:

placement decision
debug/session metadata
VFS epoch metadata
artifact registration

It should not add unnecessary source copying, artifact uploading, image rebuilding, or network filesystem traffic.