Clusterflux

Clusterflux runs a Rust-defined workflow as one distributed virtual process. The async main runs serverless, provisioning nodes to run tasks through rootless containers. Tasks on nodes exchange data simply and efficiently.

The user experience is built to be as much as possible like building a regular program. The processes are debuggable as normal through a debugger adapter.

The primary use case is to consolidate build processes into a single streamlined developer experience. An example program can be seen below:

use clusterflux::prelude::*;

#[clusterflux::task(capabilities = "command")]
pub async fn compile(source: SourceSnapshot) -> Result<Artifact> {
    let executable = fs::output("hello-clusterflux")?;
    Command::new("cc")
        .args([
            "-Os",
            "-static",
            "-s",
            "fixture/hello-clusterflux.c",
            "-o",
            executable.as_str(),
        ])
        .cwd(source.mount()?)
        .env("SOURCE_DATE_EPOCH", "0")
        .network_disabled()
        .run()
        .await?;
    fs::publish(&executable).await
}

#[clusterflux::main]
pub async fn build() -> Result<Artifact> {
    let source = source::current_project().snapshot().await?;
    let compile = clusterflux::spawn!(compile(source))
        .on(clusterflux::env!("linux"))
        .await?;
    compile.join().await
}

After setup, this build pipeline could be deployed as easily as launching it through your IDE. This repository includes a VS Code extension to make development as straightforward as possible. A full collection of CLI tools is included for advanced usage.

Clusterflux is explicitly local-first. It is trivial to provision existing hardware as resources. Bulk data will typically not leave the local network, allowing maximum throughput. The same capability, however, also makes it possible to leverage cloud resources easily.

Start with Getting started. It takes you through authentication, project setup, node enrollment, a run, debugging, task restart, and artifact download.

Build a real executable

The primary example is intentionally small: hello-build snapshots its source project, spawns one container-backed compile task, and publishes the resulting executable as a retained artifact.

clusterflux bundle inspect --project examples/hello-build
clusterflux run --project examples/hello-build build
clusterflux artifact list --process <process-id>
clusterflux artifact download <artifact-id> --to ./hello-clusterflux
chmod +x ./hello-clusterflux
./hello-clusterflux

The final command prints hello from a real Clusterflux build. The source uses only the public SDK path: current-project snapshot, spawn!, Command::run, and artifact publication. See recovery-build for two same-definition task instances, a real command failure, and operator restart.

What you get

• One virtual process with distinct task instances and restart attempts.
• Bundle-declared environments resolved by digest.
• Native work only on nodes you attach.
• Metadata-first artifacts whose bytes remain on retaining nodes by default.
• VS Code debugging backed by coordinator task and attempt snapshots.
• Full and partial Debug Epochs with explicit consistency status.
• Human Authentik sessions plus scoped public-key identities for agents and nodes.
• A public coordinator, node runtime, CLI, SDK, and DAP adapter for self-hosting.

Install from this checkout

cargo install --path crates/clusterflux-cli --bin clusterflux
cargo install --path crates/clusterflux-node --bin clusterflux-node
cargo install --path crates/clusterflux-coordinator --bin clusterflux-coordinator
cargo install --path crates/clusterflux-dap --bin clusterflux-debug-dap

On NixOS or another system with Nix, the equivalent package is available with nix profile install .#clusterflux-tools.

Rootless Podman is required on Linux nodes that build or run a declared Containerfile environment. Install VS Code when you want the graphical debug workflow.

First run

For the hosted service:

clusterflux login --browser
clusterflux auth status
clusterflux project list
clusterflux node enroll --project-id <hosted-project-id> --json
clusterflux node attach --project-id <hosted-project-id> --node workstation \
  --enrollment-grant "$ENROLLMENT_GRANT"

The attach command creates and stores a local node key by default, then exchanges the short-lived grant once. Start clusterflux-node --worker from the project directory. See Nodes for the complete sequence and the advanced explicit-key option.

Choose an entrypoint and run it:

clusterflux bundle inspect --project examples/hello-build
clusterflux run --project examples/hello-build build

Inspect the result:

clusterflux process status
clusterflux task list
clusterflux logs
clusterflux artifact list
# Use the `artifact` value returned by the list command, for example:
clusterflux artifact download hello-clusterflux-4f61c2... --to ./hello-clusterflux

To run your own coordinator instead, follow Self-hosting. The hosted website is not required for self-hosted projects.

Documentation

• Getting started
• Architecture
• Nodes
• Environments
• Artifacts
• Debugging
• Task ABI
• Self-hosting
• Security model
• Security reporting