CLI reference

Install ax first: see Install ax. Use ax --help or ax <command> --help for exact options. Start with your first remote job for setup and a complete example.

The local connection broker keeps authenticated connections warm between calls. It starts automatically when a command needs a connection.

Submit and recover jobs

For a configured runner, use its saved name and keep the submission receipt:

ax exec run --node build-host --cwd /srv/project --format json -- bash scripts/check.sh
ax exec status "$job_id"
ax exec output "$job_id"
ax exec wait "$job_id"
ax exec cancel "$job_id"

• Submission returns request_id, node_id, node_name, target_id and job_id immediately. node_name is null when an ID was supplied.
• Follow-up commands resolve unique jobs from the selected caller's local journal across workspaces. Use --node and/or --target to disambiguate; supply both for an unrecorded job.
• run --follow follows output until terminal; wait is an alias for watch.
• Without a format option, stdout and stderr carry the remote command's bytes — no receipt, no routine state messages. Failures keep their exit status and diagnostics.
• Use --format json for submission and state events, or ax exec ls to recover a job after interrupting a raw follow.

Return on output silence — --output-idle on run, watch or wait returns after an interval without new stdout or stderr. It implies following on run:

ax exec run --node build-host --output-idle 2m -- bash scripts/check.sh
ax exec wait "$job_id" --cursor 42 --output-idle 30s --format json

• Durations: positive whole numbers with ms, s, m or h; a bare number means seconds.

• Retained history is drained before the idle window starts. New bytes on either stream reset it, including partial lines; state changes do not.

• At the deadline, one fresh output/status snapshot checks for completion, output still in flight and capture truncation (two broker RPCs, only when the deadline is reached). Connection and authorization failures remain errors.

• An idle return leaves the job running and exits zero: it says why this wait ended, not whether the command succeeded.

• Raw mode reports job ID, state and next cursor on stderr. Structured mode adds a wait_return event with return_reason (output_idle or terminal), job, state and next_cursor. Resume the same job and cursor to wait again.

• --timeout-ms separately limits remote execution; --output-idle never cancels it.

• Linux runners with the push-watch adapter send output and state changes as they happen over the existing connection; older runners poll. After upgrading the client, run ax exec daemon restart.

• A brief connection loss resumes the same job from the last received output cursor. If recovery fails, use ax exec watch "$job_id" — do not resubmit. Cancelling a local wait leaves the remote job running.

• Only a recorded terminal state and complete retained output establish completion; an explicit output-idle return establishes neither completion nor a hang.

• Before contacting the target, ax commits the request ID, exact request and local working directory to its journal. Detached and structured submissions also print the request ID before sending; raw followed submissions report it on a submission error instead.

• After an uncertain submission, repeat the exact command with its original --request-id; changed arguments are rejected. An already saved acceptance returns the same job reference locally.

• An unresolved request may be sent again only within the original server-advertised request-retention window, with a safety margin. After that, reconcile the original execution before starting more work.

Send stdin

ax exec run --node build-host --stdin --follow -- bash -s < scripts/check.sh
printf 'hello\n' | ax exec run --node build-host --stdin --follow -- /bin/cat

• --stdin reads to EOF before submission. Input is limited to 1 MiB and the runner's advertised max_stdin_bytes; an oversized input fails before submission.
• The bytes become part of the saved immutable request; exact retries must provide the same input.
• Without --stdin: nonempty redirected input is rejected, as is an open pipe without available input. Closed empty pipes and /dev/null are accepted without waiting. Use </dev/null when a harness intends no input.
• There is no interactive stdin stream after submission.

List local jobs

ax exec ls
ax exec list
ax exec ls -a
ax exec ls --all --refresh
ax exec ls --json

• Default scope: the exact canonical local submission directory — not the remote --cwd or its subdirectories. -a / --all includes every local directory for the selected profile's caller node. Another caller's jobs are never included.
• The index covers submissions through this ax CLI and its MCP submit tool; it does not enumerate unrecorded jobs submitted by other software.
• Rows include retained completed jobs and unconfirmed submissions, newest first, with the host, command, recorded state and age.
• Status is cached by default; status/wait observations update the cache. --refresh requests current status within a ten-second total network budget and keeps cached rows when a host is unavailable. JSON includes observation timestamps and refresh errors.

Inspect a node

ax exec info "$node_id"
ax exec alias build-host "$node_id"
ax exec info build-host
ax exec info build-host --format json

• info reads the node's authorized targets and capabilities: node ID, local alias when used, target IDs, OS/architecture, execution account, environment, profiles and their guarantees, adapters, limits, retention and backend readiness. Human output uses named profile details, readable byte sizes and durations.

• It also snapshots the broker's actual RPC connection path: direct UDP address or relay URL, connection state and RTT. A relay does not expose the remote peer's IP address.

• This is execution capability information, not a hardware inventory. It never launches a job. Individual target capability failures are included in the result; failure to discover the node's targets returns an error. The complete lookup has a fifteen-second budget.

• Aliases are local to the selected operator profile and also work wherever a command accepts --node. Saving an alias does not enroll a caller, grant access, choose a target or change network addresses. An alias already naming a different node is rejected. SSH deployment saves the alias and connection configuration together.

List saved nodes

ax exec nodes
ax exec nodes --refresh
ax exec nodes --format csv

• Lists the selected profile's saved aliases and nodes with configured direct addresses. Multiple aliases for the same node share one row.

• Default is a local read: it does not dial nodes or generate an identity. It reads current connection snapshots from an already running broker, and the last persisted info, check or node-refresh observation for this profile.

• Cached results show freshness: cached, observed_at_ms and observation_age_ms; the human readiness column shows their age. A node without an observation remains unchecked with readiness unknown. No broker means no pooled connections; an unresponsive broker is reported as unknown, not offline.

• --refresh queries authorized targets and capabilities: ten-second total network budget, at most three seconds per node.

• reachable means the target query succeeded; ready means at least one authorized target reports a ready backend. Denials, unavailable connections and other errors are distinguished.

• Partial capability failures leave readiness unknown unless another target is ready. Nodes skipped after the budget expires keep any cached observation and are not labelled freshly checked. Failed checks replace earlier readiness with an unknown result.

• JSON/CSV/TSV include target counts, observation timestamps, freshness and errors.

• A single-target check records checked_target_id: it can prove that target ready, but not whole-node target counts or that every other target is unavailable. Cached observations never grant access.

• The human table abbreviates node IDs and labels the target probe result CHECK; structured output retains full IDs. Node rows add connection_state, transport, remote_address, relay_url, rtt_ms, connection_observed_at_ms and connection_error (info groups these under connection).

• A connection snapshot records the current transport state; --refresh checks whether the node answers now. Paths can change after a snapshot.

• This is a list of known nodes, not network discovery or proof of permission to execute. Use ax exec info <ID-or-name> for target details.

Structured output

All commands that return structured results support the global --format json|csv|tsv, before or after subcommands. --json is a shorthand for --format json; select one format option at a time.

ax exec info build-host --format json
ax exec ls --all --format csv
ax exec targets --node build-host --format tsv
ax profile list --format json

• JSON preserves each command's result shape.

• CSV and TSV include a header row: top-level arrays produce one row per item; a top-level object produces one row. Nested objects and arrays remain JSON inside properly quoted cells, so commas, tabs, quotes and newlines survive. info's targets column carries its full target/capability array. Empty job lists still emit their column headers.

• Scalars use their JSON text, except strings are written as plain cell text; an absent field is empty, null is written as null.

• wait, watch and run --follow stream NDJSON when JSON is selected. CSV/TSV streams use one event,data header (tab-separated for TSV), with the complete event as JSON in data.

• A followed submission starts with a submitted event; output bytes retain Exec's base64 encoding in structured modes. Without a format option, following output keeps raw stdout/stderr behavior.

• Diagnostics stay on stderr. Shell completions and the MCP protocol keep their own wire formats.

Metrics, tracing and operational logs

• The ax CLI/broker and embedded execution host enable Aster's otel instrumentation and shared aster-otel pipeline. Standard RPC metrics and spans use that pipeline; Exec service and broker operational events use structured tracing logs. The durable audit outbox remains responsible for audit acceptance and delivery. Remote stdout/stderr stays retained job output, outside operational logs.

Export is opt-in. For an OTLP HTTP/protobuf collector:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
ax exec info build-host

• Set the same variables on the runner's service environment to collect server telemetry.
• Default service name: ax; OTEL_SERVICE_NAME overrides it.
• A running local broker inherits settings when it starts; restart it after changing them. It keeps writing operational stderr to its protected daemon.log.
• ASTER_LOG sets tracing filter directives; ASTER_LOG_STDERR=0 disables the stderr sink; OTEL_SDK_DISABLED=true disables the OTel pipeline.
• Collector exporters default to none, so ordinary local commands require no collector. No additional metrics HTTP listener is opened. The pipeline initializes before Tokio starts and flushes and shuts down after it stops.

Local identity and storage

• ax defaults to ~/.ax; --home / AX_HOME overrides it. --profile / AX_PROFILE selects an operator profile. An existing Aster identity is not imported automatically.

• The copied Aster keystore uses the ax keychain service by default, macOS Keychain or Windows Credential Manager where supported, and protected root-key files on headless hosts. AX_KEYSTORE=os|file selects the backend; AX_KEYRING_SERVICE overrides the service namespace. Node keys and enrollment credential files use Aster's existing protected-file mechanisms.

• The caller journal is local, with a private directory and files. It can contain sensitive arguments and action chains and must not be synchronized as job history.

• Execution history uses a Turso database at ~/.ax/exec-jobs/<profile>/jobs.sqlite3. The connection broker keeps it open; when the broker is stopped, local history commands can open it briefly. Requests are committed before submission; SQL indexes cover caller, workspace and job lookup. Full-text indexing is not enabled yet.

• Upgrading starts a fresh execution history; earlier job records are not imported. A job missing from local history can still be addressed with its job ID, node and target. Node observations and deployment provenance keep their existing storage.

Credential scopes before registration

• A host can opt in with credential_scopes: true in its local ceiling (default: false in core Exec). Ax deployment enables it for new runners and preserves it on existing ones; ax deploy ssh HOST --credential-scopes true|false selects an explicit change.

• With it, the embedded Exec service can authorize a directly enrolled caller before it has an Exec grant, using the exact aster.exec.discover, aster.exec.submit, aster.exec.status, aster.exec.output, aster.exec.watch and aster.exec.cancel scopes in its current verified admission evidence.

• Each scope permits only its named operation; aster.exec carries all six, within the host's configured local ceiling. The generic exec scope does not grant execution.

• Root-authored grant policy always takes precedence. Revoked, expired, deleted or unavailable grants never fall back to scopes.

• Registration claims replace the credential's claims, so promotion before the Exec grant arrives preserves scoped access; a registration that narrows the scopes takes effect on the next call.

• Deployment configures the runner's issuer and observation access. ax enroll invite --node NAME creates a persistent agent invite. Owner review and promotion use the explicit enrollment commands; see runner invites.

Runner setup

• Use ax deploy ssh HOST --name NAME to install and configure a runner. It saves exec run target/cwd defaults; explicit --target and --cwd override them. User installation is the default; add --system for machine-wide Linux setup.
• The connection broker requires Unix; the supervised execution backend uses Linux/systemd or macOS/launchd. See where jobs run and the SSH deployment reference for configuration paths.
• The key, profile, enroll, trust and grant commands support explicit operator provisioning. ax exec setup checks Linux installation paths; ax serve --config <FILE> starts the runner API.

Harness provenance

CLI and MCP submissions can record harness, version, session and local workspace alongside each job. Exact retries preserve the original origin. See harness integration for flags, environment variables, per-call MCP context and deployment history.

Workspace files

ax exposes ax workspace setup, push, pull, status and ax exec run --workspace. See Workspace file transfer for selection, preview, verification and retry semantics. --local-workspace continues to record harness provenance and never enables file transfer.