Install ax over SSH

Deployment uses your existing OpenSSH connection to install ax on the runner. It detects the remote platform, picks an artifact, and registers ax's execution-host service.

ax deploy ssh build-host --plan
ax deploy ssh user@host
ax deploy ssh user@host --system

• Default: user installation. --system installs machine-wide on Linux via a root SSH account or noninteractive sudo.
• Authentication: SSH keys/agent by default; --interactive permits terminal prompts. Host-key checking follows your OpenSSH configuration and is never disabled.
• SSH aliases work directly. Overrides: -p PORT, -i KEY, -J JUMP, -F CONFIG.

Linux and macOS user installations include runner setup:

ax deploy ssh user@build-host --name build
ax exec run --node build -- /usr/bin/id -u

Deployment configures the service, authorizes the deploying caller, saves the runner name and defaults, and checks a diagnostic job.

• Jobs run as the SSH account. A root SSH login must choose a non-root account with --run-as ACCOUNT. --system grants no implicit root payload access.
• --install-only transfers and registers without provisioning — for a host you maintain manually.
• User mode runs service and jobs as the SSH account. System mode separates the service account from the job account.

Progress and output

• Human output shows the current installation step, and reports a step still running after 15 seconds (reminders every 30). --quiet hides progress.
• JSON, CSV and TSV keep progress hidden and return the final structured result.
• --timeout bounds each SSH probe or installation operation (default 120 s).
• --plan probes platform and artifact selection but installs nothing and creates no credentials. It does not validate every provisioning prerequisite.

Artifacts and verification

• Without --bundle, the installer transfers its own executable when OS, architecture and ABI match.
• For another published platform, a signed release CLI downloads and verifies its own release version on the source machine, then transfers it over SSH. The runner needs no access to the download site; only the requested platform is downloaded.
• Verified entries under AX_HOME/cache/deploy work offline; each reuse re-verifies signature and executable. --plan can populate this cache without installing.

The source executable must match its own entry in that release's signed manifest. Unpublished versions and unsupported platforms require --bundle; deployment never substitutes the latest version.

To prepare an offline bundle, install ax on a machine matching the target platform and package that binary:

ax deploy bundle --out ./ax-linux-bundle
# Transfer this bundle to the deploying machine, then:
ax deploy ssh build-host --bundle ./ax-linux-bundle/manifest.json

• A bundle is an executable plus a JSON manifest of OS, architecture, ABI, version, size and SHA-256. It is an operator-trusted local bundle, not a signed release.
• Local checks: manifest bytes, paths inside the bundle directory, native format and CPU. The host verifies size, SHA-256, ax --version and the deployment protocol before activation.
• Windows uses a short encoded PowerShell loader with the installer framed on stdin (avoids cmd.exe command-line limits); no PTY for binary transfer.
• CPU names normalize to x86_64 and aarch64; detection covers Apple Silicon under Rosetta and Windows-on-ARM.
• Linux GNU artifacts need a glibc host; musl-static artifacts must have no dynamic ELF interpreter. The staged binary is run to catch loader/glibc/CRT problems before the launcher changes. Universal Mach-O is rejected; provide one build per architecture.

Installation locations and upgrades

• Add the launcher directory to PATH where needed; startup files are never edited. Windows uses a launcher plus an atomic release pointer, so upgrades work while the old executable is open.

• User Linux/macOS: existing home, installation and service ancestor directories must be real directories owned by the SSH account, without group or other write permission (~/.local/bin may be group-writable). An error names the path and the required correction. Managed ax and versions directories are still made private automatically.

• Installations are serialized per destination directory; bytes are staged and verified before the launcher or release pointer changes. Repeating an identical installation reconciles the existing release.

• Old versions stay for running processes and manual rollback; there is no automatic pruning or rollback command yet. Identities, credentials and job databases are preserved.

• Runner provisioning restarts the API when it installs a changed issuer delegation; job services stay separate. macOS also reloads the API for a changed host service definition; Linux --install-only leaves a running service on its previous binary until an explicit restart or reboot. Neither path restarts running jobs.

• An unmanaged launcher is refused by default. --replace explicitly permits backup and replacement of a regular executable/launcher or supported symlink; directories and unsafe managed paths are refused. Windows: relocate an existing unmanaged ax.exe manually. After a lost connection, rerun the same deployment to reconcile activation. A POSIX lock left by an external SIGKILL needs inspection before removal; the installer does not guess which lock is stale.

Linux boot services and readiness

Both modes register ax.service (ax serve --config <FILE>) — the execution host, not the caller connection broker. Its API, privileged helper and job-supervisor roles share the ax executable. Per-job transient units use the ax-job- prefix.

• User boot startup needs a working user systemd manager and lingering. The installer enables lingering using existing local policy, with no implicit sudo; if an administrator is needed, the result reports that action and boot_enabled: false.

• Jobs use separate transient user services, so restarting the API does not stop them.

• User mode trusts other processes running as that account: it does not protect user-owned credentials or state from them. Private configuration: ~/.config/ax; state: ~/.local/state/ax; supervisor launcher: ~/.local/share/ax/supervisor.

• System mode additionally registers ax-helper.service and a fixed ax-supervisor role launcher. Paths: /etc/ax/helper.json, /run/ax, /var/lib/ax-helper; API state: /var/lib/ax.

• The API unit runs as ax-service-user, created by provisioning; it must have its expected home, noninteractive shell and no extra groups. The execution account is the SSH user or explicit --run-as account, and is not created automatically.

• Repeat runs check helper binding, configuration, state ownership and the node key; unexpected changes are refused rather than replacing identities or owner choices.

• Units stay conditional on their configuration file being present. Registration and enablement are reported separately from configuration and execution readiness.

• Bootstrap reports exec_ready: true only after authenticated capabilities and a /usr/bin/id -u job whose retained stdout matches the selected payload UID. Its request ID is persisted before submission, so an ambiguous attempt is reconciled with that same ID.

• A reported known terminal outcome permits a fresh diagnostic on a later deployment; a lost outcome requires reconciliation. Diagnostic jobs appear in local job history with harness origin. --install-only responses retain exec_ready: false.

• Ordinary provisioning checks: Aster enrollment, policy synchronization, an authenticated caller RPC, and a diagnostic job with retained output. runner.connectivity records the caller's selected RPC path, address or relay URL, RTT and observation time. A working relay is valid connectivity; a direct path is not required.

• If installation succeeds but an Aster check fails, the command fails and reports the stage (aster_enrollment, aster_policy_sync, aster_rpc), ssh_succeeded: true, installed: true, exec_ready: false. These details are also kept in deployment history.

• SSH can work while UDP or relay traffic is blocked. Network errors suggest checking firewall rules, relay access, trust and runner logs; an Exec authorization denial remains a denial. The installation is retained for diagnosis and a safe retry. --plan and --install-only skip these readiness checks.

• The check uses the established Aster connection; the caller needs no separate inbound listening port. Reconnecting reloads the profile's saved direct addresses, so a runner restart can publish a new UDP port without restarting the client broker.

Deployed runners also serve persistent agent invites; see runner invites. The open execution profile inherits the payload account's access and is not a filesystem/network sandbox.

macOS user service

Deployment installs ~/Library/LaunchAgents/dev.getaster.ax.host.plist and starts dev.getaster.ax.host in the user's launchd domain as a Background session — the SSH connection does not own the host process. Startup is per user login, not a pre-login root service. --system remains Linux-only.

launchctl print user/$(id -u)/dev.getaster.ax.host

• Logs: ~/.local/state/ax/host.stdout.log and host.stderr.log. Configuration and identity paths match the Linux user installation.
• Every job has its own launchd supervisor, so restarting the host API does not stop running jobs. Status, output and the final result remain available after reconnecting.
• The open profile inherits the account's filesystem, network and sudo permissions. Cancellation targets the job's process group; a process that deliberately creates another session can escape that group. This is not a sandbox or Linux cgroup containment.
• Unsupported CPU, memory, process-count and file-size limits are rejected, as is an explicit allow_sudo: false. Execution deadlines and retained-output limits remain supported.

Standalone trust, state and renewal

• Bootstrap reuses or creates the selected profile's root through ax's Aster keystore, persistent caller key and enrollment credential file. An existing unrelated policy namespace is not adopted automatically; use a fresh profile for this workflow.
• The host's private node key stays on the host. Signed Exec grants are published using Aster Docs, and the host holds only a read capability for that policy.
• Retained policy supports startup while the deploying caller is offline; policy updates and revocations still depend on synchronization.
• Bootstrap reconciles the target's current root grant before publishing, and refuses a revocation or owner-policy conflict. It does not re-author other hosts' cached grants.

Deployment also provisions a bounded issuer delegation and observation-write access for the runner. Only the saved deploying caller can request an invite:

ax enroll invite --node build-host --format json

• The ticket is a bearer secret until spent or expired. Issuance is explicit; deployment creates no invite.

• The agent profile issues persistent credentials with aster.exec, using a single-use invite valid for 15 minutes. Exec scope fallback is enabled for newly provisioned runners, within the local ceiling. Existing runners retain their setting; --credential-scopes true|false changes it explicitly. Observation review and root promotion are separate owner actions.

• Caller and host credentials start with a seven-day lifetime. Redeployment issues a fresh host credential and refreshes the managed caller credential when it has less than two days remaining.

• Protected files are atomically replaced; live processes use Aster's verified credential-replacement API. A changed issuer delegation requires an API restart. Other owner-selected credential files are not replaced automatically. The result reports credential and delegation expiry times and the observations namespace.

• The runner can renew credentials it issues until root registration takes over; the root must still replace its delegation before expiry. Removing an SSH key or purging an installation does not revoke issued ax credentials or Exec grants.

• In open, normal OS permissions decide which directories a job can use. The supervisor switches to the job account before entering cwd; the API does not resolve private directories through root. Old directory-restricted policies require explicit migration instead of silently gaining wider access.

• The job gets its account's HOME/USER/LOGNAME/SHELL and a predictable PATH. It does not inherit the deployment environment or SSH credentials. CLI exec run uses saved target/cwd defaults, with explicit overrides; MCP still supplies both fields.

• Jobs have no deadline by default. --timeout-ms requests one, subject to any configured owner/grant maximum. New configurations use max_timeout_ms: 0 for no maximum. No separate fixed job lifetime is imposed.

• Bootstrap protocol 4 requires a current bundle so an older runner cannot be mistaken for this behavior. Existing owner deadline limits remain in effect on redeployment.

• Durable job records and retained output are always required. A separate audit journal is optional: generated configurations have neither a local journal nor an HTTP collector; owners can enable the existing local or collector-backed fail-closed audit mode. Output retention stays 24 hours and request retention seven days, separately from audit rotation and running job lifetime.

• Redeployment preserves owner limits and other service settings while checking pinned trust, identity, account and backend paths. It does not automatically migrate between accounts or accept a replacement identity after purge. Local aliases and pins survive undeployment. Owners who change backend resource limits in system mode must keep helper and API settings consistent.

• Restricted cwd scopes in delegated action chains are rejected in open too. Use target/profile scopes or bind the exact request digest instead; neither is a filesystem sandbox. The harness schema permits deadlines above 24 hours.

WSL runners

A WSL runner executes Linux commands. Native PowerShell execution requires a native Windows backend, which is not available yet.

For a Windows OpenSSH endpoint that dispatches commands into WSL Bash, select the transport explicitly:

ax deploy ssh wsl-host --wsl-dispatcher --name wsl-build --bundle ./ax-linux-bundle/manifest.json
ax undeploy ssh wsl-host --wsl-dispatcher --plan

• --wsl-dispatcher uses a constant-size SSH command: a Bash loader and a length-framed encoded command on stdin, followed by any binary payload. It applies to probes, installation, provisioning and removal.
• A failed WSL probe is an error; it never falls back to native Windows installation. Use ordinary deployment for a Linux sshd running inside WSL.

Windows must also keep the distribution alive after SSH disconnects. Run the supplied script as the Windows user who owns the distribution, from an ax checkout:

powershell.exe -NoProfile -File .\scripts\wsl-runner.ps1 -Action Install -Distribution Ubuntu
powershell.exe -NoProfile -File .\scripts\wsl-runner.ps1 -Action Status -Distribution Ubuntu

• The script registers ax-wsl-runner-Ubuntu, starts it immediately, and starts it again at that user's logon: wsl.exe --distribution Ubuntu --exec /bin/sleep infinity with the user's interactive token and limited privileges.
• No stored password, no task time limit, three retries at one-minute intervals. Repeated installation preserves a running task; a conflicting task is refused. -TaskName selects another managed task name.
• This provides availability after user logon, not before logon after a Windows reboot. It does not make running jobs survive a Windows reboot or WSL shutdown.

After deployment, close the SSH session and verify with ax exec info wsl-build and an ordinary ax job. Linux service reports identify their boot scope as the Linux distribution and report when a Windows keepalive is required.

After undeploying the runner, remove its task on Windows:

powershell.exe -NoProfile -File .\scripts\wsl-runner.ps1 -Action Remove -Distribution Ubuntu

Removal stops only the managed keepalive task; it never terminates the WSL distribution or other jobs. Task installation and removal must use the same Windows account, distribution and task name.

Execution platforms

• Linux with systemd/cgroup v2, or macOS with a user launchd manager. Native Windows execution remains unavailable; WSL uses the Linux backend.
• Copying a CLI binary alone does not establish a working runner. Check the deployment's exec_ready result or run ax exec info <host>.

Local deployment history

• Each ax deploy ssh attempt, including --plan, is recorded locally before SSH calls. ax deploy ls covers the current profile and workspace; ax deploy ls -a covers every workspace in that profile. JSON, CSV and TSV are supported. Successful responses include the operation ID and origin.
• Interrupted attempts remain in_progress; failed calls are marked failed_outcome_unverified because the host may have changed. See harness integration to attach harness and session context.

Remove an installation

Inspect removal first, then remove the selected installation using the same SSH connection options as deployment:

ax undeploy ssh build-host --plan --format json
ax undeploy ssh build-host
ax undeploy ssh build-host --system --plan
ax undeploy ssh build-host --system --stop-services

• Progress shows SSH connection and inspection/removal phases on stderr, with reminders while an operation is waiting. --quiet hides progress; JSON, CSV and TSV suppress it. Only the final verified result confirms removal.

• Default scope is the SSH user's installation. --system selects only the Linux machine-wide installation (root or noninteractive sudo). Removal does not require the installed binary to run, or a matching local bundle.

• --plan inspects files and runtime blockers without changing remote files or services. A successful plan can report removal.blocked: true; inspect its reasons before proceeding. Checks run again during actual removal.

• Removal deletes the managed launcher, checksum-verified versioned executables and managed service files; system removal also deletes the managed supervisor launcher.

• Refused: unmanaged launchers, modified service commands, symlinked installation directories, damaged releases and extra files inside release directories. Deployment and removal share an installation lock. Repeating removal after success reports changed: false. Empty directories or lock files may remain; launcher backups are preserved rather than restored automatically.

• Running managed services block removal unless --stop-services is supplied. Active ax-job-*.service units block Linux/systemd removal even with that flag, including jobs associated with another installation. macOS also checks launchd job processes and refuses removal while they are running. Wait for jobs or cancel them through the Exec API first. Removal never cancels jobs or kills client processes.

• When explicitly permitted, removal stops only the selected managed host services, then checks jobs and installed processes again before deleting files. A race or failure at that stage can leave services stopped; inspect the host before retrying. Inability to inspect jobs/services also blocks removal.

• Stop installed CLI/MCP clients and profile daemons first; existing runtime sockets under the default ax home require inspection even if possibly stale. These checks cover managed services, installed executable paths and standard ax runtime paths — they are not a lock against independently launched processes or custom external runtimes.

Configuration, credentials, logs, state and retained job history are preserved by default. To delete standard ax data paths as well:

ax undeploy ssh build-host --purge --plan
ax undeploy ssh build-host --purge

• Purge does not follow POSIX symlinks, crosses no filesystem boundaries during deletion, and refuses detected mount points. Windows purge refuses reparse points within its data tree.

• Untouched: external/custom state directories, OS keyring entries, OS accounts, user lingering, SSH settings, payload workspaces and the other installation scope. Payload files explicitly placed inside a purged ax data directory are deleted along with that directory.

• Removal attempts record harness/workspace origin in the same local history as deployment, with action: "undeploy"; older entries default to "deploy". ax undeploy ls and ax deploy ls show this shared history, with -a and JSON/CSV/TSV support. Failed or interrupted removal can be partial and is never reported as a rollback. Purging local credential files does not revoke external enrollment credentials or grants.

• Tested: POSIX removal and mocked Linux service/job checks have local automated coverage. An isolated Linux container passed system removal, active-job refusal, repeat removal and explicit system purge preserving user state. A real Linux host passed a read-only removal plan. Windows has a PowerShell removal path but has not been tested on a live host. Reboot behavior and removal of a configured production execution host are not established by those tests.

Allow sudo in Linux jobs

Linux runners block privilege elevation by default. To let authorised jobs use an account's existing sudo permissions, the runner owner can set:

"allow_sudo": true

• User installation: add this top-level setting to ~/.config/ax/server.json, then run systemctl --user restart ax.service.
• System installation: add it to /etc/ax/helper.json, then restart ax-helper.service. The helper's configuration decides in system mode.
• Omitting the setting, or setting it to false, blocks elevation for new jobs. Restarting the service does not change jobs already running.

ax exec info <runner> advertises the sudo adapter when permitted. This reports runner policy; it does not promise that sudo is installed or that a particular command is allowed. Use sudo -n in jobs so a password requirement fails immediately. Ax does not change sudoers, supply passwords or open a terminal. The effective setting is retained in each sealed job specification.

• Enabling it removes the payload's no_new_privs restriction, including for other setuid and file-capability programs.
• This setting applies to every authorised job on that runner. Enable it only when those callers may use the payload account's sudo permissions. Sudo may permit full host administration, including changing ax or moving work outside its supervision; account permissions and working-directory limits do not contain root. Use another runner account or a narrower sudoers policy when callers need less access. An inherited no_new_privs restriction causes an enabled job to fail at launch; ax cannot clear that kernel flag.