Remote Run Machine (--remote[=role])¶
Some verbs are host-bound: iOS capture (xcrun simctl) only runs on macOS, an Android emulator needs its own machine, and generally a run/build/capture target may need hardware/toolchain that isn't on your day-to-day box. mship run/capture/build --remote[=role] executes the same go-task target on a different, already-bootstrapped mship workspace, reached over the relay, and streams its output back to your terminal.
This doc covers the model, how to configure it, and how to read the failure messages it produces.
Relay credential boundary: relay-native registrations resolve a short-lived bearer at each Git/HTTP boundary through the authenticated host directory. They do not store a standing daemon token, follow redirects, use a direct route, or fall back to local execution. A real relay-only dirty-source operator smoke test remains a release gate; examples and automated tests are not evidence for that physical-host gate.
The model¶
A "run host" is just another mship workspace, already set up (mothership.yaml present, repos cloned) on the machine you want the verb to actually execute on — a Mac with the right simulator, an Android box, a beefier build machine, whatever. That machine runs:
mship serve --relay
exactly like the phone-pairing flow: it dials out to the relay (so NAT/"somewhere else entirely" is fine) and is reached through a directory-selected public relay URL. Relay-mode operators never copy the daemon standing token into their registration.
- The remote materializes the task's branch —
git fetch+ a worktree at.worktrees/<task>/<repo>, mirroring the local worktree layout. Remote execution always operates on a task's branch; there's no ad-hoc remote run (the remote needs a branch to check out). - The remote runs the repo's go-task target (
run/capture/build) with the verb's existing task/capture variables, using the explicit server runtime environment described below rather than inheriting the daemon's entire environment. - Output streams back live on stderr (not a final blob), leaving structured command results on stdout parseable.
--quietsuppresses progress; the remote task's exit code becomes your local process's exit code. - For
capture, produced artifacts (screen.png,layout.*) are pulled home automatically.
The client imposes no read-idle timeout on a valid execution response body: a quiet compiler or running app must not trigger HTTPX's default five-second read timeout. Response headers, HTTP error bodies, connection establishment, writes, and pool acquisition retain five-second timeouts. There is no overall client run deadline; interrupt the command when you want to stop waiting. This does not disable timeouts imposed by a proxy or relay on the route.
Accepted execution streams send protocol keepalives every 30 seconds, including quiet profile owners, log observers, and legacy remote builds. Clients discard these frames: they are not application logs and do not replay an operation. Update both client and run host for the typed keepalive event. The proxy's idle timeout must remain longer than this cadence; request authentication and pre-acceptance deadlines are unchanged.
Without --remote, legacy unprofiled mship run, capture, and build behavior is unchanged. A task-bound run that selects a repository declaring run_profiles deliberately enters the profile-aware path even without an explicit --profile; see Profile-aware runs and observations.
Packaged backends on a run host¶
A configured repos.<repo>.run_backends.<project-name>.builtin is available on
a run host without copying an adapter, creating a project wrapper, or
bootstrapping a second Python environment. The host must run the same mship
installation that serves the request; delegated tasks use its
"{{.MSHIP_SESSION_PYTHON}}" -I -m mship.backends <builtin> [discover]
entry point.
The host still needs the external platform prerequisites for the selected target: Android SDK/adb, Flutter and its prepared toolchain, macOS/Xcode for iOS, a configured browser engine/driver, or PlatformIO with board and serial permissions. Installing or preparing those dependencies is a host-operator step, not discovery. Discovery remains read-only and must not launch a browser, boot hardware, reset a board, or upload firmware.
Bindings, browser paths, serial identities, SDK paths, and credentials remain
private to the host. They are keyed by the configured project backend name and
are passed only through mship's sealed request/context/bindings files; do not
place them in committed mothership.yaml or forward them as operation
arguments. A task override inherits every builtin action it does not name. An
override that delegates to a native builtin must preserve the builtin's session
owner and use the supplied sealed context rather than selecting or creating a
different device session. See Run backends for
the three supported configuration modes.
Declaring roles (mothership.yaml)¶
mothership.yaml is public (this repo), so it only ever names logical roles — never a URL or token:
run_hosts: [ios-sim-host, android-emu-host]
repos:
ios-app:
capture:
platforms: [ios]
run_host: ios-sim-host # optional: this repo's default role for --remote
run_hosts is the workspace's full list of roles anyone on the team might map. A repo can optionally declare run_host: <role> as its own default, so mship capture --remote (bare, no =role) auto-resolves without every operator having to type the role name.
Mapping a role to a connection (mship run-host)¶
Named host registrations are private. User registrations live in
$XDG_CONFIG_HOME/mothership/run-hosts.yaml (or ~/.config/mothership when
XDG is unset, empty, or relative); project registrations live in the private
.mothership/run-hosts.yaml. A same-named project entry replaces the complete
user entry. Neither location belongs in mothership.yaml.
# on the remote machine (the run host itself):
mship pair # prints a groundcontrol://add?... link (+ QR)
# on your operator machine: reusable registration
mship run-host add studio --scope user --role ios-sim-host \
--pair-link 'groundcontrol://add?...'
# a project-specific complete replacement (the default scope is project):
mship run-host add studio --role ios-sim-host \
--url https://mac-abc123.relay.example.com --token <serve-token>
mship run-host list # safe name, role, URL and winning scope; never token
mship run-host remove studio --scope project
Existing local role-to-connection files must be migrated explicitly; preview first, then apply. Migration writes an owner-private exact backup beside the original file and preserves each old role as an exact named host:
mship run-host migrate --scope project
mship run-host migrate --scope project --apply
After migration an old role remains restricted to its exact converted host.
Only deliberately opt into a pool with mship run-host allow-role <role> --all
or list approved names with repeated --host NAME.
MSHIP_RUN_HOST_<ROLE>_URL / MSHIP_RUN_HOST_<ROLE>_TOKEN env vars are a
per-role override for one already eligible host; both variables can also form a
legacy environment-only registration. They never distribute credentials across
a pooled role (role upper-cased, - → _).
Relay-native registration¶
Direct registrations remain deliberately direct: --url/--token and the direct
--pair-link keep their existing private {url, token} mapping and may use the
documented direct environment overrides. A relay registration stores only
relay, host_id, workspace_id, and the directory-selected instance_id;
it never stores a public route, refresh, fleet credential, short-lived bearer,
or daemon standing token.
The relay owner first approves/enrols the host and issues the already-existing
account link with mship relay fleet-token --label <coordinator>
--relay-domain <relay> --store-dir <relay-store>. Transfer that
groundcontrol://add-relay?... link over a secure out-of-band channel. On the
coordinator, enter it only at the echo-hidden prompt:
mship run-host pair-relay
mship run-host add studio --role ios-sim-host \
--relay relay.example.com --host-id <approved-host-id> \
--workspace-id <workspace-id>
pair-relay consumes but never mints or displays the fleet credential. A
direct groundcontrol://add?... link cannot seed relay pairing. If pairing is
missing/revoked, transfer a new account link and pair again; if directory
selection or instance binding fails, re-enrol/approve the host or correct the
stored identity mapping.
Profiled runs report safe source-preparation and relay-authentication diagnostics
on stderr, identifying the failing repository or selected host.
Missing or revoked pairing includes the mship run-host pair-relay recovery
command even with --json or --quiet. These diagnostics do not add retries or
allow local fallback.
A final execution-endpoint authentication rejection is reported after any permitted pre-start retry; a successful refresh/retry emits no authentication error. Relay diagnostics identify the host/workspace registration to check and the relay pairing command. Direct registrations get a command template requiring a fresh direct pair link; it preserves the registration's roles, tags and preference. These authentication diagnostics never print credentials or HTTP error bodies. Source-transfer failures report the failed stage and recovery checks, not captured Git error output, which can contain server-supplied text.
To replace a legacy direct record, select the approved identity first and use the explicit preview/apply flow. It writes a private byte-for-byte backup before replacement; it never treats the legacy token as a refresh credential or keeps it as a hidden fallback:
mship run-host migrate studio --relay relay.example.com \
--host-id <approved-host-id> --workspace-id <workspace-id>
mship run-host migrate studio --relay relay.example.com \
--host-id <approved-host-id> --workspace-id <workspace-id> --apply
For a relay record, every dirty-source Git push/delete and every legacy or
typed execution independently authenticates GET
https://enroll.<relay>/hosts, selects the stored host/instance, exchanges its
refresh only with that selected HTTPS public origin at /host/token, and binds
the configured workspace under /workspaces/{workspace_id}. Redirects are
refused. A request may make at most one refreshed retry after a definite
pre-start 401/403; source mutation, cleanup, accepted streams, timeouts, and
framing/disconnect failures are never replayed.
The Mac Studio retirement gate remains operator-only: do not remove its
standing-token mapping or temporary Tailscale Serve forwarder until an operator
has run a real relay-only dirty-source build and recorded safe route,
source-receipt, host/workspace tool-route, completion, and coordinator
process-tree evidence showing gradle_or_java_build_descendant_execs: 0.
Loopback/unit evidence is not that gate.
Using it (--remote[=role])¶
mship run --remote # bare: auto-resolve the role (repo's declared run_host, else the sole configured run_hosts entry)
mship run --remote=ios-sim-host # explicit role
mship build --remote=android-emu-host
mship capture --repo ios-app --remote=ios-sim-host
run/build --remote require a resolvable task (--task, MSHIP_TASK, or cwd) — the remote checks out that task's branch, so there's no ad-hoc remote run. capture --remote has the same requirement (no ad-hoc remote capture).
Bare --remote (no =role) auto-resolves in this order: the target repo's declared run_host, else the sole entry in run_hosts if there's exactly one. Two or more roles with nothing chosen is an ambiguous-role error (see below).
Profile-aware runs and observations¶
run_profiles is the opt-in, task-bound path for durable target-aware runs. A
bare run selects each configured repository's default_run_profile; when no
default is declared, human TTY output presents a profile chooser, while a
non-interactive run requires --profile. --profile chooses a configured
profile, --host restricts it to a configured host registration, and --target
restricts it to a discovered configured target alias:
# Default profile, or the TTY chooser for each selected profiled repository.
mship run --task feature-a --repos app
# Explicit profile launch, constrained to an approved host and target alias.
mship run --task feature-a --repos app --profile android-usb \
--host lab-android --target pixel-8
Each selected profiled repository must resolve against the active task. Before
any application launches, mship discovers and preflights every selected profile;
the dependency graph's ready boundaries remain in force. Each profiled
repository gets one separate run ID. Selected repositories without profiles run
their ordinary run tasks in the task's worktrees, not just their build tasks.
Foreground task completion and background healthchecks gate dependent launches,
including dependencies between ordinary and profiled repositories. Mship stops
the ordinary processes it owns when the profiled streams finish, fail, or are
interrupted. Ordinary repositories do not receive synthetic run IDs; requesting
a profile, host, or target for one is an error rather than a fallback.
Ordinary processes in mixed runs require POSIX process groups and non-reaping child observation. Cleanup retains each group leader's identity until its remaining group members terminate, escalating to a forced stop when needed. Children that deliberately create a different process group or session are outside that ownership boundary.
A ready acknowledgement is not a successful final outcome. A subsequent nonzero
backend exit records the run as failed and makes the command fail. A zero exit
records stopped; an authenticated cancellation receipt matching the acknowledged
owner and generation also remains a successful stop.
Automatic observation selectors¶
capture and logs observe recorded runs rather than selecting a new device.
For a task-bound repository with profiles (or existing recorded candidates),
they automatically reuse exactly one matching active, acknowledged AppRun.
Several matching runs use the TTY chooser when available or require a more
specific selector in non-interactive use; no matching run tells the operator to
establish one with mship run. The optional selectors narrow recorded runs:
--run-id selects its identity, and --profile, --host, and --target
filter the profile, original host, and attested target alias respectively.
# Reuse the sole acknowledged run for app, if there is exactly one.
mship capture --task feature-a --repo app
mship logs app --task feature-a
# Observe a particular recorded run; these do not launch or rediscover a target.
mship capture --task feature-a --repo app --run-id <run-id>
mship logs app --task feature-a --run-id <run-id>
The recorded private binding determines the observation platform. capture
infers it when --platform is omitted; a supplied --platform must agree with
the recorded platform. --kind remains an artifact-kind selector
(image, layout, or all), not a backend operation selector. A supplied
--remote must name one of the recorded host's roles, rather than rerouting
the observation. logs --all means every configured service/repository; it
never means every device or every recorded run.
The run ID is an observation identity, not a device selector. An observation uses the exact recorded host registration, owner reference, generation, source revision, profile/backend revision, and private binding. A changed host, configuration, target, source identity, missing binding, inactive run, or unacknowledged owner is refused. Mship does not transfer source, rerun setup, rediscover a target, launch a replacement, or fall back to legacy capture/logging for that observation.
Profile configuration, host bindings, and examples¶
run_profiles names a backend, eligible host roles, and reviewed options.
run_backends may select a packaged builtin directly, or route selected
discovery/operation actions through ordinary repository task keys. A builtin
without overrides has no Taskfile requirement; an override inherits every
unnamed action.
The run-target configuration example shows builtin-only, builtin-plus-override, and custom task-routing modes. Its Flutter extension Taskfile and custom-routing Taskfile delegate to installed adapters rather than copying adapter code. The packaged sources are Android, Flutter, iOS, browser, and PlatformIO.
Keep executable paths, target aliases, SDK/app templates, device identities,
credentials, and state directories in the owner-private
$XDG_CONFIG_HOME/mothership/run-target-bindings.yaml (or
~/.config/mothership/run-target-bindings.yaml), never in mothership.yaml.
The binding key is the project's configured backend name, not a builtin name.
Discovery is read-only: normal launch and observation do not install SDK
components, create emulators, pair or provision devices, start a daemon, or
adopt an ambient target. The selected host must already have the required
SDK/toolchain and prepared application or simulator state. host_tools provides
explicit mise tool installation and readiness inspection only; it does not
provision platform components or targets.
Android and Flutter retain their concrete native owners and binary provenance
rules; a task override must preserve that framework ownership. Fully
project-owned backends retain the established protocol and may opt into only a
compatible android or flutter session owner. Physical iOS remains
unavailable until a concrete native USB owner exists.
Native Android component checks treat com.example.app/.MainActivity and
com.example.app/com.example.app.MainActivity as the same activity in discovery,
profile/binding validation, APK inspection, and post-install acknowledgement.
The configured spelling is preserved; another package or activity is not a
match. Malformed or ambiguous activity resolver output is rejected.
The browser launch owner retains one Playwright connection and the actual page for the whole run. Logs and capture use its authenticated, owner-private Unix control socket, not a second Playwright connection or a search for a matching page. Version-2 run receipts bind that socket and capability to the exact run, managed instance, and page generation; legacy receipts are rejected rather than adopted. Playwright and its browser binaries remain explicit host prerequisites.
Console and page-error listeners attach before navigation. Late log subscribers
receive a bounded backlog, then live events; cancelling a subscriber does not
close the browser. The configured page URL remains an identity constraint:
navigation elsewhere disconnects logs and rejects observation/capture without
selecting another page. The backend leaves the application's window.name
untouched. Failed captures remove only artifacts they created, and normal owner
shutdown removes its private control resources.
PlatformIO intentionally maps its durable run operation to board monitoring.
Its upload mapping is a separate typed, finite operation, not a new generic
CLI command or a lifetime-owning session.
Owner cleanup and provenance¶
On normal mship close, mship resolves the exact recorded host registration
and asks the exact recorded owner/generation to stop. Only after a confirmed
stop does it mark the run stopped, delete its metadata, and remove an
unreferenced private binding. A failed, unknown, unreachable, changed, or
unmatched host/owner leaves its recovery metadata and private binding intact
and blocks close; it is never redirected to a role-derived replacement host.
An attached profile run exits successfully when it receives the exact owner's
authenticated cancellation receipt, even if hot reload advanced the source
revision and close already deleted the controller record. A missing record or
disconnected stream alone is not proof of termination.
Source provenance and binary provenance are distinct. A source snapshot
certifies the worktree revision delivered to a host; it does not attest an APK,
app bundle, or running binary. The internal
InstallFromResult(result_id, artifact_id, sha256) capability verifies an
immutable result artifact and digest before granting it to an owner, but it has
no public mship install, upload, or attach command. Consumers retain platform
binary verification, installation policy, and their provenance record.
--update-and-hot-reload remains Flutter-only: it requires exactly --run-id,
one task-bound repository, a recorded active run with reload, and no
--profile, --host, or --target. It transfers a certified source update to
that exact owner; it cannot select or recreate a session. Dependency, native
build, Taskfile, or host-tool changes require setup or a fresh launch.
Source reservation drains existing observations before mutation. Each handoff phase uses a 120-second HTTP read timeout so source checks and Flutter control are not interrupted by the ordinary five-second transport default. An unknown outcome is not automatically retried; establish a fresh run rather than replaying an uncertain transaction.
Session verification boundary¶
Real authenticated direct-TCP Android acceptance has passed on Studio with
Ground Control and an Android 16 / API 36 emulator: read-only discovery,
target-pinned launch, logs, log-client cancellation without stopping the parent,
screenshot and parseable XML layout capture, and acknowledged app cleanup.
The emulator stays running after close. The APK was built and installed
separately; this does not claim an InstallFromResult provenance grant or a
paired Ground Control workspace workflow.
Android emulator identity supports both ro.boot.qemu.avd_name and the older
ro.kernel.qemu.avd_name. Foreground checks recognize current and older
resumed-activity output. The Android example resolves the host temporary
directory before creating private bindings, and layout capture separates
UIAutomator's status trailer from its validated XML document.
This is single-host emulator acceptance, not physical Android USB, cross-host replacement/outage, firmware upload, or public-relay acceptance. Separate native iOS simulator and Flutter owner evidence does not establish those gates.
Real browser acceptance has also passed on Linux using Playwright 1.63.0 and Chrome for Testing 150.0.7871.24: authenticated direct-TCP discovery and launch, startup/live console and page-error logs, log cancellation with the parent still active, recorded-run screenshot/DOM capture, replaced-instance rejection, and normal close. An independently opened browser remained usable after close. Additional real-driver checks cover stale capabilities, partial-artifact cleanup, long receipt paths, preserved application window state, and navigation identity loss. This is Chromium acceptance, not Firefox, WebKit, native Safari, or public-relay proof.
Authenticated direct-TCP Flutter source-update acceptance has passed on Studio
with prepared QuicklyGuide and an iOS 26.5 simulator. A certified Dart source
change drained an existing logs observer, applied to the recorded owner, and
hot-reloaded a visible light-to-dark theme change. The owner reference,
generation, and app process remained unchanged; the durable receipt reached
reloaded and the remote worktree was clean at the new certified revision.
Normal task close then stopped the app while leaving the simulator booted.
This does not attest the prepared binary, Flutter on Android, physical iOS,
or public-relay execution.
Pinned host tools¶
A repo may optionally declare a strict host_tools.mise manifest and native
lock in mothership.yaml. The declaration is read only from the selected
host's materialized task worktree; global, ancestor, local, fragment, and
environment-selected mise configuration are isolated and rejected if discovered.
# Read-only, selected-host diagnosis. Requires one task and repo.
mship doctor --remote=android-emu-host --task feature-a --repo app
# The only command that installs declared mise tools. It runs bare `mise install`,
# rechecks the same declaration, and records a safe host-local readiness receipt.
mship bootstrap --host-tools --remote=android-emu-host --task feature-a --repo app
Normal remote execution never installs tools: it verifies readiness before
repository setup and runs the authorized task as mise exec -- task <actual>
with mise auto-install disabled. These commands never install Android SDK
components, accept licenses, touch devices, or execute a caller-side probe.
The two-credential model¶
Two different credentials are in play, and they never mix:
- Relay pairing token (the run-host's own serve bearer token, handed to you via
mship pair/--pair-linkor--url/--token) — this is what YOUR box uses to authenticate to the REMOTE'smship serve --relay. It lives only in the gitignored.mothership/run-hosts.yamlon your machine, keyed by role. - The remote's own git credentials — the remote box fetches the task branch using its own git auth (a normal git credential helper, SSH key, or the
/gh-tokenbroker for a credential-less/cloud remote). No GitHub token ever crosses the wire between your box and the remote.
Nothing secret is ever committed to mothership.yaml — that file only ever holds role names.
Where capture artifacts land¶
mship capture --remote writes artifacts to the exact same local path a local capture would use:
.mothership/captures/<task-slug|_adhoc>/<UTCts>-<platform>/
so discover_artifacts and anything reading captures locally (including an agent) sees them unchanged, regardless of whether the capture ran locally or on a remote host.
Known limitations¶
These boundaries apply to remote execution:
symlink_dirs/bind_filesare not replicated on the remote worktree.task setupnow runs there (see "Dependencies are derived there, not copied"), so a repo whose deps come from tracked manifests works. A repo that depends on symlinked gitignored material from your source checkout still does not.- Remote task stdout is streamed to your terminal verbatim. There is no ANSI / control-sequence sanitization — the remote host is trusted. Don't point
--remoteat a host you don't control. - A
run_host:set under acapture:block inmothership.yamlis silently ignored.CaptureConfighas norun_hostfield; only the repo-levelrun_host(documented above under "Declaring roles") is honored. Putrun_host:directly on the repo, not inside itscapture:block.
What travels to the run host¶
run --remote, build --remote, and legacy capture --remote prepare
the source you are looking at, including work you have not committed.
Selected live-session capture (--run-id, or its sole healthy recorded-session
case) does not: it observes the recorded owner without source preflight,
transfer, or setup. The materialized legacy paths inspect each selected repo
before dispatch and take one of three paths:
- Working tree differs from HEAD — tracked edits, untracked files, or both →
mship builds a commit from your working tree and pushes it straight to the
run host, onto a throwaway ref (
refs/mship/run/<task>/<repo>). The host resets a worktree to that ref and runs it. Nothing is pushed to origin on this path, and nothing on your machine changes: your HEAD, your branch, your index andgit statusare exactly as you left them, and the synthesized commit belongs to no branch. mship names it as a throwaway run ref in the output for that reason — do not build on it. - Clean, but origin is missing the branch or is behind it → mship pushes the
branch to origin for you, then dispatches. There is nothing extra to send, so
this is the old, fast path. If origin has a commit you do not, mship refuses
— a push cannot fast-forward from behind, and the run would execute a commit
you have never seen. It prints the
git pull --ff-onlythat fixes it. - Mid-merge, mid-rebase, or with unmerged paths → mship refuses, and names the command that unblocks you. Files in that state hold conflict markers, and a remote failure over a conflict marker tells you nothing about the edit you were making.
It also still refuses a worktree that is not on the task's branch, a repo whose git state it cannot read, and a worktree that is missing — in each case naming which, because the remedies differ.
Every repo the run will actually touch is checked, not just the one you are
standing in: a task has a branch per repo and the run host materializes each
separately. --repos / --tag narrow the check as well as the run, so work in
progress in a repo you excluded neither blocks the run nor gets sent.
Why the run host and not origin¶
Routing uncommitted work through origin would publish it. git add -A sweeps in
untracked files, so a debug dump, a data sample or a throwaway script with a
token in it would land on GitHub. Refs under refs/mship/ are outside the
default fetch refspec but they are not private — git ls-remote enumerates them
and anyone with read access can fetch them, which on a public repo means anyone —
and deleting the ref afterwards does not retract the objects, because they stay
reachable by sha. The destination is your own machine, so there is no reason for
a third party to be in the path. Real history goes to origin; throwaway state
goes host to host.
The run host accepts these pushes on a purpose-built endpoint: direct records
use /git/<repo>, while relay records use
/workspaces/{workspace_id}/git/<repo> after an independently resolved
short-lived bearer. Both accept only declared repos and writes under
refs/mship/run/*. It is not a mirror, not a remote you add by hand, and not a
path for real history. Each run force-updates its own ref, and mship close
deletes the task's scratch refs from the host.
Nothing here changes what mship finish requires. The scratch namespace is not
a branch, is not PR-able, and no code path merges or branches from it — so
nothing reaches a PR unreviewed.
The guarantee, and where it stops¶
On the clean path mship pushes the exact sha it resolved HEAD to during
inspection — <sha>:refs/heads/<branch> — rather than letting git resolve HEAD
(or the branch) a second time when the push runs moments later. On the dirty path
the same sha becomes the synthesized snapshot's parent. Either way, if something
else commits in the worktree in between — a subagent, a background job — the run
still carries the commit every check actually cleared.
That guarantee ends at origin, and this is a real limit, not a hypothetical one: the commit pushed is the commit inspected; that is not the same claim as "the commit executed is the commit inspected." Once a push to origin lands, the branch there is a mutable ref, and anyone with push access can advance it before the run host fetches it — after mship has finished checking, on a different machine's clock, outside this process entirely. Closing that gap would need the run host to materialize an immutable revision instead of resolving a branch at fetch time. The dirty path already does exactly that: it materializes a specific commit from a ref nothing else writes, with no fetch at all. The clean path does not.
Capture provenance¶
Legacy remote capture uses the same repo-scoped source preflight as remote
run/build. An unsafe checkout or failed source transfer stops that legacy
capture before dispatch; a git_root child uses its canonical repository for
transfer.
Selected live-session capture is the exception. A capture of a recorded run
(--run-id, or the sole healthy matching recorded session) is an
/exec/session-capture observation: it validates the persisted owner and
capability but performs no source transfer, materialization, setup, launch
replacement, or target rediscovery. It observes the source identity already
recorded for that run, not the caller's current working tree.
Neither kind rebuilds or relaunches the app. Legacy source preparation therefore does not prove which binary is visible on the device; selected-session observation does not make an installed binary match a newer local tree. Run or explicitly update the app first when a fresh build/source handoff is required.
Attached remote evidence records source preparation separately from device provenance. For dirty legacy trees it names the synthesized snapshot SHA and throwaway ref, not the parent HEAD. It explicitly states that the remote checkout and running binary provenance were not verified; it does not label a remote image as a local clean-HEAD capture.
Dependencies are derived there, not copied¶
Git carries source, not node_modules. So after materializing, the run host runs
task setup in that worktree, rebuilding dependencies from the manifests the
push just delivered.
That is keyed, or it would defeat the fast loop this exists to enable:
- setup runs the first time a worktree is materialized for a task on that host — so the first remote run on a fresh host is the slowest it will ever be, a one-time cost rather than a regression;
- and again whenever the repo's declared
setup_inputs(its manifests and lockfiles —package.json,uv.lock,build.gradle) differ from what that host last set up at.
A source-only edit, the common case, pays nothing. A dependency change pays once.
A repo that declares no setup_inputs gets setup on first materialization
only, because there is nothing to invalidate against — declaring them is what
buys re-run-on-change. A repo that defines no setup target at all is skipped
rather than failed. If setup fails, the run stops and you see setup's own output.
What does not travel¶
- Gitignored files.
.envand other secrets, build output, virtualenvs,node_modules. Where they can be rebuilt from tracked manifests that is now setup's job; where they cannot — secrets, platform state — they simply are not there, and you put them on the run host yourself. symlink_dirs/bind_files. Still not replicated on the run host.- Your machine. The source is exact and the dependency environment is derived from it, but the run host is not a clone of your box.
Internal tool-runner API¶
This is an adapter-facing Python API and authenticated POST /exec/tool route,
not a public shell command. It does not change the legacy mship run, build,
or capture CLI routes.
Requests, preparation and results¶
mship.core.remote_tool.ToolRequest accepts task, a configured repo, and
an argv tuple; argv values are literal arguments, never joined into shell
source. task_key is an alternative to argv: it requires an empty argv and
is resolved on the host through that repository's configured tasks mapping.
An unknown key is invalid; callers cannot name an arbitrary host command through
this field.
input_files is a mapping of environment-style names to UTF-8 contents, not a
mapping of caller-selected paths. Reserved host identity names
(MSHIP_TASK, MSHIP_REPO, and MSHIP_SOURCE_REVISION) are rejected.
For a profile request, the host validates MSHIP_TARGET_REQUEST_FILE against
its configured profile, backend, logical task, options, and certified source
revision, then derives its own MSHIP_TARGET_BINDINGS_FILE. An observation
requires that same validated request input plus its
MSHIP_TARGET_CONTEXT_FILE; context is additional recorded-owner authority,
never a substitute for a request. A request that supplies profile inputs
without a valid profile request, supplies context outside observation, or fails
this configured-task validation, is invalid.
The operation registry materializes accepted input contents as randomized,
owner-private 0600 regular files under its private operation directory. Child
environment names hold only those private file paths—never the input contents;
the paths, contents, argv, and environment are not published to the task journal
or result events. The registry checks file ownership and identity before use and
removes the inputs during terminal cleanup. These files are request-scoped
private inputs, not a provisioning channel or a persistent device/session
binding.
ToolRequest also accepts explicit env, relative cwd (default "."),
preparation ("discover", "launch" by default, or "observe"),
source_revision, run_ref_repos, paired owner_ref / generation for
observation, and output/timeout limits. The server derives the worktree and
rejects an absolute cwd, parent traversal, or a resolved cwd outside it. This is
cwd confinement, not a filesystem sandbox for trusted project code.
| Policy | Source/setup behavior | Output and lifetime |
|---|---|---|
discover |
Prepare and verify task source; never invoke or update the setup cache | Require positive independent caps, at most 1 MiB stdout and 256 KiB stderr, and a finite positive timeout. Return bounded bytes, not incremental inventory fragments. |
launch |
Prepare and verify source; run cached setup when needed; verify source again after setup | Stream stdout/stderr without retaining the complete output in memory. Collection caps must be None; an optional timeout bounds the backend process, not preceding source transfer/setup. |
observe |
Validate the existing owner; no source transfer, materialization, setup, launch replacement or launch-flock acquisition | Empty argv queries status. Nonempty argv runs a separately owned observation process in the existing context. Collection caps must be None; timeout is optional. |
ToolResult exposes status, exit_code, owner_ref, generation,
source_revision, and binary stdout / stderr. completed means an ordinary
process exit, including a nonzero exit code: require both completed and
exit zero before parsing discovery inventory. Limit, timeout, cancellation and
infrastructure failures do not return a partial discovery payload.
Other statuses are invalid, busy, unsupported, materialization_error,
launch_error, stdout_limit, stderr_limit, timeout, cancelled,
evidence_error, unknown, auth_error, and protocol_error. running is
valid in a started event and as the final answer to a status-only observation,
not as a final answer to an executing request. Pre-admission failures may lack
an owner; accepted results must include owner/generation/source identity.
Owner and generation must remain unchanged. Readiness, completion, and
observation results must match the expected source revision. A terminal
cancelled result on an accepted launch stream instead proves that exact
owner stopped: it may carry a newer source revision after hot reload, even
when close has already deleted the controller record. It is not proof of
successful execution or source content. Missing identity remains a protocol
error.
Admission is typed rather than boolean:
ToolOperationRegistry.admission_status(task) returns available, busy,
unknown, or evidence_error. busy means a live, verified operation owns
that task; unknown means durable or in-memory ownership cannot be safely
validated (including a restarted process that found a nonterminal record); and
evidence_error means the admission evidence itself could not be read or
validated. Only available admits a new launch. Never treat unknown or
evidence_error as an invitation to replace or signal an operation.
ToolEvent.kind is started, ready, keepalive, stdout, stderr, or
result; output is binary data, and lifecycle events carry result.
keepalive is a nonce-authenticated empty protocol frame, emitted only after
ownership is accepted so a quiet long-lived session keeps the relay stream
alive; it is not task output and is not delivered to application callbacks.
The callback must consume output incrementally rather than retain an unbounded
event list. Setup output can precede the backend's started event.
Source, backend, and cleanup entry points¶
snapshot_remote_source(*, task_obj, target_repos, config, shell) returns an
immutable SourceSnapshot of certified source revisions. It inspects the chosen
repositories once; dirty repositories become private synthesized Git commits,
while clean repositories retain their inspected HEAD. The snapshot stores source
identities rather than caller paths or a host connection.
prepare_remote_source(*, task_obj, target_repos, config, shell, host, resolver,
output, on_prepared=None, snapshot=None, on_transfer=None) resolves a fresh
operation credential before each source mutation and transfers that source. It
returns PreparedSource(run_ref_repos, source_revisions).
Supplying a SourceSnapshot is authoritative: preparation neither reinspects nor
resynthesizes the local tree, so each eligible host receives the same certified
revision. on_transfer(git_repo, ref, sha) runs only after a successful dirty
source delivery.
RemoteBackendExecutor(*, task_obj, config, shell, output, store,
event_sink=None, transport=None) is the internal adapter that snapshots a
repository once, prepares every already-selected eligible host, records each
successful dirty transfer, and sends validated typed tool requests. It has no
local fallback and does not invent a source revision after a host-local
materialization failure.
resolve_launch(*, config, task, repo_name, profile_name, host_name, remote_role,
target_alias, registry, preferences, execute, choose) validates the configured
repository/profile/backend, prepares eligible hosts through execute.prepare,
discovers candidates, ranks them, and returns a SelectedTarget. Despite its
name, it does not launch an application. It requires an injected executor and
selection callback; it is not a CLI parser or caller-provisioning API.
Observation uses the persisted run's launch origin as provenance, not as the
requested follow-up operation. A configured logs or capture operation must
also be granted by that run's stored capabilities. Request/context JSON rejects
duplicate object members and excessive nesting before spawning any task.
record_run_ref_receipt(state_dir, *, task, host, repo, ref, sha) records a
successful scratch-ref transfer against the exact registered host name, scope,
and endpoint fingerprint—never its bearer token. Source-update transfers advance
the receipt to the new SHA; cleanup still uses an expected-SHA lease, not an
unconditional force-delete. cleanup_recorded_run_refs exposes this source-ref
cleanup independently of task teardown.
record_remote_worktree_receipt(state_dir, *, task, host, repo, sha) also records
clean profile and ordinary remote-command materialization. Task close reconciles
the exact session owners but retains remote worktrees, cleanup receipts and
scratch-ref leases; it does not request remote deletion.
The explicit cleanup_remote_task_worktrees API remains available independently
of close. Its caller must stop external writers first: the host's task lock only
excludes Mothership executions. The host verifies the registered worktree,
expected branch or detached ref, and recorded HEAD, and refuses tracked,
untracked or ignored content. It pins HEAD under refs/mship/recovery/ before
non-forced Git removal. Newer source, user changes, symlinked targets, busy owners
and uncertain outcomes retain cleanup receipts. Local mship prune does not
remove remote materializations. Remote cleanup and leased scratch-ref cleanup
remain explicit operations on the recorded host, never automatic close effects.
Configured roles alone never authorize ref or worktree deletion: no recorded delivery means no remote cleanup. Changed or unavailable host identities retain their receipts rather than falling back to another host. If receipt persistence fails after a push, preparation attempts leased rollback and reports unresolved cleanup if rollback fails.
mship.core.remote_client.exec_tool(*, request, host, resolver, event_sink=None,
transport=None) resolves a credential at the operation boundary and sends the
typed request with redirects disabled. Relay authentication may refresh once
only before any streamed event is accepted; there is no token fallback or local
fallback.
This internal surface powers the profile CLI path and recorded-run observation;
it is not a caller-provisioning or arbitrary-command API. It grants no
automatic device/session recovery, SDK/device provisioning, or public
mship install, upload, or attach command. Native/device and real relay
acceptance remain separate evidence boundaries.
Environment, ownership and durable evidence¶
The server builds the tool environment from a narrow host-runtime allowlist, not the daemon's complete environment. Entries present on the host are retained in these categories:
- Tooling and user runtime:
PATH,HOME,TERM,TMPDIR,TMP,TEMP,USER,LOGNAME,XDG_CACHE_HOME,XDG_CONFIG_HOME,XDG_DATA_HOME, andXDG_RUNTIME_DIR. - Locale:
LANG,LC_ALL, andLC_CTYPE. - Desktop session:
DISPLAY,WAYLAND_DISPLAY, andDBUS_SESSION_BUS_ADDRESS. - Android and Java:
ANDROID_HOME,ANDROID_SDK_ROOT,ANDROID_USER_HOME,ANDROID_AVD_HOME,JAVA_HOME, andGRADLE_USER_HOME. - Apple toolchains:
DEVELOPER_DIRandSDKROOT.
PATH, HOME, and LANG default to os.defpath, the server home directory,
and C.UTF-8 when absent. The request's explicit env is then applied, and
the server-owned identity values MSHIP_TASK, MSHIP_REPO, and
MSHIP_SOURCE_REVISION are applied last, so a request cannot override its
task, repository, or certified revision. The inherited base deliberately
excludes daemon secrets and controls: for example MSHIP_SERVE_TOKEN,
MSHIP_GH_TOKEN, MSHIP_GH_BROKER_URL, MSHIP_GH_APP_ID,
MSHIP_GH_APP_KEY, MSHIP_WORKSPACE, and run-host mapping variables are not
ambient tool variables. Put task-command configuration in explicit env; do
not depend on daemon credentials or control fields reaching a tool process.
A trusted repository env_runner may wrap the command using positional "$@"
forwarding; argv values are never interpolated into that shell source.
The server-only ToolContext(task, repo, worktree, source_revision, env_runner)
feeds one workspace-scoped ToolOperationRegistry. Its run, observe,
status, and admission_status methods are the common owner below both wire
adapters. Processes use stdin DEVNULL, binary pipes and a new POSIX session.
Unsupported process-group ownership fails before preparation/spawn; Linux
requires usable process-status information. Descendants that deliberately
leave the owned process group/session are outside this contract.
ShellRunner.spawn_argv(args, cwd, env) accepts a Path or a directory-FD
int for cwd. Registry launch and observation pass a duplicated, pinned
directory FD, not a pathname reconstructed from that FD. The internal isolated
exec bootstrap keeps that FD private while preserving the exact argv,
server-selected environment, binary pipes, new session, and exec-error
reporting. A spawn factory transfers exclusive Popen ownership to the registry
when it returns; it must not poll, wait, communicate with, reap, or otherwise
supervise that process. The registry alone collects output and performs
deadline/disconnect cleanup. /proc/self/fd/... is not an API or a supported
macOS cwd mechanism.
The launch root FD pins the worktree directory inode. Before a launch or nonempty observation executes, the registry verifies that the original worktree location still names that inode, then descends to the requested cwd descriptor-relative with no symlink following. A replacement or unverified root yields a typed refusal rather than running in a substituted directory.
Disconnect/deadline cleanup stops and reaps the owned group and coordinates
outstanding observers before releasing context availability. Uncertain cleanup
is unknown, not successful cancellation. A durable terminal status may become
visible while the disconnected HTTP response is still unwinding its task flock;
a competing launch can briefly receive busy.
Private records and separate raw stdout/stderr live beneath the configured
workspace state directory's remote-tool-operations/, namespaced by canonical
workspace and task hashes. Creation, opening, replacement and publication use
descriptor-relative operations with no-follow checks. Generated operation
directories are 0700; generated records and output files are 0600; owner,
regular-file, link-count and privacy checks are made at native-readable safe
checkpoints before metadata or output is trusted. Metadata publication is
atomic and synced. The existing task journal receives only safe
identity/status references—not argv, environment or raw output. Storage failure
is explicit; output is not silently discarded as successful execution.
After restart, terminal records are evidence only. Historical nonterminal
owners become unknown and their task remains unavailable; no stored identity
authorizes PID-only signaling, replay, or a new observation process. The runner
does not implement automatic recovery or a reconnect scheduler. Reconcile an
uncertain context through independently verified host maintenance, not by
blindly clearing its admission record.
The wire uses the response's X-Mship-Exec-Nonce and bounded length-prefixed
JSON events, with strict base64 for bytes. Request bodies are capped at 128 KiB,
event frames at 2 MiB, and output events at 16 KiB. Child bytes cannot become
control records. Legacy exit/artifact framing also has an authenticated empty
keepalive control record for quiet run, build, and session-capture streams;
clients consume it without rendering task output.
Verification boundary¶
Linux real-process loopback verification covers dirty exact-source transfer,
discovery without setup, literal argv and environment isolation, launch/setup,
live observation and cancellation isolation, disconnect cleanup, durable status,
output limits, quiet deadlines, and legacy run/build/capture artifact extraction.
The native remote-tool-runner spec and remote-tool-runner-507 task journal
record the implementation commit, test-run references, and observed proof.
Fixture source revisions identify disposable test projects, not the runner
implementation. This is not macOS/device testing or public-relay proof; the
actual #506 integration and relay-path release gate above still applies.
The Linux descendant-cancellation regression publishes its PID readiness file by atomic rename after writing it. File existence must mean the PID is readable, so parallel CI cannot observe an empty file before testing process cleanup.
Immutable declared task results¶
Typed POST /exec/tool requests that resolve a configured task_key with
task_outputs receive a server-created MSHIP_OUTPUT_DIR, immutable
declaration file, and producer-manifest path. On child reaping, the host copies
only each exact declared regular file into private storage by descriptor, hashes
it while copying, and persists a result before transient worktree/run-ref
cleanup. This is generic media support: zero-byte text/plain, archives, and
images are all valid when declared; there is no Android/package/device rule.
The existing capture contract is unchanged: capture still owns
MSHIP_CAPTURE_DIR, recognized capture names, nonce-framed tar transfer, and
evidence attachment. Declared task results never emit or parse capture tar
frames.
Authenticated clients use the existing serving workspace bearer boundary:
GET /task-results?task_slug=…,work_item_id=…, orrepo=…lists safe summaries (at least one selector is required);GET /task-results/{result_id}returns the safe immutable metadata; andGET /task-results/{result_id}/artifacts/{artifact_id}streams exactly the selected private blob only after a pinned-descriptor digest and length pre-pass, withContent-Length,ETag, andX-Mship-SHA256.
Opaque IDs are workspace-scoped. Guessed/cross-workspace IDs are 404,
expiration is 410, unavailable artifacts are 409, and integrity failures are
422 without filesystem, token, manifest, stdout, or endpoint disclosure.
Consumers must select both IDs and require outcome.status == completed,
exit_code == 0, availability == published, and a matching retrieved digest
before use. They separately own platform binary verification, installation,
session policy, and their own provenance record.
Troubleshooting¶
Start with mship net status. It reports every connectivity edge on this machine
— serve, relay, each run-host role, the GitHub auth model in effect, and whether
git is routed through a relay egress — with a status code and the fix for each
unhealthy one. mship doctor reports the same checks inline as a
connectivity/* group, and GET /net/topology on serve returns the same JSON.
mship net status # human topology view
mship net status --json # the same structure, for scripts
mship net status --no-network # configured state only, no probes
The table below is the reference for what each code means. The first six rows are
states mship net status detects for you; the rest surface only while a remote
task is running.
| Symptom | Code | Meaning | Fix |
|---|---|---|---|
unknown run-host role '<role>'; not declared in this workspace's \run_hosts:` list|run_host_unknown_role| You passed--remote=(or a repo declaredrun_host: ) but that name isn't inmothership.yaml'srun_hosts:list — likely a typo. | Add the role torun_hosts:inmothership.yaml`, or fix the typo. |
|||
ambiguous run-host: multiple roles are configured (...) and none was specified |
run_hosts_ambiguous_default |
Bare --remote with 2+ roles in run_hosts: and no repo-declared default. |
Pass --remote=<role> explicitly, or declare run_host: <role> on the repo. |
run-host role '<role>' is declared but has no connection mapped on this machine; run \mship run-host add |run_host_unmapped| The role exists inmothership.yaml, but *this* machine never mapped it to a{url, token}. |mship run-host add (get the link by runningmship pair` on the remote). |
|||
remote host at <url> is unreachable via relay (...) |
run_host_unreachable |
Couldn't even connect — the remote isn't running mship serve --relay, the relay is down, or the pairing is stale. |
Confirm the remote is up and mship serve --relay is running there; re-pair if the relay subdomain changed. |
remote workspace not bootstrapped at <url> (503) |
run_host_not_bootstrapped |
The remote's mship serve --relay is reachable, but that machine has no workspace config wired in (no mothership.yaml, or serve was started without one). |
Bootstrap that machine as an mship workspace and restart mship serve --relay there. |
remote host at <url> rejected the bearer token (401) |
run_host_stale_token |
The mapped token is wrong or was rotated on the remote. | Re-run mship run-host add <role> with a fresh pair link/token. |
error: unknown repo(s) ...; known repos: ... (streamed, then a non-zero exit) |
— | The remote's own mothership.yaml doesn't have a repo of that name — usually a workspace mismatch between your box and the remote. |
Confirm both workspaces declare the same repo names, or pass --repos naming a repo the remote actually has. |
A repo's task lines print, then error: branch-materialize failed for repo '<repo>': ... (then a non-zero exit) |
— | The remote's git fetch/git worktree add for that repo's task branch failed — commonly the branch not pushed yet, or a dirty/locked worktree on the remote. |
Push the task's branch, or clear the stuck worktree on the remote (git worktree remove/prune), then retry. |
Remote task's own output ends with a non-zero __MSHIP_EXIT__ <code> |
— | The task itself failed on the remote — same as a local failure. The streamed output above the exit line is the task's real stdout/stderr. | Read the streamed output like any other failing run/build/capture. |
--remote requires a resolvable task: ... / --remote requires an active task: ... |
— | You ran --remote with no active/resolvable task. Remote execution always needs a branch to check out. |
Pass --task <slug>, or run from inside an active task's worktree. |