microVM CI + Dev Enablement
Status: PROPOSED — this record freezes on merge.
Parent: microvm-runner.md (RIG-2394, frozen) — this record enables that record’s KVM-gated impl tasks (V2a→V8) to be developed and CI-verified across the compass dev shell and GitHub Actions, with a self-hosted fallback if hosted KVM proves unreliable; it changes no decision the parent froze.
Problem / Intent
Section titled “Problem / Intent”The frozen microVM Runner record chose cloud-hypervisor as the sole VMM (D1,
microvm-runner.md:642-670), booting a guest kernel + rootfs over virtio-fs +
vsock with a passt/gvproxy-class userspace net backend (D6,
microvm-runner.md:733-750). V1 (the MicroVMRuntime seam + backend
selection) is shipped and hardware-independent, but every subsequent slice —
V2a’s boot spike through V8’s acceptance suite — must actually boot a
cloud-hypervisor microVM, which needs /dev/kvm. That environment exists
nowhere today: the dev shell (devenv.nix) carries neither cloud-hypervisor
nor virtiofsd (verified: command -v cloud-hypervisor / command -v virtiofsd both absent in the shell, 2026-08-22, though the dev box itself has
/dev/kvm crw-rw-rw- root:kvm and /dev/vhost-vsock crw-rw-rw-, VMX on all
cores); and compass CI runs a single gates job on ubuntu-latest
(.github/workflows/ci.yml:96) that installs no VMM and enables no KVM. This
record decides how the microVM backend gets developed locally and CI-verified,
unblocking V2a→V8.
Approach
Section titled “Approach”Four decisions, one per fork. Together they produce a two-tier test posture
the frozen record already anticipates: hardware-independent unit/contract tests
run in the moon battery everywhere; boot/integration tests are tagged and run
KVM-backed — “an integration test (KVM-gated, skipped where absent)”
(microvm-runner.md:462), “KVM-gated for the microVM rows”
(microvm-runner.md:487). This record turns that anticipation into a named
mechanism and the environments that exercise both tiers. The load-bearing
choice is where the KVM tests actually boot: GitHub Actions, on the free
ubuntu-latest runner compass CI already uses — with a self-hosted Woodpecker
lane held as the documented fallback if hosted KVM proves unreliable.
E-D1 — dev prerequisites: userspace binaries in devenv.nix, host-level enablement in the host config (F1)
Section titled “E-D1 — dev prerequisites: userspace binaries in devenv.nix, host-level enablement in the host config (F1)”A split, at the userspace/host boundary. cloud-hypervisor, virtiofsd, and
passt are ordinary user binaries — they open(2) /dev/kvm but need no
capability, module, or device node of their own (“runs rootless as an ordinary
process”, microvm-runner.md:660-661; the whole backend is rootless by Global
Constraint, microvm-runner.md:379-385). Userspace-only tools belong in the
dev shell, where every developer and the CI toolchain bootstrap get them from
one pin:
devenv.nix: appendpkgs.cloud-hypervisor,pkgs.virtiofsd, andpkgs.passtvialib.optionals pkgs.stdenv.isLinux [ … ]outside the parsedwith pkgs; [ … ]literal — the exact posturexvfb-runandchromiumalready use, because the toolchain-parity gate resolves every bare attr in the literal on macOS too, where these Linux-only packages do not exist (devenv.nix:127-144). At the current devenv.lock nixpkgs pin these resolve tocloud-hypervisor-52.0,virtiofsd-1.13.3, andpasst-2025_09_19.623dbf6(verified bynix evalagainst nixpkgs, 2026-08-22 — re-resolve against the repo’s own pin at implementation). Version floors are the V5 preflight’s job (“binary versions”,microvm-runner.md:536-539), not the shell’s: the shell provides one pinned version;VerifyMicroVMSupportenforces the floor wherever the Runner runs.passt, not gvproxy. D6 named the class (“passt/gvproxy-class”,microvm-runner.md:733-750), not the member; E1 instantiates it aspasstbecause passt is C (no Go runtime in the guest net path), packaged in nixpkgs, and free of the podman-machine coupling gvproxy carries. The choice is one attr; V2a may revisit if its spike finds passt wanting.- Host config (nix-config repo, out of this repo): the truly host-level
bits — the
kvm-intelmodule and/dev/kvmdevice with permissions the invoking uid can open — for the local dev box (and, if the Woodpecker fallback is ever adopted, its runner host). GitHub’s hosted runners handle their own KVM enablement in-workflow (E-D2), so this host declaration is a dev-box/fallback concern, not a GHA one. Note/dev/vhost-vsockis not required: cloud-hypervisor’s hybrid vsock is a userspace AF_UNIX socket, “not required for cloud-hypervisor’s userspace hybrid-vsock socket (D1)” (microvm-runner.md:536-539), so the host obligation is/dev/kvmalone. The dev box already satisfies this (fact above); the host-config task is to make it declared rather than incidental. Preferred permission posture:kvmgroup 0660 + user membership rather than the current world-RW node (decided — E6, OQ3 resolved).
Alternative weighed: everything (including binaries) in the host flake.
Rejected: it would make the VMM version a per-host fact instead of a
repo-pinned one — CI and every developer must resolve the same
cloud-hypervisor derivation the V5 preflight’s floor was tested against, and
devenv.lock is the only pin both sides share (the same one-pin argument the
shell already makes for postgres, devenv.nix:99-113).
E-D2 — GHA runs the KVM tests as a required leg on the free ubuntu-latest job (F2)
Section titled “E-D2 — GHA runs the KVM tests as a required leg on the free ubuntu-latest job (F2)”The KVM tests run in GitHub Actions, on the same free ubuntu-latest
gates job compass CI already uses (ci.yml:96) — as a required leg, not a
best-effort skip. This reverses the intuition that hosted runners cannot do
KVM; the 2026 reality is more specific:
- GitHub’s docs classify nested virtualization on hosted runners as “technically possible” but “not officially supported … experimental and done at your own risk” (GitHub Actions hosted-runner docs, verified 2026-08-22). There is no SLA.
- In practice it works, and the ecosystem relies on it. Determinate Systems
builds real KVM-dependent artifacts — NixOS disk images — on the free
ubuntu-latestrunner, unguarded, onpull_request/merge_group/push(DeterminateSystems/nixos-amis/.github/workflows/publish.yml:6-9,64,80-84:runner: ubuntu-latest+nix-installer-action+extra-system-features = kvm). Note the KVM build runs on PRs too, in an empty environment;environment: production(publish.yml:54) attaches only on push-to-main and gates the AWS upload credentials, not the KVM build — so the KVM build is unguarded even on PRs. Their Nix installer action enables/dev/kvmby default (“if the host supports it”,nix-installer-action/action.yml), degrading cleanly to a logged “KVM is not available” when it cannot rather than failing the step (nix-installer-action/src/index.ts:530-540,862-967). Where they use larger runners for VM work it is for cores/RAM on heavy NixOSvm-testbatteries (nix-installer/ci.yml:502-510), not because free-runner KVM is unreliable.
So compass treats free-runner KVM the way the ecosystem does: assume it is
present and require it. The rare run where GitHub withholds /dev/kvm reds
the leg; a re-run clears it. If that proves frequent, the Woodpecker fallback
(E-D3) is ready. Mechanism:
- KVM enablement step. Before the KVM suites, the
gatesjob runs the udev group-perms step the ecosystem standardizes on —KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"into/etc/udev/rules.d/, thenudevadm control --reload-rules+udevadm trigger --name-match=kvm— so the runner uid canopen(2)/dev/kvm. This is the same rule the Determinate action writes (nix-installer-action/src/index.ts:883-957); compass installs Nix viacachix/install-nix-action(ci.yml:150), so it writes the rule itself rather than adopting the Determinate action (a larger CI change; see Alternatives). The world-RWMODE="0666"here is deliberate and distinct from E-D1/E6’s tighterkvm-group 0660 for the dev box: a GHA runner is an ephemeral single-tenant throwaway where a world-RW/dev/kvmis harmless, whereas the persistent dev box warrants the group posture. - Build tag
microvm. Every boot/integration test (V2a’s boot spike, V2b’s microVM contract rows, V3/V5/V6/V7/V8’s KVM suites) is build-tagged//go:build microvm && unix, mirroring thepgtestconvention exactly: “The suites are build-taggedpgtest, so the moon battery’sgo test ./...never compiles them” (ci.yml:42-44; existing tags at e.g.go/internal/auth/harness_pgtest_test.go:1). The mooncompass-go:testtask (go test -race ./...,go/moon.yml:151-161) therefore never compiles the KVM tests, and the one-job moon battery is unchanged. The KVM suites run in a dedicated job step with-tags microvm, exactly as the pgtest suites run in their own “Real-Postgres suites” step after the battery (ci.yml:256-282) — inside the onegatesjob, no new job, no matrix. - Runtime gate inside the tag. Tagged tests call a shared helper (new
package
go/internal/microvmtest) that skips when/dev/kvmis not openable — and turns that skip into a hard failure whenCOMPASS_REQUIRE_MICROVM=1is set, mirroringpgtest.RequireDSN+COMPASS_REQUIRE_LIVE(“a skip would silently pass the suite without exercising anything”,go/internal/pgtest/pgtest.go:62-68,99-111). The GHA step setsCOMPASS_REQUIRE_MICROVM=1so a KVM-less run reds rather than silently passing — the same posture the pgtest step takes withCOMPASS_REQUIRE_LIVE=1(ci.yml:268), including its companion “assert-it-ran-rather-than-skipped” guard (ci.yml:284-300). - Every-PR exposure. The KVM step, like the pgtest step, runs on every PR
regardless of affected-detection (it is a job step, not an affected-gated moon
task), so a docs- or UI-only PR that touches no Go still pays the enable +
substitute + boot cost, and a rare withheld-
/dev/kvmred can block any PR — cleared by a re-run. This is the same all-PR exposure the pgtest service-container step already carries; it is a deliberate ONE-JOB consequence, not new risk this record introduces. - What the moon battery still verifies: the untagged tier — V1’s selection
tests, V2b’s contract-suite rows against the podman backend and fakes, V4’s
hermetic gateway suites “over … a vsock-shaped
net.Listenerfake where CI lacks the VMM” (microvm-runner.md:528-529), V5’s preflight unit tests with fake probes (microvm-runner.md:550-551). The tagged tier is compiled and run by the dedicated KVM step above.
Alternatives weighed:
- Best-effort skip on GHA (soft-gate). Run the KVM suites when
/dev/kvmhappens to be present, skip otherwise, with a nightly hard sweep. Rejected in favor of the assume-KVM required posture: the evidence (Determinate shipping production artifacts unguarded on free runners) says KVM is present the vast majority of the time, so a soft gate trades near-certain per-PR coverage for the small cost of an occasional re-run — and a soft gate lets a real regression merge on the run where KVM was absent. If reds from withheld KVM become frequent, revisit this or the Woodpecker fallback. - Larger GHA runners. KVM is officially supported there, but GitHub docs are explicit that larger runners are “always billed at the per-minute rate” for public repos too (GitHub larger-runner docs, verified 2026-08-22). Rejected: the free runner works; pay only if it stops.
- Switching compass CI to
DeterminateSystems/determinate-nix-action(which enables KVM by default). Rejected for this record: it swaps the repo’s nix installer wholesale — a bigger CI change than adding one udev step, and one that touches every job’s nix bootstrap. The udev step is the minimal change; the action swap is a separate proposal if wanted later.
E-D3 — Woodpecker on compass as the documented fallback (F3)
Section titled “E-D3 — Woodpecker on compass as the documented fallback (F3)”If hosted-runner KVM proves unreliable in practice — frequent reds from
withheld /dev/kvm, or GitHub tightening access — the fallback is to enroll
the compass repo itself on the self-hosted Woodpecker fleet and run the KVM
leg there. This is explicitly compass’s own CI, not the private monorepo’s:
compass features are tested in compass. The fleet has a registered self-hosted
Linux CI agent (x86_64, KVM-capable, docker backend) already, so the hardware
is in place; what is missing is a compass pipeline (no .woodpecker* config
exists in this repo — verified). The step shape:
microvm-tests: image: <a nix-capable CI image> devices: - /dev/kvm environment: COMPASS_REQUIRE_MICROVM: "1" commands: - nix build .#compass-guest-kernel .#compass-guest-rootfs # substituted, not rebuilt (E-D4) - cd go && go test -tags microvm -race ./...The fleet runs the docker backend (confirmed) with the repo marked trusted,
so the per-step devices: [/dev/kvm] mapping grants exactly that one device —
never privileged: true. The contract this fallback fixes is: the step grants
/dev/kvm (and only /dev/kvm — /dev/vhost-vsock is not needed, per E-D1),
sets COMPASS_REQUIRE_MICROVM=1 so skips fail loudly, and realizes the
guest-image derivations from the binary cache before the test run. This is the
same env + tag contract E-D2’s GHA step uses, so adopting the fallback is a
lane swap, not a redesign. It also realizes V8’s “CI job (KVM-labeled runner)”
(microvm-runner.md:606-608).
Separate concern — the managed service: the managed Compass will run from the private monorepo, and that monorepo will need its own KVM CI when the managed buildout starts. That is deferred to the managed-service control-plane work (the fleet control plane, RIG-2485), not this record — this record keeps compass’s own tests in compass.
E-D4 — guest image: derivations beside the agent image, kernel from nixpkgs, a project binary cache (F4)
Section titled “E-D4 — guest image: derivations beside the agent image, kernel from nixpkgs, a project binary cache (F4)”V2a produces compass-guest-kernel and compass-guest-rootfs
(microvm-runner.md:456-460); this record fixes where they live, what they
reuse, and how CI avoids rebuilding them:
- Placement: a standalone
guest-image/directory, sibling ofagent-image/, exposing the two attrs from adefault.nix. Not a container build — no devenv container module involved — but the same standalone-not-a-moon-project postureagent-image/established, plus a thincompass-guest-imagemoon project whosecitasknix builds both attrs withinputsglobs scoped toguest-image/**(+ the reused agent-image inputs), so a guest-image PR pays the build pre-merge and every other PR skips it via the affected gate (ci.yml:25-36) — the same pre-merge-build-gate move the agent image made (compass-agent-image-publish.md:179-181). - Kernel: the root
devenv.lock-pinned nixpkgs kernel (pkgs.linuxPackages’ bzImage, direct-kernel boot — cloud-hypervisor boots an uncompressed/bzImage kernel directly, no bootloader). This makes the kernel free in CI: it is substituted from cache.nixos.org, never built. A size/boot-time-optimized custom config is a deliberate deferral (non-load-bearing Open Question) — Q-budget’s measurements (microvm-runner.md:839-842) will say whether it matters. - Rootfs: reuse the agent image’s toolchain closure.
agent-image/already assembles exactly the payload the guest needs — the toolchain derivation + the bundled agent entrypoint (agent-image/devenv.nix:34-36,agent-image/toolchain.nix) with the agent uid-1000 identity (agent-image/devenv.nix:83-101).compass-guest-rootfsimports that sametoolchain.nix, passing root’spkgs— so the rootfs pins to the rootdevenv.lock(the E3 inputs pin), diverging from agent-image’s own nixpkgs pin, a parity note V2a must honor (one closure, two artifact shapes: OCI layers there, a block/filesystem image here — the exact format (erofs vs ext4) and the writable-/nixmechanism the toolchain’s in-guestnixneeds (agent-image/toolchain.nix:47-53) are V2a’s to decide, not this record’s) and adds what the guest additionally needs: thecompass-guestdbinary as init, the egress prerequisites — “nft,getent,awk, and a writable/etc/resolv.conf” (microvm-runner.md:446-449) — and the virtio-fs mount unit for the stable session path. - Caching: a self-hosted nix binary cache.
ci.ymlalready names its substituters explicitly and refusesaccept-flake-configprecisely so cache trust lives in a reviewed file (ci.yml:156-168). The self-hosted fleet already runs a nix binary cache (attic, S3/R2-backed) for its own first-party closures, with a public/main push path separated from a PR-scoped internal one and no push token ever handed to a fork PR. This record reuses that cache rather than standing up hosted cachix: extend the sameextra_nix_configblock with that cache’s public substituter + trusted key so the GHA job pulls the rootfs pre-built (the kernel needs none —cache.nixos.org). Population stays a main-only workflow (push: branches: [main]+ path filter +workflow_dispatch, the least-privilege separate-workflow shape the agent-image publish lane established,compass-agent-image-publish.md:139-191; the push token is a repo secret PR events never see). If the rootfs is small enough to build inside the KVM step under budget, the cache is an optimization rather than a hard dependency — a call V2a’s build-mechanism spike settles.
Global Constraints
Section titled “Global Constraints”- ONE JOB, NOT A MATRIX — inviolable. No new GHA job, no matrix entry, no
project/task enumeration in workflow YAML (
ci.yml:4-23). The KVM suites run as steps within the existinggatesjob, exactly as the pgtest suites do (ci.yml:256-300) — never a new job or matrix leg. The tagged tests stay out of the moon battery by build tag; they enter CI as a dedicatedgo test -tags microvmstep, not a workflow conditional. Any new publish/cache-push lane is a separate least-privilege workflow, never a widened gate token (compass-agent-image-publish.md:139-191). - The two-tier tag rule. Anything that opens
/dev/kvm, spawns cloud-hypervisor/virtiofsd/passt, or requires the guest image is tagged//go:build microvm && unixand callsmicrovmtest.Require(t)first. Untagged tests MUST run on a KVM-less box. A tagged test MUST self-skip where KVM is absent and MUST hard-fail underCOMPASS_REQUIRE_MICROVM=1(thepgtest/COMPASS_REQUIRE_LIVEposture,go/internal/pgtest/pgtest.go:62-68). The GHA KVM step sets the require flag. - One nix pin. VMM/virtiofsd/passt versions and the guest kernel come from
the
devenv.lock-pinned nixpkgs (or repo-pinned inputs) — never a per-host install,go install, or GHAsetup-*action. Version floors are enforced at runtime by V5’sVerifyMicroVMSupport, not by the shell. - Cache trust lives in reviewed files. Substituters + trusted public keys
are named in
ci.yml’sextra_nix_config;accept-flake-configstays off (ci.yml:156-168). Adding a cache is a workflow-file change, reviewed as such. - No secrets on PR events. The cache-push credential lives only in the main-only push workflow; PR runs (including fork PRs) are pull-only against public caches. The KVM enablement step needs no secret.
- Toolchain-parity gate discipline. Linux-only packages are appended
outside the parsed
with pkgs; [ … ]literal indevenv.nix(devenv.nix:127-144); nothing here may break the parity gate’s macOS resolution. - Rootless everywhere. No CI step or dev instruction may require root
beyond the one-time host-config declaration (E-D1) and the GHA KVM-enable
step (which uses the runner’s passwordless sudo, not the test processes’); the
test processes run as the invoking user, matching the backend’s own
constraint (
microvm-runner.md:379-385). - This record adds no product behavior. It changes packaging, CI, and test
scaffolding only; the microVM backend’s own contracts stay owned by
microvm-runner.mdV1-V8.
Ordered by dependency. E1+E2 unblock local development of V2a immediately; E3-E5 land the CI legs; E6 is the host declaration tracked here but executed in its owning repo.
E1 — dev shell: VMM toolset
Section titled “E1 — dev shell: VMM toolset”Add cloud-hypervisor, virtiofsd, and passt to the compass dev shell as
Linux-only appends, with a rationale comment following the xvfb-run/chromium
pattern (devenv.nix:127-144).
- Interfaces: produces
cloud-hypervisor,virtiofsd,passton the dev shell PATH (nix attrspkgs.cloud-hypervisor,pkgs.virtiofsd,pkgs.passt, appended vialib.optionals pkgs.stdenv.isLinux). Consumes nothing new. Deliberately excluded from the toolchain-parity attrs (Linux-only, same as chromium); the KVM CI step realizes them out-of-band from the same pin (E5), exactly as the chromium e2e step does (ci.yml:213-224). - Test cycle:
devenv shellon Linux resolves all three binaries (command -v);bun tools/toolchain/parity.tsstays green; on macOS the shell still evaluates (parity gate unbroken).
E2 — microvmtest gate package
Section titled “E2 — microvmtest gate package”The shared runtime gate the tag rule requires: go/internal/microvmtest with
Require(t *testing.T) — skips with a named reason when /dev/kvm is not
openable by the current uid, fails when COMPASS_REQUIRE_MICROVM=1, and
locates the VMM/virtiofsd binaries and guest-image paths from env with PATH
fallbacks.
- Interfaces: produces
microvmtest.Require(t *testing.T) EnvwithEnv{KernelImage, RootfsImage, VMMPath, VirtiofsdPath string}; env varsCOMPASS_REQUIRE_MICROVM,COMPASS_TEST_GUEST_KERNEL,COMPASS_TEST_GUEST_ROOTFS(defaulting to thenix buildresult symlinks); themicrovmbuild-tag convention documented in the package comment. Consumed by every V2a+ integration test. The pure decision function is split from the*testing.Tdispatch for unit-testing, mirroringdecideDSNSource(go/internal/pgtest/pgtest.go:81-97). - Test cycle: untagged unit tests over the decision function (absent device → skip; absent + require → fail; present → proceed); the package compiles untagged so the moon battery type-checks it everywhere.
E3 — guest-image derivations + moon build gate
Section titled “E3 — guest-image derivations + moon build gate”guest-image/default.nix exposing compass-guest-kernel (pinned-nixpkgs
bzImage) and compass-guest-rootfs (agent-image toolchain closure +
compass-guestd init + nft/getent/awk + writable /etc/resolv.conf +
virtio-fs mount of the stable session path), plus the compass-guest-image
moon project gating the build pre-merge.
- Interfaces: produces nix attrs
compass-guest-kernel,compass-guest-rootfs(the V2a interface,microvm-runner.md:456-460— V2a consumes these instead of inventing its own); importsagent-image/toolchain.nixunchanged (agent-image/devenv.nix:34-36); producesguest-image/moon.ymlwith acitask (nix buildboth attrs) andinputsglobs covering the full rootfs closure —guest-image/**,agent-image/toolchain.nix,agent-image/entrypoint.nix,tools/toolchain/toolchain-tools.nix,tools/toolchain/versions/**(the repo-pinned bun the closure resolves through,agent-image/toolchain.nix:45),packages/compass-agent/**,package.json,bun.lock, plusdevenv.lock+devenv.yaml.agent-image/toolchain.nixis a{ pkgs, ... }function (agent-image/toolchain.nix:35-36), soguest-image/default.nixcalls it with root’spkgs: the kernel and the whole rootfs closure resolve from the rootdevenv.lockpin (the one in this inputs list), not agent-image’s own pin — a deliberate divergence V2a must honor for parity. A glob that misses one of these silently skips a build that should have run (ci.yml:29-33) and leaves the E4 cache stale — the same closure-approximation the publish lane documents (compass-agent-image-publish.md:198-217). The final glob set is contingent on V2a’s image-shape/build-mechanism choice (erofs vs ext4; whether theforks/**inputs the publish lane lists enter the rootfs closure). - Test cycle:
nix build ./guest-image#compass-guest-kernel #compass-guest-rootfssucceeds on Linux;moon run compass-guest-image:cigreen; an unrelated-path PR does not schedule it (affected check). Boot verification itself is V2a’s job, not this task’s. - Sequencing note: lands with a stub
compass-guestd(V2a grows the real one); the derivation shape and the moon gate are this record’s, the boot behavior is V2a’s.
E4 — binary cache substituter + main-only push lane
Section titled “E4 — binary cache substituter + main-only push lane”The self-hosted cache the GHA job substitutes the rootfs from, and the workflow that populates it.
- Interfaces: adds the cache name + public key to
ci.yml’sextra_nix_config(ci.yml:165-168, one reviewed line each forextra-substituters/extra-trusted-public-keys); produces.github/workflows/push-guest-image-cache.yml(push: branches: [main]with the E3 path filter +workflow_dispatch;contents: readonly; the push token as a repo secret), building both attrs and pushing the closure. Consumes E3’s attrs. - Test cycle: a
workflow_dispatchrun populates the cache; a follow-up CI run’s log shows the rootfs substituted, not built; a fork-PR-shaped run (no secret) still substitutes read-only.
E5 — GHA KVM leg: enablement step + tagged suites in the gates job
Section titled “E5 — GHA KVM leg: enablement step + tagged suites in the gates job”The primary KVM test lane: two steps added to the existing ubuntu-latest
gates job (ci.yml), mirroring the pgtest step pair (ci.yml:256-300).
- Interfaces: produces (1) a “Enable KVM” step writing the udev group-perms
rule +
udevadm reload/trigger(E-D2); (2) a “microVM suites” step (working-directory: go) that realizes E3’s attrs from E4’s cache, puts the E1 VMM binaries on PATH out-of-band (the chromium-e2e pattern,ci.yml:213-224), and runsgo test -tags microvm -race -timeout <n>m ./...withCOMPASS_REQUIRE_MICROVM=1+ the capture-then-exit redirect the pgtest step uses (ci.yml:271-282); (3) an “assert the microVM suites ran rather than skipped” guard deriving its skip-text + package-count from source, the pgtest guard’s exact shape (ci.yml:284-300). Consumes E2’s env contract, E3’s attrs, E4’s cache. - Test cycle: on a PR the step runs the tagged suites KVM-backed and the
assert-ran guard proves they executed (not skipped); a deliberately broken
boot test reds the step; a run where GitHub withholds
/dev/kvmreds via the require flag (the accepted rare cost, E-D2).
E6 — host KVM declaration (nix-config repo)
Section titled “E6 — host KVM declaration (nix-config repo)”Declare what today is incidental: KVM enablement + device permissions on the local dev box (and the Woodpecker runner host if the fallback is adopted), in the host nix config. GHA handles its own KVM in-workflow (E5), so this is a dev-box/fallback concern.
- Interfaces: produces host-config settings:
kvm-intelmodule,/dev/kvmudev rule (kvmgroup, 0660 recommended — OQ3) + user/agent membership inkvm. Explicitly NOT produced:/dev/vhost-vsockenablement (not needed — hybrid vsock is userspace,microvm-runner.md:536-539, contingent on V2a’s spike confirming) or any VMM binary (E1 owns binaries). - Test cycle: after converge, the invoking uid opens
/dev/kvm(the exact check V5’s preflight performs,microvm-runner.md:536); localgo test -tags microvmboots a VM on the converged dev box.
- E1 — dev shell: cloud-hypervisor + virtiofsd + passt (Linux-only appends)
- E2 —
go/internal/microvmtestgate package +microvmbuild-tag convention - E3 —
guest-image/derivations +compass-guest-imagemoon build gate - E4 — binary cache substituter + main-only push workflow +
ci.ymllines - E5 — GHA KVM leg: enable-KVM step + tagged
go test -tags microvmsuites + assert-ran guard in thegatesjob - E6 — host KVM declaration: device/module/group in nix-config (dev box + Woodpecker-fallback host)
Open Questions
Section titled “Open Questions”Resolved (2026-08-22 — grounding the CI reality and the self-hosted fleet):
- OQ1 — where the KVM tests run → RESOLVED (E-D2). GitHub Actions, on the
free
ubuntu-latestjob, as a required leg (assume-KVM, like Determinate’s production NixOS-image build). Woodpecker on compass is the documented fallback (E-D3) if hosted KVM proves unreliable; not the primary. - OQ3 —
/dev/kvmpermission posture on the dev box → RESOLVED (E6). The local/self-hosted boxes run testing environments only (production runs on hyperscalers), so E6 declareskvm-group0660+ explicit membership rather than the incidental world-RW node. (GHA runners set their own perms via the E5 udev step.) - OQ4 — cache hosting → RESOLVED (E-D4). The self-hosted fleet already runs
a nix binary cache (attic, S3/R2-backed) for its first-party closures, with a
public/main push path separated from a PR-scoped internal one and no push
token ever handed to a fork PR. This record reuses that cache rather than
standing up hosted cachix; E-D4/E4 wire its public substituter + key into
ci.yml.
Non-load-bearing (documented deferrals):
- OQ2 — Woodpecker-on-compass pipeline shape. Only relevant if the E-D3
fallback is adopted: the exact
.woodpeckerpipeline file + how it reuses E4’s cache config. Deferred until the fallback is triggered; the step contract (E-D3) is fixed. - OQ5 — custom guest kernel config. The pinned-nixpkgs kernel is
substituted free but generic; a minimal virtio-only config would cut boot
time and rootfs closure. Defer until Q-budget’s V2a/V8 measurements
(
microvm-runner.md:839-842) show the generic kernel misses the budget. - OQ6 — nightly KVM-sweep need. With the KVM suites running on every compass
PR (E-D2), a separate nightly sweep is unnecessary; the “CI (full sweep)”
step (
ci.yml:245-253,moon run :cigatedif: github.event_name != 'pull_request') already re-runs the whole battery on each landing to main and on the nightly schedule (ci.yml:62-66). Revisit only if the assume-KVM posture is downgraded to best-effort.