Koder Runtime Strategy — kbox as the universal runtime

draft

infraRFC001 — Koder Runtime Strategy

Status

*raft*— opened 20260520 from the kbox 10-wave session. Owner ratifies before Phase 0 work begins.

Summary

Replace Docker, LXC, and (where feasible) QEMUKVM across the Koder Stack with `infranetbox` (kbox) as the universal containerruntime substrate. Three substitution axes with different gaps + phasing per axis. Explicit boundary: QEMU stays for Windows builds, kernel testing, and arbitrary-distro chroots — kbox does not try to be a full hypervisor.

Motivation

Today the Stack runs four different runtimes for overlapping problems:

Tech Where Pain
Docker s.poc.vivver.com (4 containers), CI runners, dev One more dependency to keep current; supply-chain (Docker Inc) outside the Stack
LXC/Incus s.khost1 (5 CIMMG LXCs + cert daemon + Koder VPN hub) Different mental model than Docker; per-host install + CLI culture; not aligned with Koder Stack tooling (no Jet integration, no Koder ID, no shared image layer cache)
QEMU/KVM Win10 VM + klinux chroot + Koda toolchain VM (planned) Necessary for the cases that need it; no Stack-side abstraction over libvirt
(cosign etc.) tooling around runtime —

Convergence on one runtime — kbox — gives uniform multi-tenancy (#108), uniform tracing (#109+#122), uniform OAuth (#110), uniform signing (#114+#121), one set of dashboards, one runbook, one tooling install per host. This RFC says *ow*that convergence happens without losing the cases QEMU genuinely covers.

Three substitution axes

Axis A — Docker → kbox

*ap:*smallest. kbox v1.1.0 has OCI runtime + image builds + registry + compose + CRIU + farm. Waves 110 added the Stacknative pieces (TOML config, multitenancy, tracing, OAuth, signing, OTLP, SBOM, quotas). Pair #13 in `selfhostedpairs.md is at stable` for singlenode; the open gates are operational, not feature.

*hat's missing for production_proven:*real deployment + soak + Hub publication + bench numbers. Code-complete.

Axis B — LXC → kbox

*ap:*medium. LXC runs *ystem containers*(full OS userland + systemd + persistent multiservice). kbox today runs *pplication containers*(singleprocess tree). Translating Incus workloads (5 CIMMG LXCs) requires kbox to grow:

  • kbox run --system (PID 1 = systemd, runs forever, multi-process)
  • kbox attach <id> (shell into running system, distinct from exec <id> <cmd>)
  • kbox profile create/apply (Incus-style reusable config template)
  • kbox snapshot create/restore (disklevel snapshot, distinct from CRIU livemigration)
  • Stateful disk pool UX matching lxc storage create

None of these are blocked technically — they're feature work not yet on the backlog.

Axis C — QEMU/KVM → kbox

*ap:*largest, partially outofscope. Three workload classes use QEMU today:

  1. *indows guest*(Flutter Windows build via k.win WinRM) — kbox cannot run Windows. Hard boundary.
  2. *ernel testing / klinux ISO chroot*— needs separate kernel. Hard boundary unless kbox grows a microVM dispatch.
  3. *trongisolation Linux workloads*(Koda toolchain VM planned per `policies/testhost-isolation.kmd`) — these CAN move to microVM if kbox grows the dispatch.

*ecision: split this axis in two.*

  • *.1 microVM dispatch*— kbox spawns Firecracker / Cloud Hypervisor for Linux workloads that want kernellevel isolation. Same UX as kbox run, just `-runtime=microvm`. Covers the third class above.
  • *.2 permanent QEMU scope*— Windows + kernel testing stay in QEMU forever. Document this as an intentional boundary, not a gap. Open a small infra/net/vm/ module (or extend infra/linux/x or similar) that wraps libvirt/QEMU with the Stack's tooling (Koder ID auth, tracing, OTLP) so the QEMU side at least integrates with the same observability + auth surfaces.

Phasing

Phase Scope Owner-side gate Estimated duration
* — Docker production_proven* Replace Docker on s.poc.vivver.com (4 containers). Land bench numbers in perf-baseline.md. Publish to Hub. First external deployment + 7-day soak ~1-2 months
* — System mode* Add kbox run --system + attach + profile + snapshot. Pilot: replace cert daemon LXC on s.khost1. Cert daemon (already stable per project_certs_deployed_skhost1) runs in kbox-system for 7 days without regression ~2-3 months
* — LXC at scale* Migrate CIMMG LXCs (db, app, neogestor, tv-fila, esus) + Koder VPN hub. All 6 LXCs migrated; Incus dependency removable from s.khost1 (kept as fallback) ~1-2 months
* — MicroVM dispatch* Add Firecracker backend to kbox. kbox run --runtime=microvm. Target: Koda toolchain VM per test-host-isolation.kmd. Toolchain VM runs in kbox/Firecracker without regression vs raw QEMU ~3-4 months
* — QEMU boundary* Document permanent QEMU scope (Windows + kernel testing). Open infra/net/vm/ module wrapping libvirt with Stack tooling (auth + tracing + OTLP). Win10 VM + klinux ISO chroot still in QEMU but tracing pipes into the same Tempo as kbox containers ~1 month

Total elapsed: ~8-12 months, depending on parallelism with other Stack waves.

Per-phase implementation tickets

Opened against infra/net/box/backlog/pending/:

  • *128*Phase 0 — Docker production_proven activation (deploy s.poc.vivver.com, soak, bench, Hub publish)
  • *129*Phase 1 — System container mode (--system + attach + profile + snapshot)
  • *130*Phase 2 — CIMMG + cert-daemon LXC migration
  • *131*Phase 3 — Firecracker microVM dispatch backend
  • *132*Phase 4 — infra/net/vm/ module bootstrap for permanent QEMU scope

Out of scope

  • *indows guest in kbox.*No. Stays in QEMU.
  • *eplacing Incus itself.*Incus stays installed on s.khost1 as fallback during Phase 1-2 migration; removable only after Phase 2 completion.
  • *ross-architecture emulation*(arm guest on amd host etc.). Edge case — handled by QEMU when needed.
  • *ive container migration across hosts.*kbox has CRIU localhost migration already; crosshost depends on kdb cluster state (kbox#115, blocked on kdb#414).

Open questions

Q1 — System-mode default for attach

LXC's attach enters the container as root with a fresh login shell; Docker's exec runs a new command in an existing namespace. For kbox attach, do we mimic LXC (fresh-shell semantics) or just alias to exec /bin/sh?

*ecommendation:*Mimic LXC (fresh login session, reads /etc/profile). Operators migrating from lxc shell expect this. Add kbox exec for the Docker-style behaviour as already shipped.

Q2 — Single Stackwide runtime config or perhost?

Today koder-box.toml is perhost. For multihost Stack management (Koder Console fleet view), do we want one global runtime policy distributed via kdb, or stay per-host?

*ecommendation:*Stay perhost for Phases 02. Revisit when kbox#115 (cluster state) ships.

Q3 — Firecracker vs Cloud Hypervisor vs Kata

Three opensource microVM options. Tradeoffs:

  • *irecracker*— minimal, AWSbacked, welldocumented; Linux x86_64 + arm64 only
  • *loud Hypervisor*— Intel-backed, similar surface, slightly heavier
  • *ata Containers*— full OCI-shim layer over Firecracker/CH; more abstraction, less control

*ecommendation:**irecracker* Smallest surface area, longest production track record, simplest integration. Reevaluate if a specific limitation bites (no GPU passthrough is the mostcited; if Koda toolchain testing needs GPU we'd revisit).

Q4 — Incus removal threshold

After Phase 2, when is it safe to apt remove incus from s.khost1? Risk: a future LXC need surfaces that kbox-system hasn't covered yet.

*ecommendation:*Keep Incus installed but *nactive*(no running containers) for 90 days postmigration. Hard removal after 90 days with no rollback request. Document the rollback procedure (recreate containers from snapshots) in infra/net/box/docs/migration-from-incus.kmd.

Acceptance criteria for this RFC

  • Owner ratifies → Phase 0 work starts on s.poc.vivver.com
  • 5 tracking tickets opened in infra/net/box/backlog/pending/
  • registries/self-hosted-pairs.md row 13 references this RFC in the notes column
  • meta/docs/stack/modules/box.md gains a "Runtime strategy" section pointing at this RFC + the phasing

Cross-references

  • registries/self-hosted-pairs.md row 13 (Koder Box) — current gate state
  • policies/self-hosted-first.kmd — the gates this RFC closes
  • policies/test-host-isolation.kmd — Phase 3 / 4 boundary (test workloads in VMs)
  • infrastructure/servers.md — list of hosts using each runtime today
  • infra/data/kdb#414 — blocks Phase 2.5 multi-host clustering (kept separate from this RFC)

Source: ../home/koder/dev/koder/meta/docs/stack/rfcs/infra-RFC-001-koder-runtime-strategy.kmd