Phase 43

Regression Stress Ci

A layered test infrastructure that catches SMP race conditions through focused QEMU regression tests, looped stress tests with automatic artifact capture, seed-based iteration for reproducibility (guest-side timing variation deferred), host-side property and concurrency testing for extracted scheduler/fork/IPC models, and CI tier mapping that runs regressions on every PR and stress tests nightly.

Complete

Learning goal

Understand tiered testing strategies (host tests, regression, smoke, stress)

Why this phase matters

How it works in m3OS

Phase 43c adds a layered test infrastructure for catching SMP race conditions. The system has four tiers: host-side property/concurrency tests (proptest, loom), QEMU regression tests for specific SMP-sensitive paths, smoke tests for full-boot validation, and looped stress tests for timing-dependent failures. Phase 53 later turns the automated half of this stack into the published headless/reference gate bundle: cargo xtask check (which already runs the host-side cargo test -p kernel-core --target x86_64-unknown-linux-gnu tier plus cargo test -p passwd --target x86_64-unknown-linux-gnu --no-default-features --features host-tests --test passwd_host in CI), RUSTFLAGS='--cfg loom' cargo test -p kernel-core --target x86_64-unknown-linux-gnu --test allocator_loom, cargo xtask smoke-test --timeout 300, and cargo xtask regression --timeout 90. Nightly cargo xtask stress --test ssh-overlap --iterations 50 --timeout 90 is sustaining evidence rather than a per-PR or final-close rerun.

How it builds on earlier phases

This phase builds on Kernel Trace Ring and turns that foundation into Regression Stress Ci.

Implementation outline

Add regression and stress subcommands to xtask
Implement regression test framework with pass/fail detection
Register fork-overlap, ipc-wake, and pty-overlap regressions
Implement stress test loop with seed and artifact capture
Add proptest dev-dependency and scheduler/fork property tests
Add loom dev-dependency and IPC concurrency tests
Add regression step to PR and build CI workflows
Create nightly stress workflow with artifact upload
Create documentation

Task progress snapshot

0 of 111 tracked tasks are complete.

Documentation targets

Documentation deliverables have not been listed yet.

Deferred follow-on work

Full syzkaller-style syscall fuzzing
QEMU gdbstub integration in xtask
lockdep-lite integration with stress tests
Performance regression benchmarks
Coverage-guided fuzzing for kernel syscall paths
Kernel watchdog timer (per-core, dumps on stuck scheduler)

Source materials

Important code

Code spotlight cards

Add `regression` subcommand to xtask. Why it matters: The existing test command runs kernel unit tests and the smoke-test runs a single-pass boot validation; neither is designed for targeted race-condition regression tests that need specific QEMU configurations and timeout handling. Focus on `regression` and `cmd_regression` in `xtask/src/main.rs` while you trace the snippet.

fn cmd_regression(args: &RegressionArgs) {
    let all_tests = regression_tests();
    let tests_to_run: Vec<&RegressionTest> = if let Some(name) = &args.test_name {
        let found = all_tests.iter().find(|t| t.name == name);
        match found {
            Some(t) => vec![t],
            None => {
                eprintln!("Unknown regression test: {name}");
                eprintln!(
                    "Available: {}",
                    all_tests
                        .iter()
                        .map(|t| t.name)
                        .collect::<Vec<_>>()
                        .join(", ")
                );
                std::process::exit(1);
            }
        }
    } else {
        all_tests.iter().collect()
    };

    println!("regression: running {} test(s)", tests_to_run.len());

View on GitHub

Regression test guest-side binary framework. Why it matters: Regression tests need a guest-side component that exercises the specific code path and prints structured pass/fail markers to serial output. Focus on `regression_harness` in `userspace/regression/` while you trace the snippet.

View on GitHub

Fork overlap guest test. Why it matters: The RIP=0x4 crash occurs during overlapping fork() calls from different processes; this test reproduces the exact pattern by forking rapidly from multiple parents. Focus on `fork_overlap_test` and `RIP=0x4` in `userspace/regression/fork-overlap.c` while you trace the snippet.

View on GitHub

Key files

xtask/src/main.rs — cmd_regression, cmd_stress, test registries
kernel-core/tests/scheduler_props.rs — Proptest scheduler invariants
kernel-core/tests/ipc_loom.rs — Loom IPC concurrency tests
kernel-core/tests/fork_props.rs — Proptest fork queue invariants
.github/workflows/pr.yml — PR CI with regression step
.github/workflows/build.yml — Build CI with regression step
.github/workflows/nightly-stress.yml — Nightly stress workflow

How real OSes differ

Linux uses KernelCI, syzkaller (coverage-guided syscall fuzzing), and kselftest for automated testing at scale
Production stress testing uses thousands of iterations across diverse hardware
CI-integrated fuzz testing (OSS-Fuzz) continuously explores syscall paths
Lock ordering validators (lockdep) run in-kernel rather than as host models
Hardware-specific timing variation (NUMA, different CPU vendors) affects which interleavings are exercised

Success criteria

`cargo xtask check` passes
`cargo xtask test` passes
`cargo test -p kernel-core` passes including proptest tests
`cargo xtask regression` is a valid subcommand
`cargo xtask stress --test fork-overlap --iterations 1` is a valid subcommand
CI workflows updated with regression step and nightly stress schedule

Related phases

Keep exploring the roadmap

Kernel Trace Ring

Every kernel crash includes a structured timeline of recent scheduler, fork, and IPC events. The per-core lockless trace ring records events with timestamps, and the auto-dump on panic/fault merges all cores into a single sorted timeline in the serial output. The ring is host-testable, accessible from userspace, and feature-gated for zero-cost release builds.

Rust Cross Compilation

Rust programs compiled on the host run natively inside the OS. A documented toolchain setup enables writing Rust code on the host, cross-compiling to an x86-64 ELF binary linked against musl, and deploying it to the OS's disk image. Stretch goal: a minimal Rust program compiles inside the OS itself.