m³OS docs GitHub

Phase 43

Crash Diagnostics

Every kernel crash is self-diagnosing. When the kernel panics or takes a fatal fault, the serial output includes full CPU register state, current task metadata, and per-core scheduler context. `debug_assert!` invariants at every scheduler, fork, IPC, and stack boundary catch state corruption at the point of origin rather than surfacing later as an opaque `RIP=0x4` fault.

Complete

Learning goal

Understand what information a kernel crash dump should contain and why each field matters (registers, task state, per-core run queues).

Category

infrastructure

Why this phase matters

Every kernel crash is self-diagnosing. When the kernel panics or takes a fatal fault, the serial output includes full CPU register state, current task metadata, and per-core scheduler context. debug_assert! invariants at every scheduler, fork, IPC, and stack boundary catch state corruption at the point of origin rather than surfacing later as an opaque RIP=0x4 fault.

How it works in m3OS

Phase 43a makes every kernel crash self-diagnosing. The panic handler and all fault handlers now print full CPU register state, current task metadata, and per-core scheduler context. debug_assert! invariants at scheduler, fork, IPC, and stack boundaries catch state corruption at the point of origin rather than surfacing later as an opaque fault.

How it builds on earlier phases

This phase builds on SSH Server and turns that foundation into Crash Diagnostics.

Implementation outline

  • Create kernel/src/panic_diag.rs with dump_crash_context() (registers, task info, per-core state).
  • Update the panic handler in kernel/src/main.rs to call dump_crash_context().
  • Add try_lock_scheduler() and Scheduler::get_task() for safe non-blocking scheduler access.
  • Enrich page fault, GPF, and double fault handlers to call dump_crash_context() with additional context.
  • Add debug_assert! checks in pick_next, yield_now, block_current, wake_task, run, and enqueue_to_core.
  • Add debug_assert! checks in make_fork_ctx, fork_child_trampoline, sys_fork, and spawn_fork_task.
  • Add debug_assert! checks in recv_msg, send, and reply.
  • Add Task::stack_bounds() and stack validation assertions in run.
  • Verify cargo xtask check and cargo xtask test pass.

Task progress snapshot

63 of 63 tracked tasks are complete.

Documentation targets

  • Documentation deliverables have not been listed yet.

Deferred follow-on work

  • Full lockdep-lite checker (Phase 43b or beyond)
  • Allocator poisoning / redzones (KASAN-style, Phase 43b or beyond)
  • Register dump via NMI (requires NMI handler work)
  • Backtrace / stack unwinding (requires frame pointer chain or DWARF unwinder)
  • Structured crash dump to disk for post-mortem analysis

Source materials

Important code

Code spotlight cards

Code spotlight

Dump CPU registers on panic

kernel/src/panic_diag.rs L56-L79

Dump CPU registers on panic. Why it matters: The previous panic handler printed only the file, line, and message; without register state, the developer cannot determine what value was in RSP or any GPR at the point of failure. Focus on `capture_registers` and `_panic_print` in `kernel/src/panic_diag.rs` while you trace the snippet.

/// Read all general-purpose registers, RFLAGS, CR2, and CR3.
///
/// Note: RIP cannot be directly captured from inline assembly — the values
/// here reflect the state at the point of capture inside this function, not
/// at the original panic site. The panic location printed by the caller
/// (file:line) is the best proxy for the faulting instruction.
fn capture_registers() -> RegisterSnapshot {
    let rax: u64;
    let rcx: u64;
    let rdx: u64;
    let rsi: u64;
    let rdi: u64;
    let rsp: u64;
    let r8: u64;
    let r9: u64;
    let r10: u64;
    let r11: u64;
    let r12: u64;
    let r13: u64;
    let r14: u64;
    let r15: u64;
    let rflags: u64;

    // Capture all GPRs in a single asm block.  rbx/rbp are LLVM-reserved and

Code spotlight

Enrich page fault handler (userspace)

kernel/src/arch/x86_64/interrupts.rs L350-L373

Enrich page fault handler (userspace). Why it matters: The userspace page fault handler prints PID, address, error code, and RIP, but not RSP, task state, scheduler state, or per-core context — exactly the information needed to diagnose the RIP=0x4 crash. Focus on `page_fault_handler` and `RIP=0x4` in `kernel/src/arch/x86_64/interrupts.rs` while you trace the snippet.

extern "x86-interrupt" fn page_fault_handler(
    mut stack_frame: InterruptStackFrame,
    err: PageFaultErrorCode,
) {
    let addr = x86_64::registers::control::Cr2::read();

    // Check if the fault came from ring 3 (user mode).
    if stack_frame.code_segment.rpl() == x86_64::PrivilegeLevel::Ring3 {
        // P17-T031: detect CoW faults — a write to a present, non-writable
        // page marked with BIT_9 (the CoW marker set by cow_clone_user_pages).
        let is_write = err.contains(PageFaultErrorCode::CAUSED_BY_WRITE);
        let is_present = err.contains(PageFaultErrorCode::PROTECTION_VIOLATION);
        if is_write
            && is_present
            && let Ok(fault_vaddr) = addr
        {
            let fault_addr_u64 = fault_vaddr.as_u64();
            if has_cow_marker(fault_addr_u64) {
                // CoW fault — resolve directly in the ISR. Safe because
                // the fault is from ring 3 (no kernel locks held) and
                // we're on a single CPU. On OOM, fall through to kill.
                if resolve_cow_fault(fault_addr_u64) {
                    return;
                }

Code spotlight

Assertions in `pick_next`

kernel/src/task/scheduler.rs L138-L161

Assertions in `pick_next`. Why it matters: pick_next selects and returns a task's saved_rsp for dispatch; a stale or zero saved_rsp causes the exact RIP=0x4 crash pattern observed in production. Focus on `pick_next` and `saved_rsp` in `kernel/src/task/scheduler.rs` while you trace the snippet.

    /// Pick the next task to run on the given core.
    ///
    /// First checks the per-core run queue for an O(1) dequeue. Falls back
    /// to a global round-robin scan if the local queue is empty (handles
    /// tasks that haven't been assigned to a queue yet). Finally, falls back
    /// to this core's idle task.
    fn pick_next(&mut self, core_id: u8) -> Option<(u64, usize)> {
        let core_bit = 1u64 << core_id;

        // Scan local run queue: find the highest-priority (lowest numeric) Ready
        // task in a single pass, then remove only that entry. No heap allocation.
        if let Some(data) = crate::smp::get_core_data(core_id) {
            let mut q = data.run_queue.lock();
            let mut best_pos: Option<usize> = None;
            let mut best_prio: u8 = u8::MAX;

            // First pass: discard stale entries from the front while scanning.
            let mut i = 0;
            while i < q.len() {
                let idx = q[i];
                if idx >= self.tasks.len()
                    || self.tasks[idx].state != TaskState::Ready
                    || self.idle_tasks.contains(&Some(idx))
                    || self.tasks[idx].affinity_mask & core_bit == 0

Key files

  • kernel/src/main.rs — Panic handler calls dump_crash_context()
  • kernel/src/arch/x86_64/interrupts.rs — Enriched page fault, GPF, double fault handlers
  • kernel/src/panic_diag.rs — Crash diagnostics module: register capture, task dump, per-core dump
  • kernel/src/task/mod.rs — try_lock_scheduler(), Task::stack_bounds()
  • kernel/src/task/scheduler.rs — Scheduler assertions (C.1–C.6) and stack validation (F.1–F.3)
  • kernel/src/process/mod.rs — Fork context assertions (D.1–D.2)
  • kernel/src/arch/x86_64/syscall.rs — sys_fork assertions (D.3)
  • kernel/src/ipc/endpoint.rs — IPC assertions (E.1–E.4)

How real OSes differ

  • Linux uses `printk` with structured log levels and a ring buffer (`dmesg`); panic output goes through a dedicated "oops" formatter that includes a full stack trace with DWARF-based symbol resolution.
  • Linux's `BUG_ON()` / `WARN_ON()` macros include file:line and produce a full register dump, similar to this phase's `debug_assert!` + crash context approach but always-on (not debug-only).
  • Production kernels use KASAN (address sanitizer), UBSAN (undefined behavior), and lockdep (lock ordering) -- much heavier instrumentation than the targeted assertions in this phase.
  • Windows uses structured exception handling (SEH) with per-thread exception chains and writes crash dumps (minidump / full dump) to disk for post-mortem analysis.
  • Real kernels support NMI-triggered register dumps across all cores simultaneously; m3OS captures registers only on the faulting core.

Success criteria

  • `dump_crash_context()` prints registers, task info, and per-core state on every panic and fault using the deadlock-safe serial path
  • All four fault handlers (page fault userspace/kernel, GPF, double fault) call `dump_crash_context()` and print additional context (RSP, CR3, error code)
  • `debug_assert!` invariants present at every scheduler, fork, IPC, and stack boundary as specified in the task list
  • `cargo xtask check` passes (clippy clean, rustfmt, 189 host tests)
  • `cargo xtask test` passes (all 8 QEMU kernel tests)
  • No `debug_assert` panics during normal boot

Related phases

Keep exploring the roadmap