m³OS docs GitHub

Phase 43

Kernel Trace Ring

Every kernel crash includes a structured timeline of recent scheduler, fork, and IPC events. The per-core lockless trace ring records events with timestamps, and the auto-dump on panic/fault merges all cores into a single sorted timeline in the serial output. The ring is host-testable, accessible from userspace, and feature-gated for zero-cost release builds.

Complete

Learning goal

Understand lockless per-core ring buffers and why they avoid contention

Category

infrastructure

Why this phase matters

Every kernel crash includes a structured timeline of recent scheduler, fork, and IPC events. The per-core lockless trace ring records events with timestamps, and the auto-dump on panic/fault merges all cores into a single sorted timeline in the serial output. The ring is host-testable, accessible from userspace, and feature-gated for zero-cost release builds.

How it works in m3OS

Phase 43b adds a per-core lockless trace ring that records scheduler, fork, and IPC events with timestamps. When the kernel crashes, the trace ring is automatically dumped alongside the crash diagnostics from Phase 43a, giving developers a structured timeline of the kernel transitions leading into the fault. The ring is host-testable in kernel-core, accessible from userspace via sys_ktrace, and feature-gated so it compiles out for release builds.

How it builds on earlier phases

This phase builds on Crash Diagnostics and turns that foundation into Kernel Trace Ring.

Implementation outline

  • Implement TraceRing<N>, TraceEvent, TraceEntry in kernel-core
  • Add host-side unit tests for the ring buffer
  • Add trace_ring field to PerCoreData (BSP and AP init)
  • Implement trace_event() and dump_trace_rings() in kernel
  • Instrument scheduler paths (6 sites)
  • Instrument fork paths (4 sites)
  • Instrument IPC paths (7 sites)
  • Wire dump_trace_rings() into panic/fault handlers
  • Implement sys_ktrace syscall and syscall-lib wrapper
  • Add feature gate (trace feature on kernel crate)
  • Validate with cargo xtask check and cargo xtask test

Task progress snapshot

0 of 76 tracked tasks are complete.

Documentation targets

  • Documentation deliverables have not been listed yet.

Deferred follow-on work

  • Binary trace format for off-target analysis tooling
  • Userspace ktrace command-line tool
  • Network-accessible trace export
  • Trace filtering (per-subsystem enable/disable)
  • Trace event for timer ISR / reschedule signal
  • TSC-based sub-tick ordering

Source materials

Important code

Code spotlight cards

Code spotlight

Define `TraceRing<const N: usize>` in kernel-core

kernel-core/src/trace_ring.rs L122-L132

Define `TraceRing<const N: usize>` in kernel-core. Why it matters: A lockless per-core ring avoids contention on the hot path; placing it in kernel-core makes it host-testable with standard cargo test just like the existing LogRing. Focus on `TraceRing` and `LogRing` in `kernel-core/src/trace_ring.rs` while you trace the snippet.

/// Per-core lockless circular trace buffer.
///
/// Holds the most recent `N` entries. New entries overwrite the oldest on wrap.
/// No mutex — safe for single-writer (owning core) use.
pub struct TraceRing<const N: usize> {
    buf: [TraceEntry; N],
    write_idx: usize,
    count: usize,
}

impl<const N: usize> TraceRing<N> {

Code spotlight

Per-core trace ring allocation in kernel

kernel/src/smp/mod.rs L56-L79

Per-core trace ring allocation in kernel. Why it matters: Each core needs its own ring to avoid lock contention; the ring must be sized to hold enough events to cover a typical fork+dispatch+fault sequence. Focus on `PerCoreData` and `TraceRing::new` in `kernel/src/smp/mod.rs` while you trace the snippet.

/// Per-core state block.
///
/// Each core has one of these, initialized during BSP init or AP bootstrap.
/// The `gs_base` MSR points to this struct so that `per_core()` can retrieve
/// it in O(1) without MMIO.
#[repr(C)]
pub struct PerCoreData {
    /// Self-pointer at offset 0 — reserved for future `gs:[0]` access.
    /// Currently unused: `per_core()` reads `IA32_GS_BASE` via `rdmsr`.
    self_ptr: *const PerCoreData,
    /// Logical core index (0 = BSP, 1..n = APs in MADT order).
    pub core_id: u8,
    /// LAPIC ID from the MADT.
    pub apic_id: u8,
    /// Set to `true` once this core has completed initialization.
    pub is_online: AtomicBool,
    /// Pointer to this core's TSS (for runtime RSP0 updates).
    tss_ptr: *mut TaskStateSegment,
    /// Pointer to this core's GDT (pre-allocated on BSP, loaded on AP).
    gdt_ptr: *const GlobalDescriptorTable,
    /// Segment selectors for this core's GDT.
    gdt_code: SegmentSelector,
    gdt_data: SegmentSelector,
    gdt_tss: SegmentSelector,

Code spotlight

Trace emit helper

kernel/src/trace.rs L10-L32

Trace emit helper. Why it matters: A single trace_event(event) call site keeps instrumentation minimal (one line per call site) and handles timestamping and core-ID tagging automatically. Focus on `trace_event` and `trace_event(event)` in `kernel/src/trace.rs` while you trace the snippet.

/// Emit a trace event on the current core's trace ring.
///
/// Compiles to nothing when the `trace` feature is off.
#[cfg(feature = "trace")]
pub fn trace_event(event: TraceEvent) {
    if !crate::smp::is_per_core_ready() {
        return;
    }
    let core_id = crate::smp::per_core().core_id;
    let tick = crate::arch::x86_64::interrupts::tick_count();
    // Safety: trace_ring is wrapped in UnsafeCell for interior mutability.
    // Only the owning core writes to its ring (single-writer guarantee via
    // gs_base per-core data). The UnsafeCell makes the mutable access sound.
    let ring_ptr = crate::smp::per_core().trace_ring.get();
    unsafe {
        (*ring_ptr).push(TraceEntry {
            tick,
            core: core_id,
            _pad: [0; 7],
            event,
        });
    }
}

Key files

  • kernel-core/src/trace_ring.rs — TraceRing<N>, TraceEvent, TraceEntry -- host-testable
  • kernel/src/trace.rs — trace_event(), dump_trace_rings()
  • kernel/src/smp/mod.rs — PerCoreData.trace_ring field
  • kernel/src/task/scheduler.rs — Scheduler instrumentation (6 sites)
  • kernel/src/process/mod.rs — Fork instrumentation (2 sites)
  • kernel/src/ipc/endpoint.rs — IPC instrumentation (7 sites)
  • kernel/src/arch/x86_64/syscall.rs — sys_ktrace syscall
  • userspace/syscall-lib/src/lib.rs — ktrace() userspace wrapper

How real OSes differ

  • Linux uses `ftrace` with per-CPU ring buffers backed by `ring_buffer.c`, supporting binary format, filtering, function graph tracing, and live streaming via `/sys/kernel/tracing/`
  • FreeBSD uses `DTrace` (and now also `ktrace`) with kernel-userspace cooperation and sophisticated filtering
  • Production systems use binary trace formats (CTF, perf.data) for efficient storage and offline analysis tools
  • Lock-free ring buffers in production kernels use memory ordering barriers and sequence counters for safe concurrent reader/writer access

Success criteria

  • `cargo test -p kernel-core` passes with all trace ring tests
  • `cargo xtask check` passes with trace feature on and off
  • `cargo xtask test` passes with trace feature on
  • `cargo build -p kernel --no-default-features` compiles without trace code
  • Trace ring dump appears in serial output after crash diagnostics on panic
  • `sys_ktrace` syscall copies trace entries to userspace buffer

Related phases

Keep exploring the roadmap