Phase 43
Kernel Trace Ring
Every kernel crash includes a structured timeline of recent scheduler, fork, and IPC events. The per-core lockless trace ring records events with timestamps, and the auto-dump on panic/fault merges all cores into a single sorted timeline in the serial output. The ring is host-testable, accessible from userspace, and feature-gated for zero-cost release builds.
Learning goal
Understand lockless per-core ring buffers and why they avoid contention
Category
infrastructure
Components touched
Builds on
Unlocked by this phase
Why this phase matters
Every kernel crash includes a structured timeline of recent scheduler, fork, and IPC events. The per-core lockless trace ring records events with timestamps, and the auto-dump on panic/fault merges all cores into a single sorted timeline in the serial output. The ring is host-testable, accessible from userspace, and feature-gated for zero-cost release builds.
How it works in m3OS
Phase 43b adds a per-core lockless trace ring that records scheduler, fork, and IPC events with timestamps. When the kernel crashes, the trace ring is automatically dumped alongside the crash diagnostics from Phase 43a, giving developers a structured timeline of the kernel transitions leading into the fault. The ring is host-testable in kernel-core, accessible from userspace via sys_ktrace, and feature-gated so it compiles out for release builds.
How it builds on earlier phases
This phase builds on Crash Diagnostics and turns that foundation into Kernel Trace Ring.
Implementation outline
- Implement
TraceRing<N>,TraceEvent,TraceEntryin kernel-core - Add host-side unit tests for the ring buffer
- Add
trace_ringfield toPerCoreData(BSP and AP init) - Implement
trace_event()anddump_trace_rings()in kernel - Instrument scheduler paths (6 sites)
- Instrument fork paths (4 sites)
- Instrument IPC paths (7 sites)
- Wire
dump_trace_rings()into panic/fault handlers - Implement
sys_ktracesyscall and syscall-lib wrapper - Add feature gate (
tracefeature on kernel crate) - Validate with
cargo xtask checkandcargo xtask test
Task progress snapshot
0 of 76 tracked tasks are complete.
Documentation targets
- Documentation deliverables have not been listed yet.
Deferred follow-on work
- Binary trace format for off-target analysis tooling
- Userspace
ktracecommand-line tool - Network-accessible trace export
- Trace filtering (per-subsystem enable/disable)
- Trace event for timer ISR / reschedule signal
- TSC-based sub-tick ordering
Source materials
Important code
Code spotlight cards
Code spotlight
Define `TraceRing<const N: usize>` in kernel-core
Define `TraceRing<const N: usize>` in kernel-core. Why it matters: A lockless per-core ring avoids contention on the hot path; placing it in kernel-core makes it host-testable with standard cargo test just like the existing LogRing. Focus on `TraceRing` and `LogRing` in `kernel-core/src/trace_ring.rs` while you trace the snippet.
/// Per-core lockless circular trace buffer.
///
/// Holds the most recent `N` entries. New entries overwrite the oldest on wrap.
/// No mutex — safe for single-writer (owning core) use.
pub struct TraceRing<const N: usize> {
buf: [TraceEntry; N],
write_idx: usize,
count: usize,
}
impl<const N: usize> TraceRing<N> { Code spotlight
Per-core trace ring allocation in kernel
Per-core trace ring allocation in kernel. Why it matters: Each core needs its own ring to avoid lock contention; the ring must be sized to hold enough events to cover a typical fork+dispatch+fault sequence. Focus on `PerCoreData` and `TraceRing::new` in `kernel/src/smp/mod.rs` while you trace the snippet.
/// Per-core state block.
///
/// Each core has one of these, initialized during BSP init or AP bootstrap.
/// The `gs_base` MSR points to this struct so that `per_core()` can retrieve
/// it in O(1) without MMIO.
#[repr(C)]
pub struct PerCoreData {
/// Self-pointer at offset 0 — reserved for future `gs:[0]` access.
/// Currently unused: `per_core()` reads `IA32_GS_BASE` via `rdmsr`.
self_ptr: *const PerCoreData,
/// Logical core index (0 = BSP, 1..n = APs in MADT order).
pub core_id: u8,
/// LAPIC ID from the MADT.
pub apic_id: u8,
/// Set to `true` once this core has completed initialization.
pub is_online: AtomicBool,
/// Pointer to this core's TSS (for runtime RSP0 updates).
tss_ptr: *mut TaskStateSegment,
/// Pointer to this core's GDT (pre-allocated on BSP, loaded on AP).
gdt_ptr: *const GlobalDescriptorTable,
/// Segment selectors for this core's GDT.
gdt_code: SegmentSelector,
gdt_data: SegmentSelector,
gdt_tss: SegmentSelector, Code spotlight
Trace emit helper
Trace emit helper. Why it matters: A single trace_event(event) call site keeps instrumentation minimal (one line per call site) and handles timestamping and core-ID tagging automatically. Focus on `trace_event` and `trace_event(event)` in `kernel/src/trace.rs` while you trace the snippet.
/// Emit a trace event on the current core's trace ring.
///
/// Compiles to nothing when the `trace` feature is off.
#[cfg(feature = "trace")]
pub fn trace_event(event: TraceEvent) {
if !crate::smp::is_per_core_ready() {
return;
}
let core_id = crate::smp::per_core().core_id;
let tick = crate::arch::x86_64::interrupts::tick_count();
// Safety: trace_ring is wrapped in UnsafeCell for interior mutability.
// Only the owning core writes to its ring (single-writer guarantee via
// gs_base per-core data). The UnsafeCell makes the mutable access sound.
let ring_ptr = crate::smp::per_core().trace_ring.get();
unsafe {
(*ring_ptr).push(TraceEntry {
tick,
core: core_id,
_pad: [0; 7],
event,
});
}
} Key files
- kernel-core/src/trace_ring.rs — TraceRing<N>, TraceEvent, TraceEntry -- host-testable
- kernel/src/trace.rs — trace_event(), dump_trace_rings()
- kernel/src/smp/mod.rs — PerCoreData.trace_ring field
- kernel/src/task/scheduler.rs — Scheduler instrumentation (6 sites)
- kernel/src/process/mod.rs — Fork instrumentation (2 sites)
- kernel/src/ipc/endpoint.rs — IPC instrumentation (7 sites)
- kernel/src/arch/x86_64/syscall.rs — sys_ktrace syscall
- userspace/syscall-lib/src/lib.rs — ktrace() userspace wrapper
How real OSes differ
- Linux uses `ftrace` with per-CPU ring buffers backed by `ring_buffer.c`, supporting binary format, filtering, function graph tracing, and live streaming via `/sys/kernel/tracing/`
- FreeBSD uses `DTrace` (and now also `ktrace`) with kernel-userspace cooperation and sophisticated filtering
- Production systems use binary trace formats (CTF, perf.data) for efficient storage and offline analysis tools
- Lock-free ring buffers in production kernels use memory ordering barriers and sequence counters for safe concurrent reader/writer access
Success criteria
- `cargo test -p kernel-core` passes with all trace ring tests
- `cargo xtask check` passes with trace feature on and off
- `cargo xtask test` passes with trace feature on
- `cargo build -p kernel --no-default-features` compiles without trace code
- Trace ring dump appears in serial output after crash diagnostics on panic
- `sys_ktrace` syscall copies trace entries to userspace buffer
Related phases
Keep exploring the roadmap
Phase 43
CompleteCrash Diagnostics
Every kernel crash is self-diagnosing. When the kernel panics or takes a fatal fault, the serial output includes full CPU register state, current task metadata, and per-core scheduler context. `debug_assert!` invariants at every scheduler, fork, IPC, and stack boundary catch state corruption at the point of origin rather than surfacing later as an opaque `RIP=0x4` fault.
Phase 43
CompleteRegression Stress Ci
A layered test infrastructure that catches SMP race conditions through focused QEMU regression tests, looped stress tests with automatic artifact capture, seed-based iteration for reproducibility (guest-side timing variation deferred), host-side property and concurrency testing for extracted scheduler/fork/IPC models, and CI tier mapping that runs regressions on every PR and stress tests nightly.