m³OS docs GitHub

Phase 52

Kernel Structural Hardening

The kernel's memory management, context management, and TLB coherence subsystems move onto the structurally hardened path defined by this phase. In the checked-in tree, the AddressSpace object, typed user buffers, batched TLB work, and frame zeroing landed, and Phase 52d completed the remaining closure work so the task-owned return-state and generation-tracking story now matches the shipped implementation end to end.

Complete

Learning goal

Understand why address-space identity should be a first-class kernel object (Redox: `AddrSpaceWrapper`, Zircon: `VmAspace`)

Category

infrastructure

Why this phase matters

The kernel’s memory management, context management, and TLB coherence subsystems move onto the structurally hardened path defined by this phase. In the checked-in tree, the AddressSpace object, typed user buffers, batched TLB work, and frame zeroing landed, and Phase 52d completed the remaining closure work so the task-owned return-state and generation-tracking story now matches the shipped implementation end to end.

How it works in m3OS

Phase 52b replaces the fragile kernel patterns that made the Phase 52 bugs possible. Where Phase 52a patched the immediate symptoms, this phase introduces structural defenses: a first-class AddressSpace object with per-core tracking, typed UserSliceRo/UserSliceWo wrappers at the syscall boundary, task-owned UserReturnState that eliminates ~40 manual restore call sites, batch TLB shootdown with address-space targeting, and frame zeroing on free. These patterns are drawn from Redox, seL4, and Zircon architecture analysis.

How it builds on earlier phases

This phase builds on Kernel Reliability Fixes and turns that foundation into Kernel Structural Hardening.

Implementation outline

  • Define AddressSpace struct, wrap existing page_table_root + VMAs
  • Update Process to use Arc<AddressSpace> instead of Option<PhysAddr>
  • Update scheduler dispatch to call activate_on_core/deactivate_on_core
  • Add current_addrspace pointer to PerCoreData
  • Implement batch tlb_shootdown_range using active_on_cores
  • Define UserReturnState, add to Task
  • Save user state to UserReturnState at syscall entry
  • Restore from UserReturnState in scheduler dispatch
  • Remove all restore_caller_context calls
  • Define UserSliceRo/UserSliceWo/UserSliceRw, convert syscall handlers
  • Add frame zeroing to free_frame
  • Add generation counter assertions around copy_to_user

Task progress snapshot

0 of 52 tracked tasks are complete.

Documentation targets

  • Documentation deliverables have not been listed yet.

Deferred follow-on work

  • VMA tree (BTreeMap or interval tree) — deferred to Phase 52c
  • Per-core scheduler with work-stealing — deferred to Phase 52c
  • Dynamic IPC resource pools — deferred to Phase 52c
  • ISR-direct notification wakeup — deferred to Phase 52c

Source materials

Important code

Code spotlight cards

Code spotlight

Define AddressSpace struct

kernel/src/mm/mod.rs L26-L35

Define AddressSpace struct. Why it matters: The kernel currently has no address-space abstraction. A process's address space is a raw PhysAddr with no metadata, no refcount, and no per-CPU tracking. This makes targeted TLB shootdown, stale-mapping detection, and thread address-space sharing impossible. Focus on `AddressSpace` and `PhysAddr` in `kernel/src/mm/mod.rs` while you trace the snippet.

/// A process's virtual address space descriptor.
///
/// Wraps the PML4 physical address with metadata for TLB shootdown
/// optimization (generation counter) and multi-core tracking.
pub struct AddressSpace {
    pml4_phys: PhysAddr,
    generation: AtomicU64,
    active_on_cores: AtomicU64,
    page_table_lock: Mutex<()>,
}

Code spotlight

Update Process to use Arc<AddressSpace>

kernel/src/process/mod.rs L88-L99

Update Process to use Arc<AddressSpace>. Why it matters: Replacing the raw Option<PhysAddr> with Arc<AddressSpace> enables shared address spaces for CLONE_VM threads and provides the foundation for all other Track A/B tasks. Focus on `Process::page_table_root` and `Process::addr_space` in `kernel/src/process/mod.rs` while you trace the snippet.

        let table = PROCESS_TABLE.lock();
        table.find(pid).and_then(|p| {
            p.addr_space
                .as_ref()
                .map(|addr_space| addr_space.as_ref() as *const AddressSpace)
        })
    }?;

    // Before per-core state exists the kernel is still effectively single-core,
    // so the current process entry cannot be concurrently reaped before callers
    // immediately dereference this pointer in a tightly-scoped unsafe block.
    NonNull::new(ptr as *mut AddressSpace)

Code spotlight

Add per-core `current_addrspace` tracking

kernel/src/smp/mod.rs L144-L167

Add per-core `current_addrspace` tracking. Why it matters: Each CPU needs to know which address space is currently loaded so that TLB shootdowns can be targeted. This is the Redox PercpuBlock.current_addrsp equivalent. Focus on `current_addrspace` and `PerCoreData::current_addrspace` in `kernel/src/smp/mod.rs` while you trace the snippet.

/// Per-core state block.
///
/// Each core has one of these, initialized during BSP init or AP bootstrap.
/// The `gs_base` MSR points to this struct so that `per_core()` can retrieve
/// it in O(1) without MMIO.
#[repr(C)]
pub struct PerCoreData {
    /// Self-pointer at offset 0 — reserved for future `gs:[0]` access.
    /// Currently unused: `per_core()` reads `IA32_GS_BASE` via `rdmsr`.
    self_ptr: *const PerCoreData,
    /// Logical core index (0 = BSP, 1..n = APs in MADT order).
    pub core_id: u8,
    /// LAPIC ID from the MADT.
    pub apic_id: u8,
    /// Set to `true` once this core has completed initialization.
    pub is_online: AtomicBool,
    /// Pointer to this core's TSS (for runtime RSP0 updates).
    tss_ptr: *mut TaskStateSegment,
    /// Pointer to this core's GDT (pre-allocated on BSP, loaded on AP).
    gdt_ptr: *const GlobalDescriptorTable,
    /// Segment selectors for this core's GDT.
    gdt_code: SegmentSelector,
    gdt_data: SegmentSelector,
    gdt_tss: SegmentSelector,

Key files

  • kernel/src/mm/mod.rs — AddressSpace struct with generation and active-core tracking
  • kernel/src/mm/user_mem.rs — UserSliceRo, UserSliceWo typed wrappers
  • kernel/src/task/mod.rs — UserReturnState in the Task struct
  • kernel/src/smp/tlb.rs — Batch TLB shootdown with address-space targeting
  • kernel/src/mm/frame_allocator.rs — Zero-on-free for user-accessible frames
  • kernel/src/process/mod.rs — Process uses Arc<AddressSpace>

How real OSes differ

  • Redox uses `Arc<AddrSpaceWrapper>` with `used_by: LogicalCpuSet` and `tlb_ack: AtomicU32` for the same purpose. Source: `src/context/memory.rs`.
  • Zircon's `VmAspace` is independently ref-counted with `active_cpus_` bitmap and PCID-deferred TLB flush. Source: `zircon/kernel/vm/include/vm/vm_aspace.h`.
  • seL4 embeds a TLB bitmap in unused PML4 entries (`tlb_bitmap.h`) — a clever space-saving trick m3OS could consider.
  • Linux uses `mm_struct` with `cpu_bitmap` for lazy TLB mode and targeted shootdown.

Success criteria

  • No `restore_caller_context` calls remain in the codebase
  • `UserSliceRo`/`UserSliceWo` are used at all syscall boundary copy sites
  • `AddressSpace.active_on_cores` correctly tracks which CPUs have each AS loaded
  • TLB shootdown for `munmap(ptr, 1 MiB)` sends 1 IPI (not 256)
  • Freed frames read as all-zeros when accessed through a stale mapping
  • `copy_to_user` reliability reproducer (real DeviceTTY/keyboard-IRQ path on SMP4) no longer fails
  • `cargo xtask check` passes
  • `cargo xtask test` passes

Related phases

Keep exploring the roadmap