m³OS docs GitHub

Phase 36

Expanded Memory

The kernel supports demand paging, large `mmap()` regions, and `mprotect()`. Virtual pages are mapped lazily — physical frames are allocated on first access via the page fault handler. This unlocks running large cross-compiled binaries (Clang, Python, Node.js, git) that allocate hundreds of megabytes via `mmap()` but only touch a fraction of the mapped pages. QEMU RAM is increased to support these workloads.

Complete

Learning goal

Understand demand paging: why real OSes don't back every virtual page with a physical frame at map time.

Category

foundations

Why this phase matters

The kernel supports demand paging, large mmap() regions, and mprotect(). Virtual pages are mapped lazily — physical frames are allocated on first access via the page fault handler. This unlocks running large cross-compiled binaries (Clang, Python, Node.js, git) that allocate hundreds of megabytes via mmap() but only touch a fraction of the mapped pages. QEMU RAM is increased to support these workloads.

How it works in m3OS

The roadmap entry defines the learning scope for this phase; a deeper implementation doc has not been matched yet.

How it builds on earlier phases

This phase builds on True SMP Multitasking and turns that foundation into Expanded Memory.

Implementation outline

  • Extend MemoryMapping with prot and flags fields.
  • Change sys_linux_mmap() to record the VMA without allocating frames.
  • Extend the page fault handler to check the VMA list and demand-map valid lazy faults.
  • Implement mprotect() syscall with page table walks and TLB shootdown.
  • Increase QEMU RAM to 1 GB.
  • Expand data partition to 1 GB.
  • Validate CoW fork still works alongside demand paging.
  • Test with large mmap regions (256 MB+) and a cross-compiled binary.

Task progress snapshot

63 of 63 tracked tasks are complete.

Documentation targets

  • Documentation deliverables have not been listed yet.

Deferred follow-on work

  • Swap to disk — page out cold memory to virtio-blk
  • setrlimit() per-process limits
  • OOM killer — kernel still panics on true physical memory exhaustion
  • Huge pages (2 MiB / 1 GiB) — performance optimization
  • Memory-mapped files (mmap with file descriptors)
  • NUMA-aware allocation

Source materials

Important code

Code spotlight cards

Code spotlight

Extend `MemoryMapping` with prot and flags fields

kernel/src/process/mod.rs L481-L492

permissions to set when demand-mapping a frame, and mprotect needs flags to split. Why it matters: The page fault handler needs protection bits to know what. Focus on `MemoryMapping` in `kernel/src/process/mod.rs` while you trace the snippet.

/// Describes a contiguous anonymous memory mapping created by `mmap`.
#[derive(Clone, Debug)]
pub struct MemoryMapping {
    /// Starting virtual address (page-aligned).
    pub start: u64,
    /// Length in bytes (page-aligned, as recorded by `sys_linux_mmap`).
    pub len: u64,
    /// Protection bits (`PROT_READ | PROT_WRITE | PROT_EXEC`).
    pub prot: u64,
    /// Mapping flags (`MAP_PRIVATE | MAP_ANONYMOUS`).
    pub flags: u64,
}

Code spotlight

Update mmap to record prot and flags in VMA

kernel/src/arch/x86_64/syscall.rs L4958-L4981

protection and flags so later fault handling and mprotect can use them. Why it matters: The VMA entry created by mmap must carry the caller's requested. Focus on `sys_linux_mmap` and `prot` in `kernel/src/arch/x86_64/syscall.rs` while you trace the snippet.

fn sys_linux_mmap(addr_hint: u64, len: u64, prot: u64) -> u64 {
    // Read flags from SYSCALL_ARG3 (r10 at syscall entry).
    // SAFETY: single-CPU, read after every SYSCALL entry stores to SYSCALL_ARG3.
    let flags = per_core_syscall_arg3();

    const MAP_PRIVATE: u64 = 0x02;
    const MAP_ANONYMOUS: u64 = 0x20;
    if flags & MAP_ANONYMOUS == 0 {
        log::warn!(
            "[mmap] non-anonymous mmap not supported (flags={:#x})",
            flags
        );
        return NEG_EINVAL;
    }
    // Mask prot and flags to supported bits only.
    const PROT_MASK: u64 = 0x7; // PROT_READ | PROT_WRITE | PROT_EXEC
    let prot = prot & PROT_MASK;
    let flags = flags & (MAP_PRIVATE | MAP_ANONYMOUS);

    let len = if len == 0 {
        return NEG_EINVAL;
    } else {
        len
    };

Code spotlight

Extend page fault handler with VMA-based demand mapping

kernel/src/arch/x86_64/interrupts.rs L349-L372

must check the VMA list to decide whether to allocate a frame or deliver SIGSEGV. Why it matters: This is the demand paging implementation — the page fault handler. Focus on `page_fault_handler` and `prot` in `kernel/src/arch/x86_64/interrupts.rs` while you trace the snippet.

extern "x86-interrupt" fn page_fault_handler(
    mut stack_frame: InterruptStackFrame,
    err: PageFaultErrorCode,
) {
    let addr = x86_64::registers::control::Cr2::read();

    // Check if the fault came from ring 3 (user mode).
    if stack_frame.code_segment.rpl() == x86_64::PrivilegeLevel::Ring3 {
        // P17-T031: detect CoW faults — a write to a present, non-writable
        // page marked with BIT_9 (the CoW marker set by cow_clone_user_pages).
        let is_write = err.contains(PageFaultErrorCode::CAUSED_BY_WRITE);
        let is_present = err.contains(PageFaultErrorCode::PROTECTION_VIOLATION);
        if is_write
            && is_present
            && let Ok(fault_vaddr) = addr
        {
            let fault_addr_u64 = fault_vaddr.as_u64();
            if has_cow_marker(fault_addr_u64) {
                // CoW fault — resolve directly in the ISR. Safe because
                // the fault is from ring 3 (no kernel locks held) and
                // we're on a single CPU. On OOM, fall through to kill.
                if resolve_cow_fault(fault_addr_u64) {
                    return;
                }

Key files

  • kernel/src/process/mod.rs — Currently in kernel/src/process/mod.rs. Each process has a Vec<MemoryMapping>
  • kernel/src/arch/x86_64/interrupts.rs — In kernel/src/arch/x86_64/interrupts.rs. Currently handles CoW faults (Phase 17)
  • kernel/src/arch/x86_64/syscall.rs — In kernel/src/arch/x86_64/syscall.rs. Currently eagerly allocates frames in a loop.

How real OSes differ

  • Real kernels (Linux, FreeBSD) use red-black trees for VMA lookup (O(log n) instead of our linear scan).
  • Reverse mapping (rmap) to find all PTEs mapping a given physical page.
  • Memory-mapped files (not just anonymous mappings).
  • NUMA-aware page allocation.
  • Transparent huge pages (THP) that automatically promote 4K pages to 2M pages.
  • KSM (Kernel Same-page Merging) for deduplication.
  • Swap with multiple swap devices and priority ordering.

Success criteria

  • `mmap(MAP_ANONYMOUS)` returns immediately without allocating physical frames.
  • First access to a lazy-mapped page triggers a page fault that allocates a frame.
  • Programs can `mmap()` 256 MB regions without exhausting physical memory.
  • `mprotect()` changes page permissions and flushes TLB.
  • Existing CoW fork (Phase 17) still works correctly alongside demand paging.
  • QEMU boots with 1 GB RAM.
  • Data partition is 1 GB.
  • A large cross-compiled binary (e.g., Python ~8 MB) loads and runs.

Related phases

Keep exploring the roadmap