m³OS docs GitHub

Phase 35

True SMP Multitasking

All CPU cores dispatch and run user tasks. The BSP-only restriction is removed by making syscall infrastructure per-core. The scheduler introduces per-CPU run queues with load balancing and task priorities, substantially improving dispatch locality and multi-core throughput even though the global scheduler lock remains on the dispatch hot path for task-state coordination.

Complete

Learning goal

Understand why per-core state (stacks, saved registers) is essential for SMP syscalls.

Category

foundations

Why this phase matters

All CPU cores dispatch and run user tasks. The BSP-only restriction is removed by making syscall infrastructure per-core. The scheduler introduces per-CPU run queues with load balancing and task priorities, substantially improving dispatch locality and multi-core throughput even though the global scheduler lock remains on the dispatch hot path for task-state coordination.

How it works in m3OS

The roadmap entry defines the learning scope for this phase; a deeper implementation doc has not been matched yet.

How it builds on earlier phases

This phase builds on Real Time Clock and turns that foundation into True SMP Multitasking.

Implementation outline

  • Move syscall statics into PerCoreData and update syscall_entry assembly.
  • Verify: two cores can handle syscalls simultaneously without corruption.
  • Remove BSP-only guard in pick_next().
  • Verify: tasks run on multiple cores (log which core runs which task).
  • Implement per-CPU run queues and move ready-queue contention off the single global queue while keeping the global scheduler lock for task-state coordination.
  • Implement load balancing (periodic migration).
  • Add priority field to Task; update pick_next() to respect priorities.
  • Implement sched_setaffinity / sched_getaffinity syscalls.
  • Implement wait queues; attach to pipes and sockets.
  • Add time accounting; implement times() syscall.
  • Stress test with many concurrent processes across all cores.

Task progress snapshot

76 of 85 tracked tasks are complete.

Documentation targets

  • Documentation deliverables have not been listed yet.

Deferred follow-on work

  • CFS / virtual runtime scheduling
  • NUMA-aware scheduling
  • CPU bandwidth throttling (cgroups)
  • Tickless idle (NO_HZ)
  • Kernel preemption
  • SCHED_DEADLINE (earliest deadline first)
  • CPU hotplug
  • Power-aware scheduling (race-to-idle, frequency scaling)

Source materials

Important code

Code spotlight cards

Code spotlight

Add syscall fields to `PerCoreData`

kernel/src/smp/mod.rs L56-L79

Add syscall fields to `PerCoreData`. Why it matters: Per-core syscall save slots are the prerequisite for letting more than one core enter the syscall path safely at the same time. Focus on `PerCoreData` and `offsets::SYSCALL_STACK_TOP` in `kernel/src/smp/mod.rs` while you trace the snippet.

/// Per-core state block.
///
/// Each core has one of these, initialized during BSP init or AP bootstrap.
/// The `gs_base` MSR points to this struct so that `per_core()` can retrieve
/// it in O(1) without MMIO.
#[repr(C)]
pub struct PerCoreData {
    /// Self-pointer at offset 0 — reserved for future `gs:[0]` access.
    /// Currently unused: `per_core()` reads `IA32_GS_BASE` via `rdmsr`.
    self_ptr: *const PerCoreData,
    /// Logical core index (0 = BSP, 1..n = APs in MADT order).
    pub core_id: u8,
    /// LAPIC ID from the MADT.
    pub apic_id: u8,
    /// Set to `true` once this core has completed initialization.
    pub is_online: AtomicBool,
    /// Pointer to this core's TSS (for runtime RSP0 updates).
    tss_ptr: *mut TaskStateSegment,
    /// Pointer to this core's GDT (pre-allocated on BSP, loaded on AP).
    gdt_ptr: *const GlobalDescriptorTable,
    /// Segment selectors for this core's GDT.
    gdt_code: SegmentSelector,
    gdt_data: SegmentSelector,
    gdt_tss: SegmentSelector,

Code spotlight

Rewrite `syscall_entry` assembly to use `gs_base` offsets

kernel/src/arch/x86_64/syscall.rs L2419-L2419

Rewrite `syscall_entry` assembly to use `gs_base` offsets. Why it matters: This is the core replacement step that swaps the earlier shared syscall statics for per-core register save slots. Focus on `syscall_entry` and `gs_base` in `kernel/src/arch/x86_64/syscall.rs` while you trace the snippet.

        fn syscall_entry();

Code spotlight

Remove BSP-only dispatch guard

kernel/src/task/scheduler.rs L132-L155

Remove BSP-only dispatch guard. Why it matters: This is the Phase 35 scheduler pivot that replaces the earlier Phase 25 BSP-only dispatch rule with all cores participating in task selection. Focus on `pick_next` and `Ready` in `kernel/src/task/scheduler.rs` while you trace the snippet.

    /// Pick the next task to run on the given core.
    ///
    /// First checks the per-core run queue for an O(1) dequeue. Falls back
    /// to a global round-robin scan if the local queue is empty (handles
    /// tasks that haven't been assigned to a queue yet). Finally, falls back
    /// to this core's idle task.
    fn pick_next(&mut self, core_id: u8) -> Option<(u64, usize)> {
        let core_bit = 1u64 << core_id;

        // Scan local run queue: find the highest-priority (lowest numeric) Ready
        // task in a single pass, then remove only that entry. No heap allocation.
        if let Some(data) = crate::smp::get_core_data(core_id) {
            let mut q = data.run_queue.lock();
            let mut best_pos: Option<usize> = None;
            let mut best_prio: u8 = u8::MAX;

            // First pass: discard stale entries from the front while scanning.
            let mut i = 0;
            while i < q.len() {
                let idx = q[i];
                if idx >= self.tasks.len()
                    || self.tasks[idx].state != TaskState::Ready
                    || self.idle_tasks.contains(&Some(idx))
                    || self.tasks[idx].affinity_mask & core_bit == 0

Key files

How real OSes differ

  • Uses a red-black tree ordered by "virtual runtime" for O(log n) scheduling.
  • Per-CPU run queues with work-stealing load balancer.
  • 140 priority levels (0–99 real-time, 100–139 nice-based).
  • SCHED_FIFO, SCHED_RR, SCHED_DEADLINE, SCHED_OTHER policies.
  • NUMA-aware: prefers to schedule tasks near their memory.
  • CPU bandwidth throttling via cgroups.
  • Tickless (NO_HZ) idle and full-system NO_HZ for latency-sensitive workloads.
  • Preemptible kernel (CONFIG_PREEMPT) — can preempt even inside kernel code.

Success criteria

  • Tasks run on all available cores (verified via per-core logging).
  • Two processes making syscalls simultaneously on different cores do not corrupt state.
  • Per-CPU run queues reduce ready-queue contention, but the global scheduler lock remains on the dispatch hot path until later work.
  • Load balancing distributes tasks across cores (no core is idle while another has >1 ready task).
  • Priority 0 (real-time) tasks always preempt priority 20 (normal) tasks.
  • `sched_setaffinity` pins a task to a specific core.
  • Wait queues: a task sleeping on a pipe wakes immediately when data is written.
  • Time accounting: `times()` returns nonzero user and system time.
  • All existing tests pass without regression.

Related phases

Keep exploring the roadmap