Per-CPU Virtualization State

Relevant source files

• src/percpu.rs

This document covers the per-CPU virtualization state management system in axvcpu, which provides a safe abstraction for initializing, enabling, and managing hardware virtualization features on each CPU core. This system ensures that virtualization capabilities are properly configured across all CPUs in the hypervisor and provides architecture-independent lifecycle management with architecture-specific implementations.

For information about the overall VCPU management and state machine, see VCPU State Machine and Lifecycle. For details about the hardware abstraction layer, see Hardware Abstraction Layer.

Architecture Abstraction

The per-CPU virtualization state system is built around the AxArchPerCpu trait, which defines the interface that architecture-specific implementations must provide. This trait abstracts the hardware-specific operations needed to manage virtualization on a per-CPU basis.

AxArchPerCpu Trait Interface

The AxArchPerCpu trait defines four core operations for managing per-CPU virtualization state:

Each architecture (x86_64, ARM64, RISC-V) provides its own implementation of this trait to handle platform-specific virtualization setup and control.

AxArchPerCpu Trait Architecture

flowchart TD
subgraph subGraph3["RISC-V Implementation"]
    RISCV_IMPL["RiscVPerCpu"]
    RISCV_H["H Extension"]
    RISCV_CSR["CSR Configuration"]
end
subgraph subGraph2["ARM64 Implementation"]
    ARM_IMPL["ArmPerCpu"]
    ARM_EL2["EL2 Setup"]
    ARM_HYP["Hypervisor Mode"]
end
subgraph subGraph1["x86_64 Implementation"]
    X86_IMPL["X86PerCpu"]
    X86_VMX["VMX Setup"]
    X86_MSR["MSR Configuration"]
end
subgraph subGraph0["Architecture Abstraction"]
    TRAIT["AxArchPerCpu"]
    TRAIT_NEW["new(cpu_id: usize)"]
    TRAIT_ENABLED["is_enabled()"]
    TRAIT_ENABLE["hardware_enable()"]
    TRAIT_DISABLE["hardware_disable()"]
end

ARM_IMPL --> ARM_EL2
ARM_IMPL --> ARM_HYP
RISCV_IMPL --> RISCV_CSR
RISCV_IMPL --> RISCV_H
TRAIT --> TRAIT_DISABLE
TRAIT --> TRAIT_ENABLE
TRAIT --> TRAIT_ENABLED
TRAIT --> TRAIT_NEW
TRAIT_NEW --> ARM_IMPL
TRAIT_NEW --> RISCV_IMPL
TRAIT_NEW --> X86_IMPL
X86_IMPL --> X86_MSR
X86_IMPL --> X86_VMX

Sources: src/percpu.rs(L5 - L19) 

Per-CPU State Container

The AxPerCpu<A> struct serves as a safe wrapper around architecture-specific per-CPU state, providing initialization checking, lifecycle management, and automatic cleanup.

Structure and Fields

The AxPerCpu struct contains two key fields:

• cpu_id: Option<usize> - Tracks the CPU ID and serves as an initialization flag
• arch: MaybeUninit<A> - Stores the architecture-specific state in an uninitialized container

This design ensures that the architecture-specific state is only accessed after proper initialization and provides compile-time safety through the type system.

Initialization and Lifecycle

The per-CPU state follows a strict initialization pattern:

1. Creation: new_uninit() creates an uninitialized state
2. Initialization: init(cpu_id) initializes the architecture-specific state
3. Usage: Methods check initialization before accessing architecture state
4. Cleanup: Drop implementation automatically disables virtualization

AxPerCpu Lifecycle State Machine

Sources: src/percpu.rs(L40 - L95) 

Safety and Error Handling

The AxPerCpu implementation provides several safety mechanisms:

Initialization Checking

All methods that access the architecture-specific state use arch_checked() and arch_checked_mut(), which verify that initialization has occurred before accessing the underlying state:

• Panics if cpu_id is None (not initialized)
• Uses unsafe code only after verification that initialization occurred
• Provides both immutable and mutable access patterns

Automatic Cleanup

The Drop implementation ensures that hardware virtualization is properly disabled when the per-CPU state is destroyed, preventing resource leaks and ensuring system stability.

Error Propagation

Methods return AxResult to propagate architecture-specific errors up to the hypervisor, allowing for proper error handling and recovery.

Sources: src/percpu.rs(L67 - L79)  src/percpu.rs(L97 - L103) 

Usage Patterns

The documentation provides a recommended usage pattern for integrating per-CPU state into a hypervisor:

Static Per-CPU Declaration

#[percpu::def_percpu]
pub static AXVM_PER_CPU: AxPerCpu<MyArchPerCpuImpl> = AxPerCpu::new_uninit();

Initialization and Enablement

let percpu = unsafe { AXVM_PER_CPU.current_ref_mut_raw() };
percpu.init(0).expect("Failed to initialize percpu state");
percpu.hardware_enable().expect("Failed to enable virtualization");

This pattern leverages the percpu crate to manage per-CPU variables and ensures that each CPU core has its own isolated virtualization state.

Per-CPU Integration Pattern

flowchart TD
subgraph subGraph2["Lifecycle Operations"]
    INIT_CALL["init(cpu_id)"]
    ENABLE_CALL["hardware_enable()"]
    RUNTIME["Runtime Operations"]
    DISABLE_CALL["hardware_disable()"]
end
subgraph subGraph1["Per-CPU Variables"]
    STATIC_VAR["AXVM_PER_CPU"]
    PERCPU_LIB["percpu crate"]
    CPU0["CPU 0 Instance"]
    CPU1["CPU 1 Instance"]
    CPUN["CPU N Instance"]
end
subgraph subGraph0["Hypervisor Initialization"]
    BOOT["Boot Process"]
    CPU_ENUM["CPU Enumeration"]
    PERCPU_INIT["Per-CPU Init"]
end

BOOT --> CPU_ENUM
CPU0 --> INIT_CALL
CPU_ENUM --> PERCPU_INIT
ENABLE_CALL --> RUNTIME
INIT_CALL --> ENABLE_CALL
PERCPU_INIT --> STATIC_VAR
PERCPU_LIB --> CPU0
PERCPU_LIB --> CPU1
PERCPU_LIB --> CPUN
RUNTIME --> DISABLE_CALL
STATIC_VAR --> PERCPU_LIB

Sources: src/percpu.rs(L23 - L39) 

Integration with VCPU System

The per-CPU virtualization state system provides the foundation for VCPU operations by ensuring that each physical CPU has the necessary hardware virtualization features enabled. This state is checked and managed independently of individual VCPU instances, allowing multiple VCPUs to share the same physical CPU infrastructure while maintaining isolation.

The is_enabled() method provides a quick check for whether a CPU is ready to run VCPUs, while the enable/disable methods allow for dynamic power management and system reconfiguration.

Sources: src/percpu.rs(L81 - L94)