Per-CPU Management

Relevant source files

• .github/workflows/ci.yml
• def.rs
• src/percpu.rs

Purpose and Scope

This document describes the per-CPU state management system in the RISC-V VCPU hypervisor, focusing on how each CPU core is initialized and configured for virtualization. The per-CPU management system handles Control and Status Register (CSR) initialization, hardware feature enablement, and interrupt/exception delegation setup required for RISC-V hypervisor extensions.

For information about overall VCPU lifecycle management, see VCPU Lifecycle and Management. For details about hardware feature detection, see Hardware Detection. For CSR register definitions and bitfield details, see CSR Definitions and Hardware Registers.

Per-CPU Architecture Overview

The per-CPU management system provides a hardware abstraction layer that initializes each CPU core for hypervisor operation. It bridges the gap between the ArceOS framework's generic per-CPU interface and RISC-V-specific hypervisor initialization requirements.

flowchart TD
subgraph subGraph2["Hardware Layer"]
    hedeleg["HEDELEG CSR"]
    hideleg["HIDELEG CSR"]
    hvip["HVIP CSR"]
    hcounteren["HCOUNTEREN CSR"]
    sie["SIE CSR"]
end
subgraph subGraph1["RISC-V Per-CPU Implementation"]
    RISCVPerCpu["RISCVPerCpu<H>"]
    setup_csrs["setup_csrs()"]
    hardware_enable["hardware_enable()"]
    has_hardware_support["has_hardware_support()"]
end
subgraph subGraph0["ArceOS Framework"]
    AxArchPerCpu["AxArchPerCpu Trait"]
    AxVCpuHal["AxVCpuHal Interface"]
end

AxArchPerCpu --> RISCVPerCpu
AxVCpuHal --> RISCVPerCpu
RISCVPerCpu --> hardware_enable
RISCVPerCpu --> setup_csrs
hardware_enable --> has_hardware_support
setup_csrs --> hcounteren
setup_csrs --> hedeleg
setup_csrs --> hideleg
setup_csrs --> hvip
setup_csrs --> sie

Sources: src/percpu.rs(L1 - L82) 

RISCVPerCpu Structure and Lifecycle

The RISCVPerCpu struct serves as the primary per-CPU state container, implementing the AxArchPerCpu trait to integrate with the ArceOS hypervisor framework.

Structure Definition

The structure uses a PhantomData marker to maintain the generic AxVCpuHal type parameter while storing no actual data, as the per-CPU state is maintained entirely in hardware CSRs.

Sources: src/percpu.rs(L10 - L13)  src/percpu.rs(L15 - L41) 

Initialization Flow

The per-CPU initialization follows a specific sequence that prepares each CPU core for hypervisor operation:

flowchart TD
subgraph subGraph0["setup_csrs() Details"]
    exception_delegate["Configure Exception Delegation (HEDELEG)"]
    interrupt_delegate["Configure Interrupt Delegation (HIDELEG)"]
    clear_interrupts["Clear Virtual Interrupts (HVIP)"]
    inst_misalign["INST_ADDR_MISALIGN"]
    breakpoint["BREAKPOINT"]
    ecall["ENV_CALL_FROM_U_OR_VU"]
    page_faults["Page Fault Exceptions"]
    vs_timer["VIRTUAL_SUPERVISOR_TIMER"]
    vs_external["VIRTUAL_SUPERVISOR_EXTERNAL"]
    vs_soft["VIRTUAL_SUPERVISOR_SOFT"]
    clear_vssip["hvip::clear_vssip()"]
    clear_vstip["hvip::clear_vstip()"]
    clear_vseip["hvip::clear_vseip()"]
end
Start["RISCVPerCpu::new(cpu_id)"]
setup_csrs_call["Call setup_csrs()"]
setup_counters["Setup Counter Access (HCOUNTEREN)"]
enable_interrupts["Enable Supervisor Interrupts (SIE)"]
create_struct["Create RISCVPerCpu Instance"]
End["Return AxResult"]

Start --> setup_csrs_call
clear_interrupts --> clear_vseip
clear_interrupts --> clear_vssip
clear_interrupts --> clear_vstip
clear_interrupts --> setup_counters
create_struct --> End
enable_interrupts --> create_struct
exception_delegate --> breakpoint
exception_delegate --> ecall
exception_delegate --> inst_misalign
exception_delegate --> interrupt_delegate
exception_delegate --> page_faults
interrupt_delegate --> clear_interrupts
interrupt_delegate --> vs_external
interrupt_delegate --> vs_soft
interrupt_delegate --> vs_timer
setup_counters --> enable_interrupts
setup_csrs_call --> exception_delegate

Sources: src/percpu.rs(L16 - L24)  src/percpu.rs(L44 - L81) 

CSR Initialization Process

The setup_csrs() function performs critical hypervisor-level Control and Status Register initialization. This unsafe function configures the hardware delegation and interrupt management necessary for virtualization.

Exception Delegation Configuration

The hypervisor exception delegation register (HEDELEG) is configured to automatically forward specific exceptions from guest execution to the guest's supervisor mode rather than trapping to the hypervisor:

Sources: src/percpu.rs(L47 - L56)  src/consts.rs

Interrupt Delegation Setup

The hypervisor interrupt delegation register (HIDELEG) forwards virtual supervisor-level interrupts directly to the guest:

flowchart TD
subgraph subGraph2["Guest Handling"]
    GuestISR["Guest Interrupt Handlers"]
end
subgraph subGraph1["HIDELEG Configuration"]
    VSTIMER_BIT["VIRTUAL_SUPERVISOR_TIMER"]
    VSEXT_BIT["VIRTUAL_SUPERVISOR_EXTERNAL"]
    VSSOFT_BIT["VIRTUAL_SUPERVISOR_SOFT"]
end
subgraph subGraph0["Interrupt Sources"]
    VSTimer["Virtual Supervisor Timer"]
    VSExternal["Virtual Supervisor External"]
    VSSoft["Virtual Supervisor Software"]
end

VSEXT_BIT --> GuestISR
VSExternal --> VSEXT_BIT
VSSOFT_BIT --> GuestISR
VSSoft --> VSSOFT_BIT
VSTIMER_BIT --> GuestISR
VSTimer --> VSTIMER_BIT

Sources: src/percpu.rs(L59 - L64)  src/consts.rs

Counter and Interrupt Enablement

The initialization process also configures:

1. Counter Access (HCOUNTEREN): Enables guest access to performance counters by setting all bits to 1
2. Virtual Interrupt Clearing (HVIP): Clears all pending virtual supervisor interrupts
3. Supervisor Interrupt Enable (SIE): Enables external, software, and timer interrupts at supervisor level

Sources: src/percpu.rs(L67 - L79) 

Hardware Enablement Interface

The per-CPU system provides hardware enablement control through the AxArchPerCpu trait implementation:

stateDiagram-v2
[*] --> Uninitialized
Uninitialized --> Initialized : new(cpu_id)
Initialized --> HardwareEnabled : hardware_enable()
HardwareEnabled --> HardwareDisabled : hardware_disable()
HardwareDisabled --> HardwareEnabled : hardware_enable()
note left of HardwareEnabled : ['has_hardware_support() == true']
note left of HardwareDisabled : ['Hardware virtualization disabled']
note left of Uninitialized : ['Per-CPU state not created']

Hardware Support Validation

The hardware_enable() method validates that the underlying hardware supports RISC-V hypervisor extensions before enabling virtualization features:

• Success Path: Returns Ok(()) when has_hardware_support() returns true
• Failure Path: Returns AxError::Unsupported when hardware lacks H-extension support

Sources: src/percpu.rs(L30 - L36) 

Unimplemented Features

The current implementation includes placeholder methods for future functionality:

• is_enabled(): Returns whether hardware virtualization is currently active (unimplemented)
• hardware_disable(): Disables hardware virtualization features (unimplemented)

Sources: src/percpu.rs(L26 - L28)  src/percpu.rs(L38 - L40) 

Integration with System Architecture

The per-CPU management system integrates with the broader RISC-V hypervisor architecture through several key interfaces:

flowchart TD
subgraph subGraph3["VCPU Management"]
    VCPUCreation["VCPU Creation"]
    VCPUExecution["VCPU Execution"]
end
subgraph subGraph2["Hardware Detection"]
    has_hardware_support_func["has_hardware_support()"]
    detect_h_extension["detect_h_extension()"]
end
subgraph subGraph1["Per-CPU Layer"]
    RISCVPerCpu_new["RISCVPerCpu::new()"]
    setup_csrs_func["setup_csrs()"]
    hardware_enable_func["hardware_enable()"]
end
subgraph subGraph0["System Initialization"]
    SystemBoot["System Boot"]
    CPUEnumeration["CPU Enumeration"]
    PerCPUCreation["Per-CPU Creation"]
end

CPUEnumeration --> PerCPUCreation
PerCPUCreation --> RISCVPerCpu_new
RISCVPerCpu_new --> hardware_enable_func
RISCVPerCpu_new --> setup_csrs_func
SystemBoot --> CPUEnumeration
VCPUCreation --> VCPUExecution
hardware_enable_func --> has_hardware_support_func
has_hardware_support_func --> detect_h_extension
setup_csrs_func --> VCPUCreation

This per-CPU initialization must complete successfully before any VCPU instances can be created or executed on the respective CPU core, ensuring that the hardware virtualization environment is properly configured.

Sources: src/percpu.rs(L1 - L82)  src/detect.rs