percore

Safe per-CPU core mutable state on no_std platforms through exception masking.

This crate provides two main wrapper types: PerCore to provide an instance of a value per CPU core, where each core can access only its instance, and ExceptionLock to guard a value so that it can only be accessed while exceptions are masked. These may be combined with RefCell to provide safe per-core mutable state.

ExceptionLock may also be combined with a spinlock-based mutex (such as one provided by the spin crate) to avoid deadlocks when accessing global mutable state from exception handlers.

Example

use core::cell::RefCell;
use percore::{exception_free, Cores, ExceptionLock, PerCore};

/// The total number of CPU cores in the target system.
const CORE_COUNT: usize = 2;

struct CoresImpl;

unsafe impl Cores for CoresImpl {
    fn core_index() -> usize {
        todo!("Return the index of the current CPU core, 0 or 1")
    }
}

struct CoreState {
    // Your per-core mutable state goes here...
    foo: u32,
}

const EMPTY_CORE_STATE: ExceptionLock<RefCell<CoreState>> =
    ExceptionLock::new(RefCell::new(CoreState { foo: 0 }));
static CORE_STATE: PerCore<[ExceptionLock<RefCell<CoreState>>; CORE_COUNT], CoresImpl> =
    PerCore::new([EMPTY_CORE_STATE; CORE_COUNT]);

fn main() {
    // Mask exceptions while accessing mutable state.
    exception_free(|token| {
        // `token` proves that interrupts are masked, so we can safely access per-core mutable
        // state.
        CORE_STATE.get().borrow_mut(token).foo = 42;
    });
}

Some more complete examples are provided in the examples/aarch64_qemu directory.

This is not an officially supported Google product.

Supported architectures

Currently only aarch32 and aarch64 are fully supported. The crate will build for other architectures, but you'll need to provide your own implementation of the exception_free function. Patches are welcome to add support for other architectures.

Derive

The derive feature enables the use of the #[percore::percore] attribute, which replaces a static with a LinkedPerCore of the same name in the .percore linker section. The wrapper stores the primary core's value and accesses the corresponding copy for the current core. This is an alternative to declaring per-core variables with PerCore.

By default, the primary core is assumed to have index 0, with the secondary cores following it sequentially. Each core has its own per-core area, and these areas are laid out contiguously in memory, like an array of sections. Only the primary core's area, represented by the .percore section is included in the image, and it contains the initial values for per-core variables. The corresponding areas for the secondary cores, represented by the .percore_secondary section, are not included in the image. The provided functions are suitable for tiny and small memory models.

Projects may use a different memory layout, provided that they copy the initial variable values into each secondary core's per-core area and implement percore::derive::PercoreLocalOffset on a type marked with #[percore::percore_local_offset] to retrieve the appropriate offset.

Pros:

• At the declaration of the variable it is not necessary to have a Cores implementation.
• Each core's per-core variables are packed together which results in more efficient cache utilization.
• Better performance.

Cons:

• Can only be used for global variables.
• More complex setup.
• Currently, this is only implemented for AArch64 targets.

It requires the following actions from the consuming project:

• Call percore::derive::percore_copy_secondary_data() on the primary core at boot time.
• Calculate the local core offset at boot time on each core using percore_calculate_local_offset(core_index).
• Store this offset in a project specific way.
• Implement percore::derive::PercoreLocalOffset for a type that retrieves the previously stored offset and mark the type with #[percore::percore_local_offset].
• Allocate .percore and .percore_secondary sections in the linker script and mark their boundaries using the __PERCORE_START__, __PERCORE_END__, __PERCORE_SECONDARY_START__ and __PERCORE_SECONDARY_END__ symbols. It is recommended to align these sections to the cache line size but at least to 16 bytes on AArch64.

Example

Initialization

The primary core initializes the secondary per-core areas. Each secondary core then calculates its offset from its linear index and stores it in TPIDR_EL1 before entering main.

use core::arch::global_asm;

global_asm!(
    "init_primary:
        bl {percore_copy_secondary_data}

    init_secondary:
        bl calculate_core_index
        /* X0 now contains the local core's linear index */

        /* Calculate and store local offset */
        bl {percore_calculate_local_offset}
        msr tpidr_el1, x0

        b main
    ",
    percore_copy_secondary_data = sym percore_copy_secondary_data,
    percore_calculate_local_offset = sym percore_calculate_local_offset,
);

PercoreLocalOffsetImpl implementation

This implementation returns the offset from TPIDR_EL1, the EL1 Software Thread ID Register.

use percore::{derive::PercoreLocalOffset, percore_local_offset};

percore_local_offset!(PercoreLocalOffsetImpl);
struct PercoreLocalOffsetImpl;

// Safety: Each core initializes TPIDR_EL1 with the offset of its valid percore area before
// accessing any percore variable.
unsafe impl PercoreLocalOffset for PercoreLocalOffsetImpl {
    fn percore_local_offset() -> isize {
        read_tpidr_el1().threadid() as _
    }
}

Linker script

The linker script places the primary core's initialized data in .percore and reserves an equally sized area for every secondary core in .percore_secondary.

.percore : ALIGN(CACHE_LINE_SIZE) {
    __PERCORE_START__ = .;
    *(SORT_BY_ALIGNMENT(.percore .percore.*))
    . = ALIGN(CACHE_LINE_SIZE);
    __PERCORE_END__ = .;
} >image

ASSERT(
    ALIGNOF(.percore) <= CACHE_LINE_SIZE,
    ".percore contains an object aligned to a larger boundary than the section's alignment."
)

.percore_secondary (NOLOAD) : ALIGN(CACHE_LINE_SIZE) {
    __PERCORE_SECONDARY_START__ = .;
    . += (__PERCORE_END__ - __PERCORE_START__) * (CORE_COUNT - 1);
    __PERCORE_SECONDARY_END__ = .;
} >image

Usage

All cores will have their own instance of VARIABLE which is initialized to 1.

use core::cell::RefCell;
use percore::{ExceptionLock, exception_free, percore};

#[percore::percore]
static VARIABLE: ExceptionLock<RefCell<u64>> = ExceptionLock::new(RefCell::new(1));

exception_free(|token| {
    assert_eq!(1, *VARIABLE.get().borrow_mut(token));

    *VARIABLE.get().borrow_mut(token) = 2;
    assert_eq!(2, *VARIABLE.get().borrow_mut(token));
});

Accessing from assembly

Assembly code can access the current core's instance by adding the offset in TPIDR_EL1 to the variable's base address. This example increments the local instance of VARIABLE.

global_asm!(
    "increment:
        /* Retrieve base address of the per-core variable */
        adrp	x0, {VARIABLE}
        add	x0, x0, :lo12:{VARIABLE}

        /* Add per-core offset stored in TPIDR_EL1 */
        mrs	x1, tpidr_el1
        add	x0, x0, x1
        /* At this point X0 contains the address to the local core's instance of VARIABLE */

        /* Increment the value by one. */
        ldr	x1, [x0]
        add	x1, x1, #1
        str	x1, [x0]
        ret",
    VARIABLE = sym VARIABLE,
);

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contributing

If you want to contribute to the project, see details of how we accept contributions.