RFC: ARM64 (aarch64) Architecture Support for Cloud Foundry#1530
RFC: ARM64 (aarch64) Architecture Support for Cloud Foundry#1530vighes wants to merge 6 commits into
Conversation
|
rkoster
left a comment
rkoster
left a comment
There was a problem hiding this comment.
Given the impact on the ecosystem I think it would only make sense to also include apple silicon support as in scope. Because this would bring a lot of value to the CF Core developers who have MacBooks and now way of easily validating component changes locally.
| 1. **Not breaking x86_64 support** — existing deployments continue unchanged | ||
| 2. **Not requiring CF operators to migrate** — ARM64 is optional, operators opt in | ||
| 3. **Not supporting 32-bit ARM** — only 64-bit ARM (aarch64) | ||
| 4. **Not supporting mixed-architecture Diego cells initially** — cells within a deployment are homogeneous per availability zone |
There was a problem hiding this comment.
why? wouldn't targetting just diego cells to run on ARM be a much smaller scope and more in line with what we have for windows?
There was a problem hiding this comment.
Fair point, the Windows cell model is a good precedent for architecture-aware placement. I initially marked mixed cells as out of scope to reduce first-implementation complexity but you are right that targeting ARM64 Diego cells alongside an x86_64 control plane is a smaller, more achievable scope and mirrors the existing Windows approach. I will revise this to acknowledge mixed-architecture cells as a valid deployment model rather than a non-goal.
|
|
||
| ### Phase 2: Platform Component Validation | ||
|
|
||
| All BOSH releases in `cf-deployment` MUST produce ARM64 binaries alongside x86_64. Go-based components SHOULD cross-compile via CI with `GOARCH=arm64`. Components with C dependencies (e.g., Garden-runC, runC) MUST be validated with native ARM64 compilation. |
There was a problem hiding this comment.
how will components which currently ship binaries as blobs be handled? For example the jdk shipped by the uaa-release.
There was a problem hiding this comment.
For components shipping binary blobs (like the JDK in uaa-release), the approach would be to add ARM64 variants of those blobs alongside the existing x86_64 ones. BOSH packaging scripts would select the correct blob based on the stemcell architecture. For the JDK specifically, ARM64 builds are readily available. I will add a note in Phase 2 clarifying how binary blob components are handled including architecture-conditional blob selection in packaging scripts.
|
|
||
| 1. **Stemcell naming convention:** How should ARM64 stemcells be identified? Proposed: `ubuntu-jammy-arm64` alongside existing `ubuntu-jammy` (implicitly x86_64). Open to community input on naming and metadata approach. | ||
|
|
||
| 2. **CI/CD long-term ownership:** ARM64 CI workers will be provided initially to unblock development. Should long-term ownership transfer to the CF Foundation, or remain as a community-contributed resource? |
There was a problem hiding this comment.
Ideally eventually we end up with CF Foundation ownership and get some AWS credits to help with offsetting the increase in CI spend.
Also related, these workers are these concourse workers running on ARM?
There was a problem hiding this comment.
Yes, these would be Concourse workers running on ARM64. For Go cross-compilation tasks, existing x86_64 workers can produce ARM64 binaries via GOARCH=arm64. But for native compilation (runC with seccomp, Ruby gem native extensions, stemcell building) and integration testing, ARM64 workers are required.
There was a problem hiding this comment.
@taylorsilva to my knowledge Concourse is supporting ARM64 and having concourse workers running on ARM64 won't be an issue. Is this correct?
There was a problem hiding this comment.
at
https://github.com/concourse/concourse/releases/tag/v8.2.4
I see https://github.com/concourse/concourse/releases/download/v8.2.4/concourse-8.2.4-linux-arm64.tgz
under releases section so looks like the answer is yes.
There was a problem hiding this comment.
Thanks for confirming Concourse ARM64 support.
Our POC cross-compiled all Go components from Apple Silicon (M4 Pro Mac), However full local integration (runC with seccomp, container creation) still requires Linux ARM64 either via Docker/Lima on Mac or a remote ARM64 instance. The Linux ARM64 work directly benefits Apple Silicon developers for compilation and unit testing, though not for full local container deployment. I'll add a note acknowledging this developer workflow benefit. |
- Support mixed-architecture cells (ARM64 Diego cells + x86_64 control plane) following the Windows cell model, removed as non-goal - Add binary blob handling strategy for components shipping pre-compiled binaries (e.g., JDK in uaa-release) - Add Apple Silicon developer workflow benefit
|
|
||
| ### Phase 3: ARM64 Stack and Buildpacks | ||
|
|
||
| An ARM64 variant of the `cflinuxfs4` root filesystem and ARM64-compiled buildpack dependencies MUST be produced to enable `cf push` application staging on ARM64 cells. |
There was a problem hiding this comment.
Similar to the stemcell: I would skip cflinuxfs4 (Jammy) and focus on cflinuxfs5 (Noble).
cflinuxfs5 + buildpacks are meanwhile integrated into cf-deployment (opt-in by an experimental ops file): cloudfoundry/cf-deployment#1323
There was a problem hiding this comment.
+1: As Stephan mentioned we should only support ARM for Noble/cflinuxfs5.
There was a problem hiding this comment.
Agreed, updated in latest commit to target cflinuxfs5 (Noble) only.
|
I totally agree with @rkoster on its value for CF developers. It would be much easier to have a local CF where I can generate test CF environments rather than using Docker to create local test environments. It would also help promote Korifi as a development environment, where I am currently trying to get Stratos to run in a Korifi environment. |
|
|
||
| The App Runtime Deployments Working Group owns `cf-deployment`. Work includes: | ||
|
|
||
| - Set up cf-deployment validation against the ARM64 stemcell (similar to existing Jammy/Noble validation) |
There was a problem hiding this comment.
Setting up another validation pipeline for ARM64 is not a problem. But we should clarify the cost increase in advance.
There was a problem hiding this comment.
Agreed, We can provide more details on expected compute hours once we scope the pipeline configurations.
|
|
||
| - Set up cf-deployment validation against the ARM64 stemcell (similar to existing Jammy/Noble validation) | ||
| - Distribute issues discovered during validation to the responsible component Working Groups | ||
| - Provide an ops-file enabling operators to deploy CF on ARM64 |
There was a problem hiding this comment.
For cf-deployment, we would need to switch to an ARM64 stemcell. The ARM64 VM types will be managed in the BOSH cloud-config. Not sure if more changes are required. Do you still have the ops files from the PoC?
There was a problem hiding this comment.
We didn't modify the actual cf-deployment repo during the POC but we did create ops files as part of the prep work. The minimal change for cf-deployment is a stemcell architecture switch:
# operations/use-arm64-stemcell.yml
- type: replace
path: /stemcells/alias=default/architecture?
value: arm64
We believe the stemcell switch + cloud-config is sufficient for cf-deployment itself, but full validation would reveal if any additional changes. I also have a sample Graviton cloud-config ready. Happy to contribute these as a follow-up PR to cf-deployment once the RFC is accepted.
|
|
||
| ### Developer Workflow Benefit | ||
|
|
||
| ARM64 Linux support also benefits CF core developers on Apple Silicon MacBooks (M1/M2/M3/M4). All Go-based CF components can be cross-compiled locally using `GOOS=linux GOARCH=arm64`, enabling faster development and unit test iteration without needing remote infrastructure. |
There was a problem hiding this comment.
I was more thinking along the lines of: cloudfoundry/bosh-deployment#497. Which is close to working but running into rosetta issues. So the aim would be to have apple silicon running ARM linux docker VM (arm linux kernel) then use the bosh docker cpi with a ARM stemcell. This should just work, but it would be great to have it be part of the testing scope. This probably would also directly benefit / speed up the review process of all the other parts of this PR, because it would give many a way to verify the proposed changes locally.
There was a problem hiding this comment.
That's a great use case. I will add local BOSH-lite on Apple Silicon as a validation target in the testing scope. Once the ARM64 stemcell is available in Docker format, this should be straightforward to verify on M-series Macs.
| Some CF ecosystem tools already have ARM64 builds: | ||
| - **BOSH CLI** — already publishes ARM64 binaries | ||
| - **CF CLI** — already publishes ARM64 binaries | ||
| - **ARM64 JDKs** — production-ready ARM64 JDKs are available |
There was a problem hiding this comment.
Paketo Cloud Native Buildpacks are also ARM64 compatible.
There was a problem hiding this comment.
Thank you, I will add Paketo Cloud Native Buildpacks to the "Existing ARM64 Support" section alongside BOSH CLI and CF CLI.
| All BOSH releases in `cf-deployment` MUST produce ARM64 binaries alongside x86_64. Go-based components SHOULD cross-compile via CI with `GOARCH=arm64`. Components with C dependencies (e.g., Garden-runC, runC) MUST be validated with native ARM64 compilation. Components shipping pre-compiled binary blobs (e.g., JDK in uaa-release) MUST include ARM64 variants of those blobs, with packaging scripts selecting the correct variant based on stemcell architecture (`uname -m`). | ||
|
|
||
| ### Phase 3: ARM64 Stack and Buildpacks | ||
|
|
There was a problem hiding this comment.
When we did this for Paketo Buildpacks, most of the work and challenges were around automation and release processes. How do you store/access the binaries since there are two copies of everything ow? How do you monitor for updates since that process is now arch specific? How do you release all this throughout pipelines? Lot of assumptions in pipelines that have to be sorted out.
On the CF side, size of Buildpacks that bundle binaries with the buildpack is going to be an issue. Assuming that’s still being done, haven’t looked in a bit. Expect Buildpacks to be twice as large or need to be spilt up some way so they’re arch specific.
There was a problem hiding this comment.
A few thoughts:
Storage/access: Architecture-specific blobs stored separately, selected at staging time based on cell architecture. Similar to how Paketo handles it today.
Monitoring for updates: The update-checking process would need to be arch-aware checking for both x86_64 and ARM64 releases of each dependency.
Buildpack size: Agree this is an issue if bundles contain both architectures. Architecture-specific buildpack variants (separate downloads per arch) seems cleaner than doubling the size of a single bundle.
Pipeline assumptions: This is likely the largest effort area for classical buildpacks. Would appreciate any lessons learned from the Paketo ARM64 work that we could apply.
I'll add a note in the RFC acknowledging that buildpack release automation is a significant workstream that benefits from Paketo's experience.
- Add Paketo CNBs to existing ARM64 support list - Add BOSH Docker CPI on Apple Silicon to testing scope (ref bosh-deployment#497) - Add buildpack automation challenges (storage, size, pipelines) to Phase 3
| - Extend `bosh-linux-stemcell-builder` to produce ARM64 stemcells (Ubuntu Noble initially, Noble/Resolute Raccoon when applicable) | ||
| - Validate the BOSH agent on ARM64 (POC has demonstrated this works) | ||
| - Set up CI pipelines producing ARM64 stemcell builds (ARM64 CI workers will be provided) | ||
| - Validate the BOSH Director (Ruby) operates correctly on ARM64 (POC validated 134 gems) |
There was a problem hiding this comment.
Do we imagine users operating environments that use both AMD64 and ARM64 stemcells? If so, there will likely be work necessary in the Director itself. e.g. currently, the cloud config can only define a single compilation VM type, which will only work for compiling against one architecture.
There was a problem hiding this comment.
I believe for environments using both AMD64 and ARM64 stemcells, the BOSH Director would need to support architecture-aware compilation either multiple compilation VM types or the ability to route compilation to the correct architecture. I will add it as an open question / consideration for the mixed-architecture deployment model. For the initial implementation (all-ARM64 deployment), this isn't a blocker since compilation VMs would all be ARM64.
|
|
||
| # Open Questions | ||
|
|
||
| 1. **Stemcell naming convention:** How should ARM64 stemcells be identified? Proposed: `ubuntu-jammy-arm64` alongside existing `ubuntu-jammy` (implicitly x86_64). Open to community input on naming and metadata approach. |
There was a problem hiding this comment.
One option would be to follow the convention set by FIPS (which the pipelines build but do not publish to bosh.io) and embed the ARM64 into the name field (e.g bosh-aws-xen-hvm-ubuntu-noble-arm64 instead of just bosh-aws-xen-hvm-ubuntu-noble) while keeping the os field just ubuntu-<line name>. It would depend on whether we want users to opt-in simply by their choice of uploaded stemcells OR if we want them to be very explicit about wanting ARM64 in the manifest itself.
Regardless of naming approach, it would make sense to me to add the architecture as an additional metadata field as well.
There was a problem hiding this comment.
Thanks for the FIPS naming precedent, that's a useful reference. Embedding architecture in the name while keeping the os field as ubuntu-noble makes sense. Adding architecture as an explicit metadata field is also a good idea regardless of naming approach it makes the stemcell self-describing. I will update the open question to reference the FIPS approach as a proposed option.
This comes to mind as a new possibility for running locally on Mac: https://github.com/apple/container |
- Add BOSH Director architecture-aware compilation consideration for mixed AMD64/ARM64 environments to Foundational Infrastructure WG scope - Update stemcell naming open question to reference FIPS naming precedent and add architecture metadata field proposal
10 participants
This RFC proposes adding ARM64 architecture support to Cloud Foundry,
enabling deployments on ARM64-based infrastructure (AWS Graviton, Ampere
Altra, etc.) with 20-40% better price/performance.
A proof-of-concept validated all major CF components (Diego, Garden-runC,
Gorouter, Loggregator, Cloud Controller, UAA, BOSH) compile and execute
on ARM64 with zero source code changes.
Looking forward to community feedback on the proposed workstreams and
open questions.
Preview link
cc @beyhan