Upgrade Racora

A Racora release is one chart with one pinned image set. Upgrading means applying the next release to the cluster you have: on the k3s platform by re-running the installer on the control node, on a cluster you run with helm upgrade. Kubernetes then rolls exactly the pods whose spec changed, on the nodes they run on. There is no RAN version per node: the RAN is the control node's release, and a worker only ever runs the pods scheduled onto it.

k3s Platform

On the server only:

curl -sfL https://get.racora.io | INSTALL_RACORA_VERSION=vX.Y.Z sh -

The re-run keeps the core provider and AMF address recorded at install time (/etc/racora/install.env) unless you give others; any INSTALL_K3S_* or K3S_* flag you set at install must be given again, because k3s rewrites its service file from them. The installer downloads the release's k3s binary only if the pinned upstream version changed, re-renders the core provider's host units, restarts k3s when its binary or service file changed, and only then stages the payload: the CRDs are re-applied out-of-band (racora-crds.yaml) and the HelmChart is rewritten with the new chart package. Same name, so k3s's helm-controller runs helm upgrade, and the Deployments roll with the new images and configuration; a DU on a worker pulls its new image from the registry. The HelmChart sets failurePolicy: retry and a 300 s timeout per helm operation: a failed upgrade leaves the previous revision deployed and serving (If an Upgrade Fails).

Before the binary swap and the payload stage, the installer runs a pre-flight: the new release is dry-run against the running cluster, inside it, by a one-off Job from helm-controller's own image, with your HelmChartConfig applied. A render refusal, an object helm cannot adopt or a value the API server refuses stops the run there, with helm's message, and nothing has changed. Then it compares the release record of the running release with the one the new release renders and prints the impact: which component images change, what each change rolls, whether cells drop, and whether the k3s pin moves (every worker then needs a node re-run). To see the report and stop:

curl -sfL https://get.racora.io | INSTALL_RACORA_VERSION=vX.Y.Z INSTALL_RACORA_PREFLIGHT=only sh -

INSTALL_RACORA_PREFLIGHT=off skips the pre-flight (the apiserver must answer for it to run; when it does not, the installer says so and proceeds).

Workers usually need nothing. Re-run node mode on a worker (with its K3S_URL, K3S_TOKEN and the same INSTALL_RACORA_VERSION as the control node) only when the k3s pin moved, which the k3s column of Release History shows: on an already provisioned host it validates and joins straight away, swapping the agent binary. After that restart, the USRP device plugin on the node must re-register (Add a Radio Node).

A Cluster You Run

A plain helm upgrade with the two overlays from the release's installer package (Install onto an Existing Kubernetes Cluster); no host is involved:

helm upgrade --install racora oci://registry.gitlab.com/cognitive-network-solutions/racora/charts/racora \
  --namespace default --version X.Y.Z --reset-values \
  --atomic --cleanup-on-fail --wait --timeout 15m \
  -f installer/platforms/kubernetes/values.yaml \
  -f installer/cores/<provider>/values.yaml \
  -f my-values.yaml \
  --set global.systemDefaultRegistry=<registry>

This is the platform's command with --reset-values and the new release's installer/ overlays. --atomic is named --rollback-on-failure on Helm 4 (the installer probes which spelling your helm has): with it helm waits for every workload of the new revision and rolls back to the previous one when the upgrade fails (If an Upgrade Fails). Pass --reset-values and your file, not --reuse-values: reused values carry every key the previous release accepted, and v0.8.0 refuses the retired core.enabled and global.amfAddr, and racora-core.mcc/mnc that disagree with global.network.plmn. my-values.yaml is the file that carries your placement and identity; the registry stays a --set (Install onto an Existing Kubernetes Cluster). For an external core put global.core.provider and global.core.amfAddr in that file: the overlay's amfAddr is a placeholder the chart refuses.

With the installer: INSTALL_RACORA_PLATFORM=kubernetes INSTALL_RACORA_VERSION=vX.Y.Z RACORA_HELM_EXTRA_ARGS='-f my-values.yaml', with the same INSTALL_RACORA_CORE, INSTALL_RACORA_AMF_ADDR and RACORA_REGISTRY as at install. This platform keeps no record, so the installer reads the deployed release's provider and refuses a run without INSTALL_RACORA_CORE that would switch it: for an external core pass INSTALL_RACORA_CORE=external INSTALL_RACORA_AMF_ADDR=<amf> on every installer run. The installer always resets values, so the file is passed on every run.

Run through the installer (INSTALL_RACORA_PLATFORM=kubernetes), the same command is first run with --dry-run=server (helm 3.13 or newer): a release that would not render or would not adopt an object stops there, and on an upgrade the release record of the running release is compared with the rendered one and the impact printed, as on k3s. INSTALL_RACORA_PREFLIGHT=only prints it and stops; off skips it.

Offline (k3s Platform)

Fetch the new release's bundle on a connected machine (`scripts/fetch-bundle.sh vX.Y.Z

`, from the repository at the tag; a release whose page lists no image archive cannot be upgraded offline), carry it over, and re-run the installer on the control node with `INSTALL_RACORA_ARTIFACT_PATH=`. The image archive is imported into the control node's containerd only; nothing propagates images to workers. Re-run `node` mode on each worker with the same bundle path, or import the archive there yourself ([Use Your Own Registry or Install Offline](registries-and-offline.md)).

Upgrading a v0.7.x Install to v0.8.0

v0.8.0 makes the core a provider, declares the network identity once, and adds the Subscriber API with its own controller. Re-running the installer on the control node does everything for an Open5GS install; what changes:

• The PLMN moves. The chart no longer has racora-core.mcc and mnc; the identity is global.network (plmn, tacs, slices), default PLMN 90170, TAC 7, SST 1. A HelmChartConfig that still keys the PLMN under racora-core without global.network is refused by the installer's pre-flight before the payload is staged or k3s restarted (an offline bundle's k3s binary is already copied into place by then), with the migration in the message: add global.network next to the old keys first (v0.7.x ignores it; the new chart accepts the pair while both agree), upgrade, then drop the racora-core key, or the whole object if the default is your identity. A pair that disagrees fails the render.
• An external core declared the v0.7.x way (core.enabled: false and global.amfAddr in the HelmChartConfig) has no RACORA_CORE in its install record, so a plain re-run would select Open5GS; the pre-flight refuses that. Re-run with INSTALL_RACORA_CORE=external INSTALL_RACORA_AMF_ADDR=<amf> (recorded from then on), then drop core and global.amfAddr from the HelmChartConfig: the v0.8.0 render is refused while they are there and v0.7.x keeps serving, and the upgrade completes once they are gone. Never drop them before the re-run, or v0.7.x deploys Open5GS.
• The core rolls once, because its chart is renamed (racora-core to racora-core-open5gs). Its Deployment, Service and subscriber claim are adopted in place, same objects, same data, and the CU-CP re-associates NGAP by itself within seconds. Later chart bumps no longer roll it.
• Values keys. core.enabled and global.amfAddr become global.core.provider (open5gs or external) and global.core.amfAddr; racora-core.* becomes racora-core-open5gs.*; the Open5GS chart's subscriber.sqn, which fed the image's QoS class field, is subscriber.qci. core.enabled and global.amfAddr fail the render with their replacement, racora-core.mcc/mnc are accepted while they agree with global.network.plmn, and the other retired keys are ignored, so move them.
• On a cluster you run the same notes apply through helm upgrade: drop the retired keys from your values file first, then upgrade with --reset-values and the file.
• New in racora-system: the racora-core-controller Deployment and the subscribers.racora.io CRD; the CRD asset is one racora-crds.yaml. The old staged nrcell-crd.yaml is removed from the manifests directory; its Addon object is left in place on purpose.
• Host units move from core-support/ to the provider module (cores/open5gs/host/), are re-rendered from there and recorded in /etc/racora/install.env, so the uninstall removes exactly those.
• Subscribers already in the core (seeded, or added through the WebUI) are untouched. Declaring one as a Subscriber adopts it (Add Subscribers).

Upgrading to v0.9.1: One Key per Component

v0.9.1 moves every image pin to global.components.<name> (repository, tag, optional registry), one key per component (Update a Component). The keys that used to sit under a subchart (racora-cu.cuCp.image.tag, racora-cu.cuIp.image.tag, racora-controller.gnbImage, racora-controller.mplaneImage, racora-core-open5gs.image.tag, the imageRegistry of a subchart, ...) are refused by the render, each with the key that replaced it. A HelmChartConfig or values file that still carries one fails the pre-flight before anything changes; move the value to its component key and re-run. The release also writes the release record, racora-release in racora-system, that the pre-flight of the next upgrade compares; the upgrade to v0.9.1 itself has no record to compare with and reports every component as possibly rolling.

Service Impact

An upgrade rolls the pods whose spec changed, and what that means on air depends on the pod:

Pod	While it restarts
CU-CP	every cell is off air and every UE is released: the DUs lose F1 and re-attach, the CU-UP re-establishes E1, the core re-associates NGAP. Its spec changes when its image pin, its pod template or the racora-cu chart version changes (the chart version is a pod label).
CU-UP	user-plane sessions drop; the cells stay on air
a DU	that cell is off air; the other cells are unaffected
CU-IP	decisions pause; the cells keep running
the core (Open5GS)	UEs re-register once the AMF is back; the subscriber database stays on its claim
racora-controller, racora-core-controller, the device plugins, the fronthaul agent	nothing on air; the controller re-reconciles every cell when it starts
the O1 collector	nothing on air; the events the O1 adapters send while it restarts are lost (they do not retry)

A release that changes what the controller puts into the DU pods it generates rolls those DUs once when the new controller reconciles them; the pre-flight report attributes nothing on air to the controller, so read the release notes. The release adding the O1 collector gives every Open Fronthaul cell's M-plane sidecar its address; the release adding O1 to the elements gives every DU and both CU planes their NETCONF server and adapter, so every cell drops once, and an element pod's start takes the NETCONF server's model install longer from then on.

The k3s restart itself is not service-affecting: pod containers keep running while k3s is down. A release whose pins change the gNB image, the CU-CP's sidecar or the racora-cu chart needs a window in which every cell may drop. The installer's pre-flight prints exactly this before it changes anything (INSTALL_RACORA_PREFLIGHT=only to see it and stop); by hand, compare the running release record with the new release's chart-values.yaml:

kubectl -n racora-system get configmap racora-release -o yaml

What Rolls, and What It Keeps

Component	State	On an upgrade
racora-controller	stateless; identity, PCI and neighbors are written to the NRCell objects	loses nothing
racora-core-controller	stateless; every Subscriber is re-applied from the API on a cadence	loses nothing
NRCell and Subscriber objects	etcd (CRDs)	durable
CU-CP, CU-UP, CU-IP, DUs	stateless; configuration regenerated from the NRCells	roll
Device plugins	stateless DaemonSets	roll
Open5GS subscriber database	mongodb on the open5gs-mongodb PersistentVolumeClaim (persistence.enabled, default on, helm.sh/resource-policy: keep)	survives rolls and upgrades
ClickHouse, Tempo	emptyDir by default; a claim with racora-monitoring.<component>.storage.persistent=true	stored telemetry is lost on a roll
Grafana	emptyDir	dashboards you built by hand are lost; provisioned ones return

The subscriber claim is ReadWriteOnce, which fits: the core is a Recreate singleton pinned to the control node, and with k3s's local-path StorageClass the volume is node-local. What an uninstall does with the claim is on Uninstall Racora; to wipe the database alone, kubectl -n 5g-core delete pvc open5gs-mongodb. Declared Subscribers are re-applied by the core controller, so a wiped database converges back to what is declared.

The CRDs have one version, v1alpha1, and no conversion. Re-applying an older racora-crds.yaml over a schema the cluster already advanced can be rejected.

Two things to keep in mind:

• A release pins its controller and CU-IP together. If you stage components by hand, move the controller first: an older controller applies decisions from a newer CU-IP without the semantics for them.
• Old release images stay on the nodes until kubelet's image garbage collection runs, which starts at its disk-usage threshold. Once the upgraded pods are verified, sudo k3s ctr images rm <ref> prunes superseded pins. A component key you set in the HelmChartConfig (global.components.<name>) still applies after the upgrade; remove it from the valuesContent once the release carries the version you wanted, and keep the object, which also carries your identity and registry (Override Chart Values).

If an Upgrade Fails

The two platforms fail differently.

• A cluster you run. The installer runs helm with rollback on failure (--atomic, named --rollback-on-failure on Helm 4; the installer probes which spelling your helm has), --cleanup-on-fail, --wait and a 15-minute timeout: helm waits for every workload of the new revision, and when one does not become ready, or the API server refuses an object, it rolls back to the previous revision and removes what the failed attempt created. On a first install the same flags remove the failed installation, so the next run starts clean. helm -n default history racora shows the revisions. A plain helm upgrade without those flags leaves the release failed; helm -n default rollback racora <revision> restores the previous one.

• k3s. The HelmChart the installer writes sets failurePolicy: retry, and the installer sets the same on your HelmChartConfig on every in-place run, because helm-controller takes the policy from the HelmChartConfig when one exists and its CRD defaults an omitted field to reinstall; keep failurePolicy: retry in your manifest (Override Chart Values). A refused render (a retired key, an identity the core cannot serve) fails the job before any revision is recorded, and nothing changes. A failure during the apply (an object the API server refuses, an object helm cannot adopt, a job interrupted mid-upgrade) records a failed revision; helm applies the release object by object, so the objects applied before the failing one already run the new release while the rest still run the previous one. Nothing is uninstalled. helm-controller's job retries on its own backoff, and again whenever the HelmChart or the HelmChartConfig changes, so once the cause is fixed the next attempt completes the upgrade.

  The release owns what its chart renders. A hand edit of a rendered field (kubectl patch, kubectl edit) makes kubectl a co-owner of that field; the HelmChart the installer writes sets forceConflicts: true, so an upgrade takes the field back instead of failing on the conflict. Change a value through the HelmChartConfig, never by editing a workload: the next upgrade replaces the edit.

To recover on k3s:

1. Read the job: kubectl -n kube-system logs -l helmcharts.helm.cattle.io/chart=racora --tail=60 (--previous for the attempt before the current one). The last lines name the object and the reason.
2. Fix the cause:

   • exists and cannot be imported into the current release: invalid ownership metadata: an object with a name the chart renders was created outside the release. Delete it so helm creates it, or hand it to helm:

     kubectl -n <namespace> annotate <kind> <name> \
       meta.helm.sh/release-name=racora meta.helm.sh/release-namespace=default
     kubectl -n <namespace> label <kind> <name> app.kubernetes.io/managed-by=Helm

   • Invalid value or is invalid: a value in the HelmChartConfig the API server refuses. Correct it in your file and apply the file again.

   • a render refusal: the message names the key and its replacement (Upgrading a v0.7.x Install to v0.8.0).

3. Let the job retry, or trigger it: apply the corrected HelmChartConfig, or re-run the installer, which rewrites the HelmChart. Confirm with kubectl -n kube-system get job helm-install-racora and kubectl -n default get secret -l owner=helm,name=racora -L status,version.

Do not change the HelmChartConfig or re-run the installer while a job pod is running: helm-controller deletes the running job on every change, and a helm upgrade killed mid-apply is what records a failed revision. Check first:

kubectl -n kube-system get pods -l helmcharts.helm.cattle.io/chart=racora

To go back to the previous release on either platform, install it again (INSTALL_RACORA_VERSION=<older>): every tag is immutable, so the result is that release as it was. To hold one component on its previous image, pin it in the HelmChartConfig or your values file (Update a Component) and remove the key once the next release ships.