Telemetry Schema

Every component that emits spans (the gNB planes, the RAN controller, the RANN-P forwarder, CU-IP) follows the conventions on this page, so that one operation, an operator editing spec.mobility or an Intelligence Function emitting a decision, reads as one trace across process, language and repository boundaries. This is the reference for reading those traces, and, at the end, for the metrics the fronthaul agent exports; how to get at either is Read Logs, Traces, and Metrics.

Data Topology — What Lands Where

gNB planes (C++)      OTLP/HTTP :4318 ─┐
controller, cu-ip,                     ├─► otel-collector ──► traces ──► Tempo
forwarder (Python)    OTLP/gRPC :4317 ─┘        │
                                                └──────────► logs ────► ClickHouse otel.otel_logs

• Traces → Tempo (monitoring namespace, 24 h retention, emptyDir — a pod restart clears history). Grafana's Tempo datasource is the query surface.
• Logs → ClickHouse otel.otel_logs (24 h TTL). Every log record emitted inside an active span carries that span's identity in the TraceId/SpanId columns. The trace-id join is the bridge between the two stores, and it is manual: Grafana's span-to-logs link does not support the ClickHouse datasource. The query is on Read Logs, Traces, and Metrics.
• The gNB's counters are log lines. The CU-CP's periodic counter lines (RRC/NGAP handover counters, enabled by metrics.periodicity.cu_cp_report_period in the controller-generated overlay) arrive as log lines; the Mobility dashboard counts log markers. The one scraped metrics source is the fronthaul agent (Metrics); the collector's own metrics on :8888 and the series Tempo's metrics-generator writes to its local WAL are neither scraped nor stored.
• ClickHouse holds otel_logs, the fronthaul agent's otel_metrics_gauge / otel_metrics_sum, and the O1 collector's o1.* tables (O1 Tables); the OTel exporter sits on the logs and metrics pipelines only, so no otel_traces table is created — traces live in Tempo.
• Not every component is a producer. The core controller (racora-core-controller) and the device plugins have no OpenTelemetry; their output is kubectl logs only and never reaches ClickHouse. Read the core controller with kubectl -n racora-system logs deploy/racora-core-controller. CU-IP is a producer: its library defaults to the in-cluster collector.

Resource Attributes (per Producer, on Every Span and Log)

key	value
service.name	cu-cp, cu-up, du (one per gNB plane Racora deploys), racora-controller, rann-p-forwarder, cuip
cluster	deployment identity. Canonical default: racora. The collector's resource processor (clusterLabel chart value) is the authority and upserts it on everything passing through; producer-side values are aligned defaults.
pod.name, pod.namespace	from the downward API (POD_NAME, POD_NAMESPACE)

Naming

• Span names: lowercase dotted <layer>.<workflow>[.<phase>] — cucp.mobility_neighbor_add.execute, dispatch.rollout_wait, rannd.emit.
• WS command spans are always cucp.<cmd>.execute, tracer scope cucp.remote_command — "is a remote command" is queryable as a class (scope/layer), "which command" is the span name.
• Tracer scopes: the gNB uses <layer>.<module> — cucp.remote_command, cucp.f1ap, du.cell_lifecycle; the Python producers use their service name, with a module suffix where one module dominates — racora-controller, racora-controller.mobility-sync, racora-controller.precondition, rann-p-forwarder, cuip.
• Attributes are namespaced by domain (mobility.*, dispatch.*, rollout.*, f1ap.*, infer.*, cmd.*, ws.*) except the bare identity attributes below.

The Layer Taxonomy

Every span carries the required attribute rann.layer, closed set:

value	emitted by
CTRL	controller reconcile/sync/WS-client spans
DISPATCH	controller decision loop
CUCP-WS	gNB CU-CP remote-command spans
CUCP-F1AP	gNB CU-CP F1AP procedure spans
DU-MNG	gNB DU lifecycle markers
RANN-P	measurement export (forwarder rannp.batch_sent, cuip rannp.batch_received)
RANN-D	cuip decision emit/serve
INFER	cuip intelligence-function evaluation
GRAPH	cuip graph substrate

Identity Attributes (Fixed Names and Types)

attr	type	meaning
nci	int64	NR Cell Identity (decimal)
neighbor_nci	int64	neighbor's NCI
pci, target_pci	int64	physical cell id
rnti	int64	UE RNTI
tac	int64	tracking area code
plmn	string	e.g. "90170"
cell	string	NRCell resource name (controller/cuip side)
cell_index	int64	DU-internal cell index (DU side)
report_cfg_id	int64	measurement report config id
decision_id	string	<function>:<cell>:<field> (RANN-D contract)
function	string	Intelligence Function: pci, anr, …
ws.cmd	string	WS command name, on controller-side spans that wrap one

Outcome and Error Convention

Every span carries two dimensions where they apply:

1. Classification: a per-domain string attribute recording what happened: dispatch.outcome, cmd.outcome (executed / tolerated / failed / unavailable), mobility.sync_outcome (applied / retry / unavailable), mobility.trigger_outcome (triggered / unknown_ue / unknown_target_cell / dispatched), f1ap.outcome, rollout.outcome.
2. OTel span status: ERROR (with a short message) when the operation failed from its caller's perspective; tolerated, converged and skipped outcomes leave the status UNSET. Tempo's error-rate views read only the span status, so the attribute says why and the status says whether it counts as an error.

Propagation Contract

leg	mechanism
controller → gNB (WS)	top-level traceparent JSON key (W3C) on every command payload, injected by the WS client from the active span; the gNB extracts only — absent/malformed ⇒ the command span is a fresh root, never an error
cuip → controller (decisions)	Decision.trace_context column (W3C), captured per decision inside its rannd.emit span; the controller uses it as the parent context for dispatch.consider
in-process gNB (cross-thread)	the WS command span follows the command onto the CU-CP executor, so a command's whole execution is one span

Not propagated. Arrow Flight do_put carries no context: the forwarder's rannp.batch_sent and cuip's rannp.batch_received are disjoint roots, correlated by time and record count. The controller's decision fetch (rannd.serve) is a fresh root; the decision's own trace_context is the stitch. WS replies carry no trace id. The gNB's handover procedures beyond the trigger command, and the DU-side remote commands (ssb_set, pci_set, sib_update, rrm_policy_ratio_set), emit no spans.

Span Catalog

gNB CU-CP Remote Commands (Scope cucp.remote_command, rann.layer=CUCP-WS)

All fourteen WS commands emit cucp.<cmd>.execute with their post-parse identity attributes and — on payloads that fail validation — a parse_error attribute (the five cell-lifecycle commands keep their original cgi.parse_error key). All fourteen commands record dispatch.outcome ∈ {accepted, rejected}; the two triggers also record mobility.trigger_outcome.

span	identity attributes
cucp.cell_lock/.cell_unlock/.cell_bar/.cell_unbar/.cell_status .execute	plmn, nci
cucp.mobility_cell_set.execute	nci, gnb_id_bit_length [+ plmn, pci, tac, band if present]
cucp.mobility_cell_remove.execute	nci
cucp.mobility_neighbor_add.execute	nci, neighbor_nci, report_configs_count
cucp.mobility_neighbor_remove.execute	nci, neighbor_nci
cucp.mobility_report_config_set.execute	report_cfg_id, report_type
cucp.mobility_report_config_remove.execute	report_cfg_id
cucp.mobility_periodic_report_set.execute	nci, [report_cfg_id], periodic_report.action ∈ {set, clear}
cucp.trigger_handover.execute	pci, rnti, target_pci, plmn, tac, mobility.trigger_outcome
cucp.trigger_conditional_handover.execute	pci, rnti, target_pcis_count, timeout_ms, mobility.trigger_outcome=dispatched

Notes: the CU-CP rejects trigger_handover for an unknown UE or an unknown target cell, and mobility.trigger_outcome names which (unknown_ue, unknown_target_cell, else triggered); trigger_conditional_handover records dispatched. Config commands additionally record dispatch.failure ∈ {queue_full, timeout} when the CU-CP executor could not take the command (otherwise indistinguishable from a rejection).

gNB Internals

• cucp.f1ap.tx.gnb_cu_configuration_update (scope cucp.f1ap, rann.layer=CUCP-F1AP) — attrs f1ap.procedure, f1ap.cells_to_be_*_count, f1ap.outcome, f1ap.success.
• du.cell_stop_started / du.cell_stop_completed (scope du.cell_lifecycle, rann.layer=DU-MNG) — zero-duration markers; _started carries cell_index, removal_mode, _completed carries cell_index, outcome=success.

Controller (rann.layer=CTRL Unless Noted)

span	attributes
cucp.mobility_sync	mobility.trigger ∈ {reconcile, resync-timer}, mobility.plan_size, mobility.sync_outcome, mobility.executed, mobility.tolerated, mobility.failed; ERROR on retry
cucp.mobility_sync.cmd (child, one per WS op)	ws.cmd (the wire command name), identity attrs from the op's kwargs (nci, neighbor_nci, report_cfg_id, plmn, report_type, report_configs_count), cmd.outcome; ERROR + exception on failed, ERROR on an unreachable surface (an old image's unavailable is tolerated, not ERROR). The WS client injects this span's traceparent, so the gNB's cucp.<cmd>.execute parents to the specific command
cucp.cell_lock / cucp.cell_unlock	ws.cmd, cell, plmn, nci (+ rollout_confirmed on unlock)
cucp.drain	cell, wait_s
dispatch.consider (rann.layer=DISPATCH)	cell, decision_id, function, dispatch.outcome (applied, a skipped_* gate, error, or deferred when the live object could not be re-read before acting); parented by the decision's trace_context
dispatch.apply (DISPATCH)	function, dispatch.field, dispatch.from, dispatch.to
dispatch.rollout_wait (DISPATCH)	function, cell, du_deployment, rollout.* — emitted only for functions whose post-apply policy waits on a DU rollout (pci; ANR applies emit none)

cucp.cell_lock/cucp.cell_unlock also carry function when driven by the decision loop.

RANN-P / CU-IP

span	attributes
rannp.batch_sent (forwarder, RANN-P)	rann.records, rann.total
rannp.batch_received (cuip, RANN-P, root)	rann.records
graph.update (GRAPH)	— (the streaming edge update; no attributes beyond the layer)
graph.swap (GRAPH)	graph.nodes, graph.edges
infer.eval (INFER, one per engine invocation)	function, infer.decisions, infer.decision_cells (capped at 32, + infer.decision_cells_truncated); the ANR engine adds anr.unknown_neighbor_count when UEs measure cells that are not declared NRCells
rannd.emit (RANN-D, one per decision)	cell, nci, graph_node_id, decision_id, function, reason, confidence, decision.current, decision.recommended, per-function fields via emit hooks (pci_* for PCI; anr.adds, anr.removes, anr.neighbor_count for ANR); captures the per-decision trace_context
rannd.serve (RANN-D, root)	decisions.count — the /decisions/active fetch

The Decision-Loop Trace

An autonomous PCI change reads as one trace:

rannd.emit (cuip: the decision)
└── dispatch.consider (controller, via Decision.trace_context)
    ├── cucp.cell_lock → cucp.cell_lock.execute (gNB)
    ├── cucp.drain
    ├── dispatch.apply (function=pci)
    ├── dispatch.rollout_wait
    └── cucp.cell_unlock → cucp.cell_unlock.execute (gNB)

and a declarative mobility change:

cucp.mobility_sync (controller, trigger=reconcile|resync-timer)
└── cucp.mobility_sync.cmd (ws.cmd=mobility_neighbor_add, nci=…, neighbor_nci=…)
    └── cucp.mobility_neighbor_add.execute (gNB, via WS traceparent)

An ANR decision combines the two: its dispatch subtree is minimal — no lock (ANR decisions carry no preconditions) and, per its post-apply policy, no rollout wait and no unlock:

rannd.emit (cuip, function=anr, anr.adds/anr.removes)
└── dispatch.consider (controller)
    └── dispatch.apply (function=anr, dispatch.field=spec.neighbors)

— and the actuation then flows through the reconcile's mobility sync exactly like an operator edit (the cucp.mobility_sync tree above, with mobility_neighbor_add/mobility_neighbor_remove command spans). The K8s spec patch is the seam between the two traces; status.lastApplied.anr.decisionId links the applied spec state back to the decision.

Verifying by Hand

A hand-sent runtime command with a traceparent lands under that trace id in Tempo, and the controller's log lines around a sync carry the same trace id in ClickHouse; the recipes are on Read Logs, Traces, and Metrics.

Metrics

Metrics are the third kind of telemetry, and so far one component produces them: the fronthaul agent on every fronthaul-ready node, which serves them on http://<node>:9478/metrics in the Prometheus text format. The collector scrapes every agent (racora-monitoring.otelCollector.scrape.fronthaulAgent, 15 s) and writes the samples to ClickHouse: gauges to otel_metrics_gauge, counters to otel_metrics_sum, one row per sample with the name in MetricName, the value in Value, the time in TimeUnix and the labels in the Attributes map (Attributes['cell']); the scrape adds node, the node the agent runs on. The Open Fronthaul dashboard reads these tables.

Metric	Kind	Labels	Meaning
racora_fronthaul_deep_idle_states_enabled	gauge		idle states deeper than 10 us still enabled on the isolated cores
racora_fronthaul_dhcp_replies_total	counter	cell, type (OFFER, ACK, NAK)	DHCP replies sent per cell and type
racora_fronthaul_link_irqs_on_isolated_cores	gauge		the fronthaul port's IRQs affined to an isolated core
racora_fronthaul_link_mtu	gauge		the fronthaul port's MTU
racora_fronthaul_link_rx_vlan_filter	gauge		1 when hardware VLAN receive filtering is on (tagged frames do not reach the host)
racora_fronthaul_link_txqueuelen	gauge		the fronthaul port's transmit queue length
racora_fronthaul_oru_healthy	gauge	cell	1 when the O-RU answers on its NETCONF port over the fronthaul port and the clock is sane
racora_fronthaul_oru_reason	gauge	cell, reason	the agent's reason for the O-RU's health (1 for the current reason label)
racora_fronthaul_oru_vlan	gauge	cell	the VLAN the O-RU was given its address on (0 untagged, -1 not attached by the agent)
racora_fronthaul_ptp_known	gauge		1 when a fresh PTP status from the host is available
racora_fronthaul_ptp_ok	gauge		1 when the clock can serve a radio (ptp4l running, the port in its role, TAI on the wire)
racora_fronthaul_ptp_phc_freq_ppb	gauge		the NIC clock's frequency adjustment
racora_fronthaul_ptp_phc_minus_system_seconds	gauge		the NIC clock minus the system clock (expected: the UTC offset)
racora_fronthaul_ptp_status_age_seconds	gauge		age of the host's PTP status file
racora_fronthaul_unknown_oru_requests_total	counter		DHCP requests from MACs no cell names

A gauge keeps its last value; racora_fronthaul_oru_reason is a set of 0/1 series, one per reason the agent has given a cell, with the current one at 1. The counters only grow (per agent process; a restarted agent starts them again at 0). A cell that is deleted takes its series with it.

SELECT TimeUnix, Attributes['cell'] AS cell, Attributes['reason'] AS reason
FROM otel_metrics_gauge
WHERE MetricName = 'racora_fronthaul_oru_reason' AND Value = 1
ORDER BY TimeUnix DESC LIMIT 20

O1 Tables

The O1 collector (O1 and the SMO Functions) stores what the O1 adapters send in the database o1 of the same ClickHouse. Nothing passes through the OTel collector: the adapters post VES events and their PM stream to the collector, which inserts typed rows.

Table	Row	Columns
o1.events	every VES event, verbatim	received_at, element, domain, namespace (the stndDefinedNamespace), event_name, event_id, source_name, sequence, body (the JSON)
o1.alarms	one 3GPP fault notification (raise, change or clear)	event_time, received_at, element, system_dn, source_name, alarm_id, alarm_name, notification_type, state (raised or cleared), perceived_severity, alarm_type, probable_cause, specific_problem, additional_text, additional_information (a map), notification_id, sequence
o1.current_alarms	a view: the alarms whose last notification is a raise	element, alarm_id, alarm_name, perceived_severity, probable_cause, specific_problem, raised_at
o1.pm	one metric sample of the PM stream	ts, element, nf_type, object_type, plmn, cell_id (-1 when the metric has none), job_id, granularity, source (stream), name, value

element is the cell's name for a DU and its O-RU, cu-cp or cu-up for the CU planes, unknown when nothing identified the sender: the adapters' bodies carry a fixed identity, so the collector takes the element from the VES basic-auth username the controller gives each adapter, unless the body names one. Retention is per table (racora-smo.collector.store.ttl). The metric names in o1.pm are the adapter's stream names: the TS 28.554-style KPIs it derives (DlUeThroughput_Cell, DLLat_gNB-DU, DLDelay_gNBCUUP, ...) and the gNB's own JSON keys flattened with dots (cells.cell_metrics.average_latency, du.du_high.mac.dl.max_latency_us); the O1 KPIs dashboard's names are checked against the pinned sources in CI.