FermiHDI HD System: Operators & Admins Manual

This manual is about running an instance. Installing one is install/ADMIN_GUIDE.md, which covers host requirements, sizing, the cluster definition, and the Ansible and Kubernetes paths. The invariants below — the cpuset split, the core counts — are the ones the installer enforces, so a deployment it produced already satisfies them.

The FermiHDI HD System is a high-performance, strictly segmented Data Management System built for massive-scale analytics and isolated tenancy. It heavily leverages C++ native processing (DuckDB, DPDK, NORM multicast) paired with Go-based control planes (DMS-Core, Proxy) for orchestration.

High-Level Architecture

The system uses three distinct planes isolated at the network level:

1. Control Plane (Port 6986): Over-the-wire HTTPS mTLS APIs for node registration, policy distribution, and query telemetry orchestration. Records never pass through this network.
2. Data Plane (Port 6987): Unidirectional, ultra-fast binary streaming using the FermiHDI HD Wire Protocol (FHDWP). Data flows natively from Storage Nodes back to the requestor over TCP.
3. Internal DMS / CA Plane (Port 6985 / 6989): Secure channels exclusively connecting the Datastore Management System components to bootstrap new nodes and provision PKI identities.

Core Component Lifecycles

Datastore Management System (DMS)

The DMS is the single source of truth for the HD Instance. * DMS-Core: The orchestration engine. It distributes dataset schemas and routing topologies to proxies. * DMS-CA: Issues cryptographic certificates for new nodes joining the cluster. * DMS-WebUX: Provides the operator UI for dataset administration.

HD Proxy (Query Gateways)

Acts as the ingress gateway for the Control Plane. * Evaluates Attribute/Policy-Based Access Control (PBAC) using Open Policy Agent (OPA). * Dynamically rewrites queries with binary bitmask filters. * Forwards authorized payloads to Storage Cluster Controllers.

Storage Cluster Controllers (SCC)

SCCs orchestrate queries across their domain. * They operate as Fan-Out nodes, broadcasting queries to their underlying Storage Nodes (SNs). * SCCs ingest raw records, hashing and distributing them identically across their SNs using NORM (NACK-Oriented Reliable Multicast) over UDP port 6988.

Storage Nodes (SN)

The raw engine layer, executing the native C++ retrieval pipelines. * They map raw FHDWP payloads directly to DuckDB Apache Arrow buffers. * Once data is found, they connect directly to the Client/Target Data Port to stream the response, bypassing the Proxy completely.

Operating Principles

• Immutable Deployments: The system is intended to be run completely inside container orchestration systems (Docker Compose or Kubernetes). Do not mutate host networking.
• Certificate Rotation: The system utilizes short-lived mTLS tokens. Loss of the DMS-CA means the entire cluster must be rebooted to regenerate the trust chain.
• Observability: All nodes emit OpenTelemetry metrics and traces to a shared OTEL Collector. Node-level health and Prom HTTP endpoints are exposed on port 8080 by default.

Container Deployment Model

All FermiHDI HD components (Storage Node, Storage Cluster Controller, HD Proxy, Message Bus Ingester, and DMS components) must always be deployed as containers, even when running as the only workload on a dedicated bare-metal server. This is a deliberate design principle to align with modern data-centre management practices and orchestration systems (Docker Compose, Kubernetes, Canonical Juju, etc.).

DPDK / SPDK Components: cpuset Split Invariant

Components that use kernel-bypass hardware (Storage Nodes and Storage Cluster Controllers with DPDK networking or SPDK storage) have an additional operational invariant that must be followed on every deployment:

The container cpuset and the MASTER_CPUSET environment variable must reference disjoint sets of CPUs. DPDK-pinned cores must never appear in the container cpuset.

Why this matters: The container runtime shim (containerd-shim on Docker, CRI-O shim on Kubernetes) runs in the host scheduler but inherits the container's cpuset cgroup. If a DPDK core appears in the container cpuset, the shim will be scheduled on that core, evicting DPDK poll-mode threads. This was measured at over 300,000 shim evictions per 90-second window in FermiHDI HD profiling — sufficient to cause measurable write-path latency spikes and query P99 regressions. The isolcpus kernel parameter alone does not prevent this, because an explicit cgroup cpuset assignment overrides isolcpus.

Correct split:

Field	Must contain
cpuset (Docker Compose / K8s pod spec)	Overflow / management core(s) only
MASTER_CPUSET (env var)	All DPDK/SPDK-pinned cores — excluded from cpuset

See hd_sn/docs/user_guide.md — Step 3a for full examples, profiling evidence, and Kubernetes guidance.

Deployment Planning: CPU Core Count Requirements

Use the following table when sizing a host for a FermiHDI HD Storage Cluster. All figures are logical CPU (thread) counts, based on profiling data from RC12 benchmark runs. These figures assume SMT (Hyper-Threading) is enabled.

Role	DPDK Cores (MASTER_CPUSET)	Overflow Cores (cpuset)	Isolation requirement
Storage Node (SN)	4	2	Overflow must be on a different physical core from DPDK cores
Storage Cluster Controller (SCC)	2	1	SMT sibling acceptable on space-constrained hosts (see note below)
Query Proxy (HD Proxy)	1–2	1	No strict isolation — proxy is not DPDK-intensive

Minimum host sizes for a single Storage Cluster (1 SCC + 3 SNs):

Scenario	Logical CPUs
Sandbox / development	16 (8 physical cores, SMT×2)
Production minimum (clean overflow isolation)	20+
Production recommended (per-SN NUMA isolation)	32+ (dual NUMA socket)

SCC on fully-allocated hosts: When no spare physical core is available for a dedicated SCC overflow (as in the 16-CPU sandbox), the SCC may safely operate with cpuset == MASTER_CPUSET. The SCC reader thread is not a DPDK PMD spin-poller; it blocks between batches. Profiling shows the resulting scheduler events on SCC DPDK cores are swapper idle handoffs, not active preemption. This is benign. On hosts with ≥ 18 logical CPUs, allocate a dedicated overflow core to the SCC.

Disaster Recovery & Troubleshooting

Node Quarantine

If an SN begins throwing Bad Batch Notice (BBN) Control Messages, quarantine the node via the WebUX. The SCC will dynamically re-route queries to healthy replicas within the Storage Cluster.

Re-establishing Data Plane Sync

If an SN falls behind due to network partition on the NORM interface (UDP 6988), reboot the SN. The SCC will trigger a cold-boot sync event over standard TCP fallback on the Control Plane to ensure data integrity before marking the node Ready.