HD System Architecture

HD by FermiHDI is an ultra-high-performance, bare-metal acceleration Data Management System designed to bypass the Linux kernel for maximum networking and storage throughput.

Holistic System Topology

graph TD
    %% External Clients
    SDK_App[HD SDK Client]
    Client_Admin["DMS Admin UX (Port 3000)"]

    %% DMS Control Plane
    subgraph Control_Plane ["Management Layer (hd_dms)"]
        DMS((Data Management System))
        OpAMP_Server[OpAMP WebSocket Supervisor]
        DMS_CA[Internal Certificate Authority]
        DMS --> OpAMP_Server
        DMS --> DMS_CA
    end

    %% Network Entry and Routing
    subgraph Edge ["Network Edge (hd_proxy)"]
        Proxy[HD Proxy]
        OPA{Open Policy Agent}
        Proxy <--> OPA
    end

    %% Data Ingestion
    subgraph Ingestion ["Message Bus (hd_message_bus_ingester)"]
        Ingester[Message Bus Ingester]
        Kafka[(Kafka)]
        RabbitMQ[(RabbitMQ)]
        S3[(S3 / Filesystem)]
        Kafka -- "Zero-Allocation Parse" --> Ingester
        RabbitMQ -- "Reads Data" --> Ingester
        S3 -- "Reads Data" --> Ingester
    end

    %% Storage Cluster Components
    subgraph Storage_Cluster ["Storage Cluster (SC)"]
        SCC[Storage Cluster Controller]
        SN1[(Storage Node 1)]
        SN2[(Storage Node 2)]

        SCC -- "Load Balances Queries" --> SN1
        SCC -- "Load Balances Queries" --> SN2
    end

    %% Hardware Accelerators
    subgraph Acceleration ["Bare Metal / Acceleration"]
        DPDK[DPDK Networking]
        SPDK[SPDK Storage / OCF]
        SYCL[GPU SYCL Bitmask Filter]

        SN1 -. "Direct PCIe" .-> SPDK
        SN1 -. "Kernel Bypass" .-> DPDK
        SN1 -. "Compute Offload" .-> SYCL
    end

    %% Operations / Observability
    subgraph Telemetry ["Local Observability Stack Sidecars"]
        OTel(Local OTel Collector)
        SentryRelay(Local Sentry Relay)
    end

    Central_Telemetry([FermiHDI Cloud Sentry / OTel Gateway])

    %% Data Flow Connections
    SDK_App -. "REST API / Custom TCP" .-> Proxy
    Client_Admin -. "HTTPS Config / APIs" .-> DMS

    Proxy == "Authorized / Modified Query" ==> SCC
    Ingester == "FHDWP Form 1 — TCP, 1200-byte MTU (Jumbo: 8500)" ==> SCC

    %% Control Flow Connections
    DMS_CA -- "mTLS Key Provisioning" --> Proxy
    DMS_CA -- "mTLS Key Provisioning" --> Ingester
    DMS_CA -- "mTLS Key Provisioning" --> SCC
    DMS_CA -- "mTLS Key Provisioning" --> SN1

    %% Observability Paths
    Proxy -. "Metrics/Logs" .-> OTel
    Ingester -. "Metrics/Logs" .-> OTel
    SCC -. "Metrics/Logs" .-> OTel
    SN1 -. "Crash Dumps" .-> SentryRelay
    OpAMP_Server -. "Dynamic Rules via W/S" .-> OTel

    OTel -. "mTLS Batches" .-> Central_Telemetry
    SentryRelay -. "Scrubbed Errors" .-> Central_Telemetry

Component Architecture Breakdown

1. Data Management System (DMS) `hd_dms`

The authoritative root of the clustered ecosystem (written in Go / Astro). - Node Registration & Topology: Maintains the complete physical layout of hd_scc and hd_storage_node agents. Nodes check-in and register their capabilities. - DMS-CA (Root of Trust): An integrated Certificate Authority that automatically provisions short-lived X.509 certificates to support strict mTLS across all cluster data paths. - Config & Schema Engine: Synchronizes JSON schemas, Open Policy Agent (OPA) governance models, and dynamic telemetry instructions.

2. HD Proxy `hd_proxy`

A highly concurrent Go application acting as the PBAC (Policy Based Access Control) firewall and query scatter/gather router. - Client Interface: Direct target for hd_sdk connections requesting data. - OPA Governance (PBAC): The proxy evaluates queries dynamically utilizing an embedded Open Policy Agent component mapping Traefik/OAuth headers to enterprise PBAC policies synced seamlessly from the DMS. - Schema Modification: Rather than rejecting restricted queries outright, the proxy rewrites query boundaries. Forbidden metadata arrays are mathematically nullified (\x00), and enforced identities are permanently masked (\xff). - Scatter-Gather Parallelization: The proxy multicasts the validated query out to all designated SCCs concurrently and assembles their disparate success/timeout hashes into HTTP 207 Multi-Status payloads.

3. Message Bus Ingester `hd_message_bus_ingester`

A lightning-fast C++20 edge microservice performing ETL/Ingestion zero-copy translations. - Zero-Allocation Parsing: Uses Glaze C++ to interpret massive JSON/CSV arrays without dynamic memory mappings. - Event Slicing: Chops continuous streams from Kafka/RabbitMQ into strict deterministic 1200-byte pieces matching SPDK/DPDK requirements. - Binary Conversion: Translates verbose String formats into dense internal RowBinary arrays.

4. Storage Cluster Controller (SCC) `hd_storage_controller`

The high-availability gateway directly preceding hardware disks. - Load Balancing: The SCC acts as the definitive controller for the Storage Nodes within its Storage Cluster (SC). It prevents N-to-N connection pooling crises by accepting ingest/query operations and logically load-balancing them across the available SN infrastructure.

5. Storage Node (hd_storage_node)

The lowest-level physical footprint (C++). - DPDK / SPDK: Uses DPDK to poll the NIC and SPDK to stream byte buffers immediately to NVMe SSD queues. No Linux Interrupts or context switching. - Heterogeneous Storage & OCF: Implements advanced topologies like Open CAS Framework (OCF) caching, combining fast cache devices (e.g., Optane/NVMe) with slower core devices. - SYCL & Crypto Offload: Offloads hashing to SmartNICs/FPGAs (Bluefield, Alveo) and compiles complex filtering requests into SYCL instructions to evaluate across clustered GPUs (or CPUs) natively over local RAM.

6. Software Development Kit (hd_sdk)

Local Application Calculus: The SDK allows data engineers to run complex logic (SQL via embedded DuckDB, Vector evaluations via USearch, Graph analysis via cuGraph) directly inside the receiving local environment without requesting the centralized Storage cluster to parse string semantics. Instead, fermihdi_client maps natively zero-copied DPDK buffer results securely.

Host Requirements for Storage Nodes with SPDK/DPDK

Storage Nodes running in bare-metal acceleration mode (SPDK NVMe + DPDK transport) require specific host-level kernel configuration that cannot be satisfied at container start time. These must be applied once per physical host before deploying any SN with PCI passthrough storage.

1. IOMMU

IOMMU must be enabled in the kernel command line and in the system firmware (BIOS/UEFI).

CPU Vendor	GRUB parameter
AMD	`amd_iommu=on iommu=pt`
Intel	`intel_iommu=on iommu=pt`

iommu=pt enables passthrough mode, which eliminates IOMMU translation overhead for VFIO-assigned devices compared to full-isolation mode.

Verify after reboot:

dmesg | grep -i iommu
ls /sys/class/iommu/

2. Hugepages — Both 2 MB and 1 GB Required

DPDK's EAL memory allocator requires a 2 MB hugetlbfs mount. The 1 GB mount present by default is insufficient on its own — DPDK will log:

EAL: 1023 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size

and fall back to a degraded allocation path, significantly reducing NVMe throughput.

Page size	Mount point	Used by	Reserve
2 MB	`/dev/hugepages2M`	DPDK EAL (SPDK NVMe + HD engine transport)	1024 pages = 2 GiB
1 GB	`/dev/hugepages`	Large DMA buffers (optional)	4–8 pages

Recommended GRUB additions (merge with existing GRUB_CMDLINE_LINUX_DEFAULT):

default_hugepagesz=2M hugepagesz=2M hugepages=1024 hugepagesz=1G hugepages=4

3. CPU Isolation

For consistent, low-jitter benchmarking and production workloads, OS scheduler noise must be excluded from the cores assigned to the SN via MASTER_CPUSET.

Recommended GRUB additions (adjust CPU range to match your topology):

isolcpus=2-15 rcu_nocbs=2-15 nohz_full=2-15

This reserves CPUs 0–1 (and their SMT siblings) for the OS, Docker daemon, and SCC. The SN occupies CPUs 2–15 via MASTER_CPUSET=2-15.

Container cpuset Split Requirement

All FermiHDI HD components are always deployed as containers — even on dedicated bare-metal hardware — to align with modern data-centre management and orchestration systems (Docker Compose, Kubernetes, etc.).

isolcpus alone is not sufficient to protect DPDK poll-mode threads when running inside containers. A cgroup cpuset assignment that explicitly lists a CPU overrides isolcpus for every process in that cgroup, including the container runtime shim (containerd-shim, CRI-O shim). The shim runs in the host scheduler but inherits the container's cpuset; if DPDK cores appear in the container cpuset, the shim is legally scheduled on those cores.

The required pattern is a strict two-group split:

Group	Specification	Used by
Overflow cores	`cpuset:` in Docker Compose / K8s pod spec	Container runtime shim, control-plane threads, non-DPDK processes
DPDK cores	`MASTER_CPUSET` environment variable	DPDK/SPDK poll threads, self-pinned via `pthread_setaffinity_np` at SN startup

The DPDK cores must never appear in the container cpuset field. The SN process self-pins its DPDK/SPDK threads to the MASTER_CPUSET cores at startup, escaping the cgroup cpuset restriction. The container shim, which never calls pthread_setaffinity_np, stays confined to the overflow cores.

# Correct Docker Compose pattern
services:
  sn-1:
    cpuset: "7,15"               # Overflow cores only — shim stays here
    environment:
      MASTER_CPUSET: "0-1,8-9"  # DPDK cores — excluded from cpuset above

# ❌ Incorrect — DPDK cores appear in cpuset; shim can preempt poll threads
services:
  sn-1:
    cpuset: "0-1,7,8-9,15"
    environment:
      MASTER_CPUSET: "0-1,8-9"

For full implementation details, profiling evidence, and Kubernetes equivalents, see hd_sn/docs/user_guide.md — Step 3a.

Per-Role Minimum CPU Core Requirements

The table below is per-component — DPDK/active cores against overflow cores. The installer works in whole instances instead, and its floors (SCC 6, SN 9, Ingester 4) are the totals fermihdi-configure allocates and validate-cluster checks. The two measure different things; where you are sizing a host, use the installer's.

The table below documents the minimum logical CPU (thread) count required per service role and the recommended configuration for production deployments. These figures are derived from profiling data collected during FermiHDI HD benchmark runs.

Role	DPDK / Active Cores (`MASTER_CPUSET`)	Overflow Cores (`cpuset`)	Notes
Storage Node (SN)	4	2 (separate physical core)	Overflow core must be on a different physical core than any DPDK core to prevent SMT sibling interference
Storage Cluster Controller (SCC)	2	1 (SMT sibling acceptable on constrained hosts)	SCC does not run a continuous DPDK PMD poll loop; idle evictions from `swapper` on the DPDK core are benign. On hosts with no free physical core, `cpuset == MASTER_CPUSET` is acceptable.
Query Proxy (HD Proxy)	1–2	1 (shared with SN overflow acceptable)	Proxy is not DPDK-intensive; no strict overflow isolation required

Minimum host configuration for a single Storage Cluster (1 SCC + 3 SNs):

Scenario	Logical CPUs Required
Sandbox / development (as in `tests/sandbox_dms`)	16 (8 physical cores, SMT×2)
Production minimum (clean per-role overflow isolation)	20+ (10+ physical cores)
Production recommended (with per-SN isolated NUMA nodes)	32+ (2× NUMA, 16 cores each)

Note on SCC cpuset == MASTER_CPUSET: On a 16-logical-CPU host where all cores are allocated, the SCC has no spare core for a separate overflow. Profiling (RC12) shows the resulting evictions are exclusively swapper idle-thread handoffs — not active preemption of DPDK work. This is safe because the SCC reader thread is not a spin-polling PMD; it sleeps between batches. If the host has ≥ 18 logical CPUs, assign 1 dedicated overflow core to the SCC to eliminate this edge case entirely.

4. Automated Host Setup

The supported path is install/, which records what a host should run in cluster.yaml and applies it: hugepage pools per NUMA node, hugetlbfs mounts, vfio-pci bindings and the kernel command line. It reports before it changes anything, is idempotent, and never reboots.

install/fermihdi-discover                            # inventory this host
install/fermihdi-configure                           # decide what it runs
install/validate-cluster cluster.yaml                # check that decision
sudo install/fermihdi-prepare cluster.yaml --apply   # apply it

Core counts, hugepage sizing and the cpuset split described below are what validate-cluster enforces; install/ADMIN_GUIDE.md gives the whole model with the numbers.

scripts/host_setup.sh remains the by-hand tool, for working out what a machine can do or fixing one thing at a time:

# Show current state
sudo bash scripts/host_setup.sh status

# Configure hugepages + persistent /etc/fstab mount (safe, no reboot needed)
sudo bash scripts/host_setup.sh hugepages

# Bind an NVMe device to vfio-pci for SPDK passthrough
sudo bash scripts/host_setup.sh bind 0000:01:00.0

# Print the full recommended GRUB command line for this host
sudo bash scripts/host_setup.sh grub-advice

After making GRUB changes: sudo update-grub && sudo reboot

Topic	Document
FHDWP MTU, Jumbo Frame support, Docker/K8s CNI compatibility	docs/fhdwp_network_guide.md
SN configuration and storage provisioning	hd_sn/docs/configuration_guide.md
SN user guide and hardware requirements	hd_sn/docs/user_guide.md
End-to-end integration test guide	docs/end_to_end_test.md
Performance benchmark strategy	BENCHMARKING.md
Host setup script	scripts/host_setup.sh
Known technical debt and performance findings	tech-debt.md