Of course. As an expert technical writer and Infrastructure Architect, I will synthesize the provided context into a comprehensive manual. Here is the "Administrator and Deployment Manual" for the FermiHDI platform, based exclusively on the information you provided.

FermiHDI Platform: Administrator and Deployment Manual

Installing an instance? install/ADMIN_GUIDE.md is the guide to that: requirements, sizing, the cluster file, all three install paths and every option they take. This manual is the platform reference behind it — architecture, networking and storage configuration, the PKI model, and operational tasks.

1. System Overview & Architecture

The FermiHDI HD System is an ultra-high-performance Data Management System designed for massive-scale analytics. It achieves extreme throughput by blending bare-metal acceleration technologies like DPDK (networking) and SPDK (storage) with a secure, Go-based control plane. This architecture bypasses traditional kernel bottlenecks, enabling a zero-copy data path where information flows from the network interface directly to storage or CPU cache layers.

1.1. Architectural Principles

The system is segmented into three distinct, isolated network planes:

1. Control Plane (Port 6986): An HTTPS mTLS network for all orchestration traffic. This includes node registration, configuration distribution, policy updates, and telemetry management. User data records never traverse this plane.
2. Data Plane (Port 6987): A high-speed, unidirectional network for binary data streaming using the proprietary FermiHDI HD Wire Protocol (FHDWP). Data flows from Storage Nodes directly back to the requesting client.
3. Internal DMS/CA Plane (Ports 6985, 6989): Secure channels used exclusively by the Data Management System (DMS) components to bootstrap nodes and provision cryptographic identities.

1.2. Core Components

The platform is composed of several specialized microservices:

• Data Management System (DMS): The central source of truth and orchestration.
  • DMS-Core: The Go-based engine that manages node topology, distributes schemas, and handles licensing.
  • DMS-CA: An integrated Certificate Authority that provisions short-lived mTLS certificates to all nodes, forming the cluster's root of trust.
  • DMS-WebUX: A Vue.js/Astro web interface (Port 3000) for system administration, configuration, and monitoring.
• HD Proxy: The ingress gateway for all data queries. It enforces Policy-Based Access Control (PBAC) using an embedded Open Policy Agent (OPA), rewrites queries based on security rules, and forwards authorized requests to the storage layer.
• Storage Cluster Controller (SCC): A C++ load balancer and orchestrator for a group of Storage Nodes. It receives queries from the Proxy and ingest data from the Message Bus Ingester, distributing the workload across its assigned nodes. It uses NORM (NACK-Oriented Reliable Multicast) over UDP port 6988 for internal data distribution.
• Storage Node (SN): The C++-based data persistence and retrieval engine. It uses DPDK and SPDK to interact directly with networking and NVMe hardware, bypassing the kernel. It can offload filtering to GPUs (via SYCL) and hashing to specialized hardware like NVIDIA Bluefield DPUs.
• Message Bus Ingester: A high-performance C++ service that connects to data sources like Kafka, RabbitMQ, or S3, performing zero-copy parsing and converting data into the binary FHDWP format for ingestion.
• HD SDK: Client-side libraries (Python, Go, Rust, etc.) that manage query execution. The SDK performs heavy computational work (SQL, Vector Search, Graph Math) locally on the data returned from the Storage Nodes, preventing computational bottlenecks within the storage cluster.

1.3. System Topology Diagram

graph TD
    %% External Clients
    SDK_App[HD SDK Client]
    Client_Admin["DMS Admin UX (Port 3000)"]

    %% DMS Control Plane
    subgraph Control_Plane ["Management Layer (hd_dms)"]
        DMS((Data Management System))
        OpAMP_Server[OpAMP WebSocket Supervisor]
        DMS_CA[Internal Certificate Authority]
        DMS --> OpAMP_Server
        DMS --> DMS_CA
    end

    %% Network Entry and Routing
    subgraph Edge ["Network Edge (hd_proxy)"]
        Proxy[HD Proxy]
        OPA{Open Policy Agent}
        Proxy <--> OPA
    end

    %% Data Ingestion
    subgraph Ingestion ["Message Bus (hd_message_bus_ingester)"]
        Ingester[Message Bus Ingester]
        Kafka[(Kafka)]
        RabbitMQ[(RabbitMQ)]
        S3[(S3 / Filesystem)]
        Kafka -- "Zero-Allocation Parse" --> Ingester
        RabbitMQ -- "Reads Data" --> Ingester
        S3 -- "Reads Data" --> Ingester
    end

    %% Storage Cluster Components
    subgraph Storage_Cluster ["Storage Cluster (SC)"]
        SCC[Storage Cluster Controller]
        SN1[(Storage Node 1)]
        SN2[(Storage Node 2)]
        
        SCC -- "Load Balances Queries" --> SN1
        SCC -- "Load Balances Queries" --> SN2
    end

    %% Hardware Accelerators
    subgraph Acceleration ["Bare Metal / Acceleration"]
        DPDK[DPDK Networking]
        SPDK[SPDK Storage / OCF]
        SYCL[GPU SYCL Bitmask Filter]
        
        SN1 -. "Direct PCIe" .-> SPDK
        SN1 -. "Kernel Bypass" .-> DPDK
        SN1 -. "Compute Offload" .-> SYCL
    end

    %% Operations / Observability
    subgraph Telemetry ["Local Observability Stack Sidecars"]
        OTel(Local OTel Collector)
        SentryRelay(Local Sentry Relay)
    end
    
    Central_Telemetry([FermiHDI Cloud Sentry / OTel Gateway])

    %% Data Flow Connections
    SDK_App -. "REST API / Custom TCP" .-> Proxy
    Client_Admin -. "HTTPS Config / APIs" .-> DMS
    
    Proxy == "Authorized / Modified Query" ==> SCC
    Ingester == "FHDWP Form 1 — TCP, 1200-byte MTU (Jumbo: 8500)" ==> SCC
    
    %% Control Flow Connections
    DMS_CA -- "mTLS Key Provisioning" --> Proxy
    DMS_CA -- "mTLS Key Provisioning" --> Ingester
    DMS_CA -- "mTLS Key Provisioning" --> SCC
    DMS_CA -- "mTLS Key Provisioning" --> SN1
    
    %% Observability Paths
    Proxy -. "Metrics/Logs" .-> OTel
    Ingester -. "Metrics/Logs" .-> OTel
    SCC -. "Metrics/Logs" .-> OTel
    SN1 -. "Crash Dumps" .-> SentryRelay
    OpAMP_Server -. "Dynamic Rules via W/S" .-> OTel
    
    OTel -. "mTLS Batches" .-> Central_Telemetry
    SentryRelay -. "Scrubbed Errors" .-> Central_Telemetry

2. Deploying an Instance

There are three supported ways to stand up an instance, and all three are driven from one file — install/cluster.yaml — which describes the whole deployment. install/ADMIN_GUIDE.md is the guide to all of them; what follows is the summary.

install/fermihdi-discover                        # what this host has
install/fermihdi-configure                       # what it should run
install/validate-cluster cluster.yaml            # is that coherent
sudo install/fermihdi-prepare cluster.yaml --apply   # make it so

cd install/ansible && ansible-playbook site.yml  # the DMS, then the nodes

What a node is actually given

Deploying by hand is a matter of getting one contract right, so it is worth stating plainly: a node holds no configuration of its own. What it is given is an identity, and everything else — record size, which SCC to talk to, which devices to use, the hash election — comes back from the DMS in the registration response.

docker run -d --name sc-1-scc --network host \
  -e FERMIHDI_NODE_NAME=sc-1-scc \
  -e FERMIHDI_DMS_URL=https://dms-core:6986 \
  -e FERMIHDI_BOOTSTRAP_TOKEN="$SCC_TOKEN" \
  -e FERMIHDI_CA_CERT_PEM="$(cat dms-ca-bundle.pem)" \
  -e MASTER_CPUSET=2-7 \
  -e FERMIHDI_CONTROL_PLANE_SUBNET=10.147.0.0/24 \
  -e FERMIHDI_DATA_PLANE_SUBNET=10.150.0.0/24 \
  -v /dev/hugepages:/dev/hugepages -v /dev/hugepages2M:/dev/hugepages2M \
  --ulimit memlock=-1:-1 --cpuset-cpus 0-1 \
  cr.fermihdi.io/hd/scc:2.0.2

Four things about that command generalise to every node:

• --cpuset-cpus is the overflow cores, not the poll cores. MASTER_CPUSET is what the process pins its reactors to, and those cores must stay out of the container's cgroup — the container runtime shim inherits it and will otherwise be scheduled onto them.
• Both hugepage mounts. /dev/hugepages is the 1 GB pool SPDK pins its EAL to; /dev/hugepages2M is the 2 MB pool DPDK's EAL needs a mount for.
• --ulimit memlock=-1:-1. The HD engine pins its arena; the default 64 KB stops it at startup. In Kubernetes this is the IPC_LOCK capability.
• The bootstrap token is per node, and the CA bundle is how the node knows it is talking to your DMS and not something else.

The Ansible layer builds exactly this command from cluster.yaml, which is the argument for using it rather than maintaining the above by hand.

Sandbox Environment

For development and testing, a pre-configured sandbox environment is available. This automates the deployment of the entire stack.

cd ./tests/sandbox_dms
./start_sandbox.sh

To shut down and clean up the sandbox environment:

cd ./tests/sandbox_dms
docker compose down -v --remove-orphans

3. Bare-Metal Host Preparation for Accelerated Storage Nodes

For maximum performance, Storage Nodes (SN) should be deployed on bare-metal hosts with specific kernel and system configurations to support SPDK and DPDK. These steps must be performed on the host machine before launching the SN container.

3.1. IOMMU Configuration

The IOMMU (Input-Output Memory Management Unit) must be enabled in the system firmware (BIOS/UEFI) and via kernel parameters to allow direct device access (VFIO).

After editing GRUB configuration, update and reboot: sudo update-grub && sudo reboot.

3.2. Hugepages Allocation

DPDK and SPDK require pre-allocated hugepages for efficient memory management. Both 2MB and 1GB pages are necessary.

Add the following to your GRUB command line: default_hugepagesz=2M hugepagesz=2M hugepages=1024 hugepagesz=1G hugepages=4

This allocates 1024 pages of 2MB (2 GiB total) for DPDK and 4 pages of 1GB (4 GiB total) for large DMA buffers.

3.3. CPU Isolation

To prevent OS scheduler noise from interfering with the data plane, isolate the CPU cores that will be assigned to the Storage Node.

Add the following to your GRUB command line (example reserves CPUs 2-15): isolcpus=2-15 rcu_nocbs=2-15 nohz_full=2-15

3.4. Host Setup

install/fermihdi-prepare applies all of the above from cluster.yaml: the hugepage pools per NUMA node, the hugetlbfs mounts, the vfio-pci bindings and the GRUB command line. It is idempotent, it reports what it would change before changing anything, and it never reboots — --verify is the pass to run after one. That is the supported path, and the one the Ansible layer drives.

install/validate-cluster cluster.yaml
sudo install/fermihdi-prepare cluster.yaml            # dry run
sudo install/fermihdi-prepare cluster.yaml --apply
install/fermihdi-prepare cluster.yaml --verify        # after the reboot

scripts/host_setup.sh remains for working on a machine by hand — deciding what it can do, or fixing one thing at a time:

# Check the current system status
sudo bash scripts/host_setup.sh status

# Configure hugepages and add to /etc/fstab
sudo bash scripts/host_setup.sh hugepages

# Bind an NVMe device to the vfio-pci driver for SPDK
sudo bash scripts/host_setup.sh bind 0000:01:00.0

# Get a recommended GRUB command line for the host
sudo bash scripts/host_setup.sh grub-advice

4. Networking & Storage Configurations

4.1. Networking

Port Assignments

FHDWP MTU Configuration

The FermiHDI HD Wire Protocol (FHDWP) on the Data Plane (port 6987) has strict MTU requirements to prevent IP fragmentation.

[!WARNING] The MTU setting is instance-wide. All nodes in a cluster must use the same MTU. Mixing modes will lead to packet loss and data corruption. Do not enable Jumbo Frame mode in cloud environments (AWS, GCP, Azure) as they do not provide reliable end-to-end support.

To enable Jumbo Frame mode, the entire data path must be verified first.

Pre-flight Checklist:

1. Verify host NIC MTU: ip link show <interface> (should show mtu 9000).
2. Verify path MTU between nodes: ping -M do -s 8472 <destination_ip>. This must succeed without fragmentation.
3. For Docker, verify network MTU: docker network inspect <network_name> | grep -i mtu.

Changing the MTU on a running instance requires a full cluster restart.

4.2. Storage Node Configuration

Storage Nodes are configured via a local config.json file.

Example config.json:

{
  "node_name": "us-east-sn-01",
  "dms_url": "https://dms-core:6986",
  "command_port": 6986,
  "production_port": 8000,
  "production_interface": "0000:04:00.0",
  "drive_strategy": "fill_sequential",
  "storage_devices": [
    { "name": "nvme_0", "address": "0000:01:00.0", "type": "nvme" }
  ],
  "use_ocf": true,
  "ocf_cache_devices": ["nvme_0"],
  "ocf_core_devices": ["hdd_bulk_1"],
  "filter_device_selector": "gpu",
  "hash_offload_platform": "nvidia_bluefield"
}

Key Parameters:

• dms_url: The address of the DMS for registration.
• production_interface: The VFIO PCIe address of the network card for DPDK.
• storage_devices: A list of storage devices, identified by their PCIe address.
• use_ocf: Set to true to enable the Open CAS Framework, a caching tier that places fast devices (ocf_cache_devices) in front of slower, larger ones (ocf_core_devices).
• filter_device_selector: Offloads query filtering to gpu, cpu, or host.
• hash_offload_platform: Specifies hardware for cryptographic hash offloading.

5. Security & TLS/PKI Requirements

5.1. mTLS and Certificate Authority

The FermiHDI platform operates on a zero-trust model, with all inter-node communication secured by mutual TLS (mTLS).

• DMS-CA: The DMS includes an integrated Certificate Authority which acts as the root of trust for the entire cluster.
• Node Registration Lifecycle:
  1. A new node starts and connects to the DMS Control Plane (port 6986) using standard TLS.
  2. The node sends an authenticated Certificate Signing Request (CSR) to the DMS.
  3. The DMS-CA signs the CSR and returns a short-lived X.509 certificate to the node.
  4. The node uses this certificate to authenticate itself for all subsequent communication within the cluster.
• Criticality: The DMS-CA is critical for cluster operation. If the DMS-CA is lost, the entire cluster must be rebooted to regenerate the trust chain.

5.2. Policy-Based Access Control (PBAC) with OPA

The HD Proxy is the enforcement point for data governance, using Open Policy Agent (OPA).

• Policy Management: Policies are written in Rego and managed through the Security > Governance tab in the DMS WebUX.
• Dynamic Updates: When a policy is updated in the DMS, it is pushed to all active Proxies via a POST /v1/proxy/config webhook, which triggers an atomic in-memory swap of the ruleset.
• Query Enforcement:
  1. A client sends a query with a JWT to an ingress server (e.g., Traefik).
  2. The ingress appends user identity headers (X-Forwarded-Role, X-Forwarded-Email).
  3. The HD Proxy receives the request and evaluates it against the active OPA policy.
  4. Instead of rejecting queries, the Proxy rewrites them:
     • Restricted Fields: If a user is not allowed to see a field (e.g., credit_score), the proxy overwrites that part of the query payload with null bytes (\x00), making it cryptographically unmatchable.
     • Enforced Fields: The proxy applies hard constraints (\xff) to enforce filters (e.g., tenant_id = 'abc') without dropping the connection.

6. Troubleshooting & Operational Tasks

6.1. Troubleshooting

• Node Quarantine: If a Storage Node begins emitting Bad Batch Notice (BBN) control messages, it can be quarantined via the DMS WebUX. The SCC will automatically re-route queries to healthy replicas.

• Data Plane Sync Issues: If an SN falls out of sync on the NORM multicast network (UDP 6988), rebooting the node will trigger a cold-boot synchronization process over the Control Plane to ensure data integrity before it is marked as Ready.

• AVX-512 Illegal Instruction Crashes: On systems using Windows Hyper-V (including Docker Desktop on Windows), a bug may cause the hypervisor to incorrectly report AVX-512 support, leading to SIGILL crashes. To resolve this, compile the C++ components with AVX-512 disabled.

  # Set an environment variable before building
  export DISABLE_AVX512=1
  
  # Or, pass a flag to CMake
  cmake -B build -DDISABLE_AVX512=ON
  

  [!IMPORTANT] Do not set this flag when deploying on bare-metal Linux, as it will prevent the use of valid AVX-512 optimizations.

• Viewing Logs: To view the logs for a specific component in a Docker Compose environment:

  docker compose -f ./tests/sandbox_dms/docker-compose.yml logs -f dms-core
  

6.2. Operational Tasks

• Observability: All nodes expose Prometheus metrics and health endpoints on port 8080. The system is designed to integrate with a central OpenTelemetry (OTel) Collector and Sentry Relay for logs, traces, and metrics.
• SDK Resource Management: The HD SDK uses asynchronous C++ reactors. To prevent resource leaks and freezes, especially in garbage-collected languages like Python, developers must clean up connections properly. Use context managers (with client:) or explicitly call client.stop() in a finally block.
• License Activation: An Enterprise License must be imported via the DMS WebUX to activate full functionality, including node registration and telemetry.