name: fermihdi-hd-troubleshoot description: Diagnose a misbehaving FermiHDI HD deployment — queries returning 0 rows or hanging, ingest that is slow/flat, SNs that won't start or store nothing, cluster-wide integrity failures, and rock/pebble startup problems. Customer-facing (uses docker/kubectl logs, host config, GRUB and mounts — no source checkout); symptom → root cause → fix.

Troubleshoot a FermiHDI HD deployment

Works with only a running deployment — no source checkout. Collect: docker logs <svc> (or kubectl logs), the container env (docker inspect), /proc/cmdline, mount | grep huge, and the VFIO bindings (ls -l /sys/bus/pci/devices/<addr>/driver). If you have the installer bundle from downloads.fermihdi.io, fermihdi-prepare --verify and lib/lib_distro.sh's fhd_verify_hugepages check host prep. Most "0 rows"/"hang" cases are host prep or a version/port mismatch, not the engine.

Query returns 0 rows or hangs in "WaitingQuery"

1. SN wrote nothing (device full). SN log: every configured device is full … bytes_written=0 opening_lba=0 closing_lba=0, Batch N: FAIL (hash_ok=0). Cause: the batch reserve (FERMIHDI_SN_BATCH_RESERVE_MB, default 4096 MB) is ≥ the SN's storage device — so it reads full and stores nothing while still "committing" (hashing); the later query finds 0 rows. Fix: give the SN a device larger than the reserve, or set FERMIHDI_SN_BATCH_RESERVE_MB to fit it.
2. Query-submission stream closed → SCC cancelled the SNs. SN log: [SN CancelQuery] … marked cancelled in the data plane; SCC: CancelQuery forwarded … (acknowledged=1); client: result listener waiting, framesTotal=0. Cause: the query client closed its query-dispatch stream right after submitting; the SCC treats a graceful close as "done" and cancels the SNs before results flow. Fix: the client MUST hold the query-dispatch stream open for the query's lifetime; upgrade an old client/ingester that closes early.
3. SCC on a non-default command port, ports not derived. The SN targets the default verify/query ports (command+10 / command+11 of 6986) while the SCC listens on a non-default base. Fix: ensure DMS discovery propagates the SCC command port so the SN derives verify/query from the learned port.
4. Empty sub-query or a non-address query target returns no rows AND no error — inspect the query the client built.

Ingest is slow or flat (low MB/s, doesn't scale with SN cores)

• Single-stream, latency/window-bound. One ordered ingest session; throughput ≈ in-flight ÷ RTT. Over a network it is far below loopback and flat across SN cores (the SNs aren't the bottleneck). Ingest is single-shard by design (chained-hash ordering + contiguous writes) — parallel senders are wrong. Levers: wider fan-out window, jumbo frames, bigger batches.
• Jumbo enabled on the sender only. An 8500 B sender against 1200 B SCC/SN fragments/stalls. Jumbo is instance-wide: set FERMIHDI_JUMBO on SCC and SN (and the sender). Confirm the SN transport logs jumbo=1.

An SN won't start, or attaches no NVMe

SN log: spdk_env_init failed, No devices attached, or … 1GB hugepages reserved, but no mounted hugetlbfs found for that size. Causes, in order:

1. No 1 GB hugetlbfs mount — SPDK uses the 1 GB pool; with only 2 MB mounted it falls back and only ONE SN fits. Mount /dev/hugepages1G (pagesize=1G) and pass it into the container.
2. No 1 GB pages reserved (hugepages-1048576kB/nr_hugepages = 0) — reserve at boot via GRUB.
3. VFIO reverted on reboot — drives are back on nvme; rebind and make it persistent.
4. RLIMIT_MEMLOCK not unlimited (--ulimit memlock=-1:-1), or wrong iova-mode (identity IOMMU groups need iova-mode=pa).

Cluster-wide integrity / hash mismatch

DMS: config sync rejected — hash_algorithm … immutable after node boot, or the query verifies with a different hash than ingest used. Seed a consistent hash config (e.g. XXH3_HYBRID/64) in the node and deployment records before the nodes boot; it can't change afterward.

Rocks hang or segfault at startup (pebble)

SCC/SN rock needs working-dir: /app in the pebble service (relative schema load → SIGSEGV otherwise). DMS-CA rock needs CA_LISTEN_ADDR=0.0.0.0:6985 or it never binds and registrations hang. Pebble passes the container env through, so runtime flags reach the binary.

SMT: query slower with SMT on at high core counts

Expected above ~the physical-core count: the query path becomes memory/interconnect-bandwidth-bound and sibling threads add contention. SMT helps query at low/moderate cores, flattens/regresses at high cores; ingest is unaffected. Not a bug.

Operational gotchas

• Address already in use starting the ingester/test client — a stale container holds the port; docker rm -f the leftovers.
• grubenv "environment block too small" — a long isolcpus overflows grubenv; write kernel args into the BLS entries (grubby) / /etc/tuned/bootcmdline.
• A vm.nr_hugepages sysctl.d file can silently cap the 2 MB pool below the GRUB reservation — reconcile them.
• VFIO drivers back on nvme after a reboot — the binding wasn't made persistent.

Escalate with

Image tag + build hash (docker inspect), /proc/cmdline, mount | grep huge, the SN and SCC container logs, and the deployment topology (see fermihdi-hd-install).