FermiHDI HD SDK — Developer Guide

How to write an application that puts records into an HD instance and gets answers back, in any of the eight languages the SDK binds.

This is the guide to hd_sdk. It assumes you can build software and have an HD instance to talk to; it assumes nothing about FermiHDI. If you are standing the instance up rather than writing against it, that is install/ADMIN_GUIDE.md.

1. One engine, eight wrappers

There is one implementation. libfermihdi_client is a C++ engine that speaks the FermiHDI wire protocol, holds the mTLS identity, and hands records to your process without copying them. Every language binding is a thin shim onto that engine — they are not eight ports of the same idea, and they do not drift, because there is nothing to drift.

Language	Binding	Package
C++	the engine itself	`FermiHDISDK.hpp`, `libfermihdi_client.so`
Python	pybind11	`fermihdi_sdk`
Go	cgo	`fermihdi_sdk`
Rust	`cxx::bridge`	`fermihdi-sdk`
Node.js	N-API	`fermihdi-sdk`
R	Rcpp	`FermiHDISDK`
Julia	CxxWrap	`FermiHDI.jl`
Scala / Java	JNA	`fermihdi-sdk`
Mojo	C FFI	`FermiHDISDK`

What follows describes the engine's model once. The per-language sections then show the same programme in each, and say only what differs.

2. The model

Two clients, because there are two jobs. IngestClient puts records in; QueryClient asks questions. They register separately, hold separate identities, and are used from different parts of an application. Nothing stops one process holding both.

Two networks, and the distinction is not cosmetic. The control plane carries registration, service discovery and query dispatch. The data plane carries records — in on ingest, out on query — over FHDWP, FermiHDI's wire protocol. Both clients take an interface name for each:

FermiHDIQueryClient client("https://proxy:6986", "eth1", "eth0");
//                          proxy (control)      data   control

Pass an empty data-plane interface and it binds every interface, which is fine on a single-homed host and wrong on any other. The SDK reads no environment variables — every address, interface and URL is passed in by the caller. That is deliberate: an application's networking should not change because a shell did.

Query results do not come back from the query call. The Proxy answers with a manifest — which batches were found, which were missing, how each Storage Cluster fared — and the Storage Nodes stream the records themselves, directly to the address your client is listening on. If your client can reach the Proxy but the Storage Nodes cannot reach your client, every query succeeds and returns nothing.

Identity is issued, not configured. register_with_dms() generates a key, submits a CSR, and comes back with a certificate signed by the deployment's CA. From then on the client's connections are mutually authenticated. You need a bootstrap token to do it once; you do not need to manage certificates afterwards.

Records are Arrow. Ingest builds arrow::RecordBatch and sends it without serialising through an intermediate form; query returns arrow::Table, which DuckDB will read in place. That is what "zero-copy" means here — not that no memory is ever touched, but that your records are not translated between representations on the way through.

3. The two clients

IngestClient

register_with_dms(dms_url, node_name, override_key_pem = "")   -> token/identity
discover_scc_targets(proxy_url)                                -> bool
set_scc_targets([SccAddress])                                     explicit alternative
add_control_header(key, value)                                    routing hints for the Proxy
load_dataset_schema(schema_json)                                  what a record looks like

append_record_to_arrow({field: value})                         -> bool
flush_arrow_batch()                                            -> RecordBatch

stream_insert(batch_id, record | records | RecordBatch, broadcast = false)
send_eob(batch_id, broadcast = false)                             end of batch
send_bbn(batch_id, broadcast = false)                             batch boundary notice

get_health_status() / get_metrics()
stop()

The shape of an ingest loop: register, discover, load the schema, then append records until you have a batch worth sending, flush it to Arrow, stream_insert it, and send_eob to close it. broadcast sends to every SCC rather than the one the batch id routes to.

QueryClient

register_with_dms(dms_url, node_name, override_key_pem = "")   -> RegistrationResult
set_scc_targets([SccAddress])                                     direct mode, no Proxy
load_dataset_schema(schema_json)
add_control_header(key, value)

generate_batch_ids_time(start_time, end_time, interval = 0)    -> [BatchID]
generate_hd_query(HDQuery)                                     -> query JSON

execute_stream(sql_or_json, callback)                             per-batch, as they arrive
execute_dataset(sql_or_json)                                   -> arrow::Table
execute_to_sql(fermihdi_sql, ...)                              -> arrow::Table via DuckDB
set_local_aggregations(agg_config)                                fold as results land

get_found_batch_ids() / get_missing_batch_ids()
had_errors() / get_batch_errors()
stop()

Two ways to consume a result, and the choice matters at scale. execute_dataset gives you a table when everything has arrived. execute_stream calls you per batch as the Storage Nodes deliver, which is what you want when the answer is larger than the memory you would like to spend on it.

Always check get_missing_batch_ids(). A query that returns fewer batches than it asked for is a normal, reportable outcome — an SCC timed out, a node was rebalancing — and it is not an error. had_errors() and get_batch_errors() tell you which, and why.

4. The same programme, in each language

Register, query, read the answer as a table.

C++

#include <FermiHDISDK.hpp>

FermiHDIQueryClient client("https://proxy:6986", "eth1", "eth0");
client.register_with_dms("https://dms-core:6986", "analytics-1");
client.load_dataset_schema(schema_json);

auto table = client.execute_dataset(query_json);
if (client.had_errors()) { /* get_batch_errors() */ }

Link against libfermihdi_client. Headers are include/FermiHDISDK.hpp; the build wants the prebuilt library tree for your CPU variant (see §6).

Python — `fermihdi_sdk`

import fermihdi_sdk

client = fermihdi_sdk.QueryClient("https://proxy:6986", "eth1", "eth0")
client.register_with_dms("https://dms-core:6986", "analytics-1")
client.load_dataset_schema(schema_json)

table = client.execute_dataset(query_json)     # pyarrow.Table, no copy
df = table.to_pandas()

pybind11, so the Arrow table crosses as a pyarrow.Table sharing the same buffers. get_shared_ingest_client() hands back a process-wide ingest client when you would otherwise build one per worker.

Go — cgo

client := sdk.NewQueryClient("https://proxy:6986", "eth1", "eth0")
defer client.Stop()
client.RegisterWithDMS("https://dms-core:6986", "analytics-1")
table, err := client.ExecuteDataset(queryJSON)

cgo owns the engine's memory; the wrapper keeps it out of Go's heap so the garbage collector never moves a buffer the engine is writing into. Stop() matters here — it releases what the finalizer will not.

Rust — `cxx`

let mut client = fermihdi_sdk::QueryClient::new("https://proxy:6986", "eth1", "eth0")?;
client.register_with_dms("https://dms-core:6986", "analytics-1")?;
let table = client.execute_dataset(&query_json)?;

cxx::bridge gives a checked boundary rather than raw FFI: lifetimes are enforced at compile time, and the engine's errors arrive as Result.

Node.js — N-API

const { QueryClient } = require('fermihdi-sdk');

const client = new QueryClient('https://proxy:6986', 'eth1', 'eth0');
await client.registerWithDms('https://dms-core:6986', 'analytics-1');
const table = await client.executeDataset(queryJson);

V8 buffers map onto the engine's directly. Calls that wait on the network are async and do not occupy the event loop.

R — Rcpp

library(FermiHDISDK)

client <- QueryClient("https://proxy:6986", "eth1", "eth0")
register_with_dms(client, "https://dms-core:6986", "analytics-1")
df <- execute_dataset(client, query_json)   # via the Arrow C data interface

Julia — CxxWrap

using FermiHDI

client = QueryClient("https://proxy:6986", "eth1", "eth0")
register_with_dms(client, "https://dms-core:6986", "analytics-1")
table = execute_dataset(client, query_json)

Scala / Java — JNA

val client = new FermiHDISDK.QueryClient("https://proxy:6986", "eth1", "eth0")
client.registerWithDms("https://dms-core:6986", "analytics-1")
val table = client.executeDataset(queryJson)

JNA resolves libfermihdi_client.so at run time, so it must be on the library path of the JVM process.

Mojo — C FFI

from FermiHDISDK import QueryClient

var client = QueryClient("https://proxy:6986", "eth1", "eth0")
client.register_with_dms("https://dms-core:6986", "analytics-1")
var table = client.execute_dataset(query_json)

5. Beyond fetching rows

The engine carries two analytical libraries so that work can happen where the records already are, rather than after moving them somewhere else.

DuckDB. execute_to_sql() runs SQL over the returned Arrow table in process. Joins, aggregates and window functions without a round trip.
USearch. HNSW vector similarity over the same buffers, for nearest-neighbour work against retrieved records.
Local aggregation. set_local_aggregations() folds results as they land instead of accumulating them, which is what makes a query larger than memory finish.

6. Building

The wrappers all need the C++ engine, and the engine needs its prebuilt library tree for the CPU variant you are targeting — avx512, avx2 or arm64:

cd hd_sdk
./prebuild_libs.sh              # builds the prebuilt/<variant> tree
cmake -B build -DCMAKE_BUILD_TYPE=Release .
cmake --build build --parallel

release_build.sh does the same and then packages every wrapper, which is what CI runs. Per-language build notes are in each wrapper's own README — hd_sdk/python/README.md and so on — and each has a tests/ directory worth reading as worked examples.

The variant has to match the host you will run on: an AVX-512 build faults on its first vectorised instruction on a host without it.

7. When it does not work

Symptom	Usually
Registration fails at the handshake	Wrong DMS URL, or the bootstrap token is spent — they are single-use per node
Registration succeeds, queries return nothing	The Storage Nodes cannot reach your client on the data plane. Check the interface you passed and what your firewall does with FHDWP
Every query returns 503 from the Proxy	The Proxy has not compiled its policy matrix yet. It is a startup race; retry
Queries return 401	Your caller has no `X-Forwarded-User` — the Proxy authenticates nobody itself, and something in front of it must set the identity headers
Fewer batches than expected, no error	Normal and reportable. `get_missing_batch_ids()` says which; an SCC timed out or a node was rebalancing
SIGILL at first use	Wrong CPU variant of the prebuilt libraries
Records go in and cannot be found	The dataset schema loaded by the ingest client and the query client disagree

8. Where else to look

Where	What
`../hd_sdk/README.md`	the SDK's own overview
`../hd_sdk/docs/architecture.md`	how the engine is put together
`../hd_sdk/docs/config_guide.md`	every configuration value
`../hd_sdk/docs/interface_binding.md`	choosing the data and control interfaces
`fermihdi_proxy_api.yaml`	the Proxy API the SDK speaks, as OpenAPI
`fhdwp_network_guide.md`	the wire protocol and its MTU
`../install/ADMIN_GUIDE.md`	standing up the instance to write against