install

Everything needed to turn a set of bare machines into a FermiHDI cluster.

New to HD, or installing one for the first time? ADMIN_GUIDE.md is the long form: what the system is, what it is made of, what it needs from your hardware and network, and every option these tools take. This file is the map of the directory.

One file — cluster.yaml — describes the whole deployment, and every tool here either writes it, checks it or applies it. That is the point of the split: the questions get asked once, the answers get reviewed as a file, and applying them is a mechanical step that can be repeated, run from Ansible, and diffed when something looks wrong six months later.

The pipeline

fermihdi-discover  →  fermihdi-configure  →  validate-cluster  →  fermihdi-prepare
  what the host          what it should          is that              make it so
  actually has           run, and where          coherent?

Tool	Runs on	Reads	Writes	Touches the host
`fermihdi-discover`	each host	`/sys`, `/proc`	facts, YAML or TSV	no
`fermihdi-configure`	each host	facts	a `cluster.yaml` host block	no
`validate-cluster`	anywhere	`cluster.yaml`	nothing	no
`fermihdi-prepare`	each host	`cluster.yaml`	the host itself	only with `--apply`
`fermihdi-k8s`	anywhere	`cluster.yaml`	Helm values, or manifests	no

And two layers that deploy what the four steps prepare for:

Layer	Deploys
`ansible/`	the DMS and every node container, across every host in the file
`helm/fermihdi`	the same deployment on Kubernetes, from values `fermihdi-k8s` renders

Only the last one changes anything, and only when told to twice — a bare run is a dry run.

ansible/ runs all four across every host at once, and deploys the DMS the cluster registers with. It builds its inventory from cluster.yaml rather than from a second file that would have to agree with it, so there is nothing to keep in step:

cd install/ansible
ansible-playbook site.yml        # prepare, the DMS, then the containers

One host, start to finish

# 1. Look at the machine. Read-only, no root needed.
install/fermihdi-discover -o facts.yaml

# 2. Decide what it runs. Interactive, and still changes nothing.
install/fermihdi-configure -o cluster.yaml

# 3. Check the answer. Step 2 does this for you at the end.
install/validate-cluster cluster.yaml

# 4. Apply it. Dry run first: this is the step that is hard to undo.
install/fermihdi-prepare cluster.yaml               # reports, changes nothing
sudo install/fermihdi-prepare cluster.yaml --apply

# 5. Reboot if step 4 said to, then confirm the kernel came up as asked.
install/fermihdi-prepare cluster.yaml --verify

Each tool takes explicit paths above on purpose. fermihdi-configure defaults to ./cluster.yaml in the working directory, while validate-cluster and fermihdi-prepare default to install/cluster.yaml beside themselves — name the file and the difference stops mattering.

For more than one host, use ansible/ rather than running the above by hand on each machine.

Kubernetes

fermihdi-k8s renders Helm values for helm/fermihdi from the same file, so a Kubernetes install is described by the same cluster.yaml as a bare-metal one:

install/fermihdi-k8s cluster.yaml -o values.yaml
helm install fermihdi install/helm/fermihdi -f values.yaml

Roles, counts, images, plane CIDRs and sizing carry across. Devices, DPDK and the cpuset split do not — a Pod has no IOMMU group — and the generator says so per cluster file rather than dropping them quietly. --manifests renders the chart instead of the values, for a cluster where helm is not what applies them.

More than one host

fermihdi-configure does one host per run, because the questions it asks can only be answered by looking at that machine's own CPU topology and devices. Build the cluster file by appending:

# on each host in turn, against a file they share
install/fermihdi-configure --append cluster.yaml

--facts lets it work from a saved fermihdi-discover --format=tsv file rather than probing, which is what makes the whole interview runnable from a control node instead of over SSH one machine at a time.

`cluster.yaml`

cluster.example.yaml — a worked four-host example: a box sharing an SCC with two SNs, an SN on kernel-path storage, an edge host, and a plain proxy host.
schema/cluster.schema.json — the schema, and the place where each field's meaning is written down.

Two things about it are worth knowing before editing one by hand:

PCI addresses must be quoted. YAML 1.1 reads an unquoted 0000:31:00.0 as a sexagesimal number, and only when every segment is below 60 — so it silently works for bus 61 and breaks for bus 31. validate-cluster catches the unquoted form before the schema sees it, because the schema's own complaint ("1860.0 is not of type string") tells nobody what to do about it.

Secrets are never stored in it. The registry password and the DMS bootstrap token are named, not held: password_env and bootstrap_token_env say which environment variable to read at run time.

export FERMIHDI_REGISTRY_PASSWORD=...
export FERMIHDI_BOOTSTRAP_TOKEN=...

The same rule covers what the DMS is installed with. On a host that runs the DMS, fermihdi-configure asks whether you have a license, the DMS-CA's ICA certificate and key, and a dataset schema — and records the paths, under dms:. The files stay where they are and are read by whatever runs the deploy; relative paths resolve against the cluster file. Each is optional and each has another way in, so answering nothing is a decision rather than an omission: the license and the ICA can arrive through the environment instead, or be applied through the WebUX afterwards. See ansible/README.md for what each becomes.

What `fermihdi-prepare` changes

It applies one host's host_prep block and nothing else. Four areas:

Area	What it does	Needs a reboot
hugepages	reserves the 2 MB pool per NUMA node through sysfs	no
mounts	hugetlbfs at `/dev/hugepages2M` (2 MB) and `/dev/hugepages` (1 GB)	no
vfio	binds the listed PCI devices to `vfio-pci`	only if the IOMMU is off
grub	IOMMU, `isolcpus`/`rcu_nocbs`/`nohz_full`, and the 1 GB pool	yes

The two mount points are not interchangeable. StorageEngine.cpp pins SPDK's EAL hugedir to /dev/hugepages whenever it finds 1 GB pages, so that EAL never eats the 2 MB networking pool — which means /dev/hugepages has to be the 1 GB mount. That is also why default_hugepagesz=1G goes on the kernel command line whenever the 1 GB pool is in use: systemd mounts /dev/hugepages at whatever the default size is.

It never reboots. 1 GB pages, IOMMU and isolcpus all need one, and when a machine goes down is the operator's call. It says which are outstanding and exits 2; --verify is the pass to run afterwards.

To survive that reboot it installs, and regenerates on every --apply:

Path	Why
`/usr/local/sbin/fermihdi-hugepages` + its `.service`	2 MB pages are not reserved in GRUB, so something has to claim them on the way up. `sysctl` cannot: `vm.nr_hugepages` means the default page size, which is 1G here.
`/usr/local/sbin/fermihdi-vfio-bind` + its `.service`	`vfio-pci` binding lives in sysfs and does not survive a reboot.
`/etc/modules-load.d/fermihdi-vfio.conf`	loads `vfio-pci` before the bind unit runs.
`/etc/fstab` entry for `/dev/hugepages2M`	the 2 MB mount, which nothing else creates.

Undoing it: /etc/default/grub is backed up with a timestamp before every edit and the path is printed, and scripts/host_setup.sh unbind <PCI> returns a device to the kernel driver.

Relationship to `scripts/`

scripts/ holds the interactive, per-machine tools — the ones you reach for when you are working out what a machine can do, or fixing one by hand:

Script	Does
`host_setup.sh`	Bare-metal host preparation, one command at a time: `status`, `hugepages`, `bind <PCI>`, `unbind <PCI>`, `grub-advice`
`setup_scc_host.sh`	SCC-specific host preparation
`setup_sn_host.sh`	SN-specific host preparation, including NVMe/VFIO binding
`recommend_topology.sh`	Inspects `lscpu` and emits CPU pinning configuration
`prebuild_libs.sh`	Builds the `libfermihdi_*` libraries
`sync_*_submodules.sh`	Submodule maintenance

The split is not "which one touches the host" — fermihdi-prepare binds drivers and edits GRUB, and it is meant to. It is where the decision was made. scripts/host_setup.sh bind 0000:61:00.0 is a decision made at a prompt, once, by whoever is at the keyboard. fermihdi-prepare makes no decisions at all: it applies ones already written down, reviewed and validated, the same way on every host, and reports what it would do before doing it. Anything here that starts asking the operator what to do belongs in scripts/ instead — or in fermihdi-configure, which is where the asking is supposed to happen.

Requirements

python3 with PyYAML, for validate-cluster, fermihdi-prepare and fermihdi-k8s. jsonschema is optional: without it the schema check is skipped and the semantic checks still run.
bash, util-linux (lsblk, lscpu, findmnt) and iproute2 (ip) for fermihdi-discover and fermihdi-configure. Both are pure shell so they run on a host with nothing installed on it yet; fermihdi-discover --check-deps reports what is missing.
root, for fermihdi-prepare --apply and nothing else.
On the machine you install from, and only for the layer you use: Ansible ≥ 2.14 with community.docker, or helm ≥ 3.

Getting it onto a host

The installer is published as an archive, with a script that fetches it, verifies it against its checksum, unpacks it and runs the read-only host inventory:

curl -fsSL https://downloads.fermihdi.io/install.sh | sh

The hd/ folder the objects live in is applied by the route, so it does not appear in the URL. It changes nothing and needs no root — the only step that changes a host is fermihdi-prepare --apply, which it does not run. Pin a version with https://downloads.fermihdi.io/2.0.2/install.sh, or fetch fermihdi-hd-install-2.0.2.tar.gz and its .sha256 directly.

scripts/package_install.sh builds that archive and publishes it.

The manual is not inside it. It is published beside it, under the version, in the three forms it is read in — and so can be corrected without reissuing the installer:


`https://downloads.fermihdi.io/2.0.2/web/`	the site
`https://downloads.fermihdi.io/2.0.2/pdf/hd-administrator-manual.pdf`	one printable file
`https://downloads.fermihdi.io/2.0.2/md/`	the Markdown both are built from

scripts/publish_docs.sh builds and publishes all three. ADMIN_GUIDE.md still ships in the archive: it is what the installer needs, and a host that cannot reach the internet still has it.

What is here

ADMIN_GUIDE.md        the guide: architecture, requirements, every option
README.md             this file
cluster.example.yaml  a worked cluster definition
schema/               the JSON Schema validate-cluster checks against
fermihdi-discover     inventory a host
fermihdi-configure    interview, and write its block
validate-cluster      check a definition against the schema and against reality
fermihdi-prepare      apply one host's host_prep
fermihdi-k8s          render Helm values from a definition
ansible/              deploy across every host in the definition
helm/fermihdi/        the Kubernetes chart

Not written yet

lib/ — shared shell helpers, if the scripts here ever grow enough to need them.
A front door. Neither layer deploys the reverse proxy and forward-auth service that sit in front of the Proxy and the WebUX and set the X-Forwarded-* headers both rely on. docs/examples/ansible_example deploys one; until you have it, both are published on the loopback address.

docs/examples/ — working Ansible, Chef, Puppet, Kubernetes and Nomad deployments. These are DMS-managed: the environment carries identity only and the DMS supplies record_size, scc_ip, storage_devices and the hash election while each node waits at its boot gate. Anything written here should assume the same unless it is explicitly a pre-provisioned installer.
The per-component configuration guides, which mark every setting that is read but currently has no effect: SCC · SN · Proxy · DMS
cpu_requirements_sn_scc.md and IMAGE_BUILD.md.