install/ansible

The same four steps install/README.md describes, run across every host at once instead of one at a time — and the DMS those hosts register with.

There is no inventory file. fermihdi-inventory reads cluster.yaml and answers Ansible's questions from it, because an inventory file would be a second copy of facts the cluster file already holds — and the two would stop agreeing on the first host anyone added.

cd install/ansible
export FERMIHDI_CLUSTER=../cluster.yaml     # optional; this is the default

ansible-playbook site.yml                   # prepare, the DMS, then the nodes

The playbooks

Playbook	Does	Changes hosts
`prepare.yml`	validates the cluster file, then runs `fermihdi-prepare --apply` on every host	yes
`verify.yml`	runs `fermihdi-prepare --verify` everywhere and reports	no
`dms.yml`	runs the DMS plane on the host whose block declares a `dms` role	yes
`deploy.yml`	runs the node containers each host's block calls for	yes
`site.yml`	`prepare.yml`, then `dms.yml`, then `deploy.yml`	yes

ansible-playbook prepare.yml --check              # dry run on every host
ansible-playbook prepare.yml -e fermihdi_reboot=true
ansible-playbook dms.yml                          # the DMS plane on its own
ansible-playbook deploy.yml --limit fermihdi_sn   # just the Storage Nodes
ansible-playbook verify.yml                       # after a reboot, or when a node misbehaves

The DMS goes up before the nodes, which is why site.yml orders them that way: every node registers with it, and one that starts first spends its early life retrying a registration that cannot succeed yet. Running deploy.yml on its own against a cluster whose DMS is not up yet is not harmful, only slow to settle.

--check on prepare.yml is a real dry run: fermihdi-prepare without --apply reports and changes nothing, so the check run tells you what every host would do. The tooling itself is copied even under --check — shipping a read-only script to /opt/fermihdi/install is not a change to the host, and without it there would be nothing to run and nothing to report.

Nothing reboots unless you pass fermihdi_reboot=true, and when you do, hosts reboot one at a time — a cluster that reboots all of its Storage Nodes at once is a cluster that has lost quorum.

What the inventory produces

./fermihdi-inventory --list | jq .        # everything
./fermihdi-inventory --host sc-host-1     # one host
ansible-inventory --graph                 # the groups

One group per role, so plays and --limit can target exactly the hosts running a thing:

@fermihdi:            every host in the file
@fermihdi_scc:        hosts running one or more SCCs
@fermihdi_sn:         ... Storage Nodes
@fermihdi_proxy:      ... and so on

Each host carries fermihdi_instances — every instance on it with the role type, dpdk flag, cpusets and devices flattened onto it — plus fermihdi_host_prep and the raw fermihdi_roles. Cluster-wide settings arrive as one fermihdi_cluster variable, and group_vars/all.yml derives the individual settings from it. That is the file to read and edit: every default and fallback the plays use is in it, in one place, rather than spread through a Python script and half a dozen roles.

Secrets

cluster.yaml holds none. Everything is read from the environment of the machine running ansible-playbook, which is what the file's *_env fields promise:

export FERMIHDI_REGISTRY_PASSWORD=...
export FERMIHDI_CA_CERT_PEM="$(cat dms-ca-bundle.pem)"

# Bootstrap tokens are per node: the instance name, upper-cased, with anything
# that is not a letter or digit turned into an underscore.
export FERMIHDI_BOOTSTRAP_TOKEN_SC_1_SN_1=...
export FERMIHDI_BOOTSTRAP_TOKEN_SC_1_SCC=...

# ... or one shared token, if that is how the cluster issues them. The variable
# cluster.yaml names in dms.bootstrap_token_env is the fallback for every node
# with no per-node token set.
export FERMIHDI_BOOTSTRAP_TOKEN=...

Deploying the DMS needs the rest of them. dms.yml refuses to start with any of the required ones missing and names every one it could not find, rather than deploying a stack that comes up and then fails every handshake made to it.

Three of them can come from a file instead of the environment — the license, and the DMS-CA's certificate and key. fermihdi-configure asks for those on the host that runs the DMS and records the paths in cluster.yaml's dms: block, where a named file wins over the environment; the run prints which source each one came from. Paths only: the files stay where they are, are read on the machine running the playbook, and relative ones resolve against the cluster file.

dms:
  license_file: certs/license.json      # -> DMS_LICENSE_PAYLOAD
  ica_cert_file: certs/dms-ca.crt       # -> DMS_ICA_CERT_PEM / DMS_TLS_CERT_PEM
  ica_key_file: certs/dms-ca.key        # -> DMS_ICA_KEY_PEM / DMS_TLS_KEY_PEM
  schema_file: certs/dataset_schema.json

Variable	Is
`FERMIHDI_CA_CERT_PEM`	the trust bundle — root, intermediate and the DMS-CA's own certificate concatenated. The same one the nodes are given, and what every DMS service verifies against
`FERMIHDI_DMS_CA_CERT_PEM` / `_KEY_PEM`	the DMS-CA's ICA certificate and key. It presents this on the wire and signs every CSR in the deployment with it
`FERMIHDI_DMS_CORE_CERT_PEM` / `_KEY_PEM`	DMS-Core's TLS identity. The certificate is the leaf and its chain
`FERMIHDI_DMS_ICA_CHAIN_PEM`	the chain between the root and the DMS-CA, so a node verifying a certificate this deployment issued can build a path
`FERMIHDI_DMS_WEBUX_CERT_PEM` / `_KEY_PEM`	the WebUX's own server certificate
`FERMIHDI_DMS_WEBUX_FE_CERT_PEM` / `_KEY_PEM`	the client certificate the WebUX's back end presents to DMS-Core. A different identity, signed for client auth, and not interchangeable with the one above
`FERMIHDI_DMS_ENCRYPTION_KEY`	what DMS-Core encrypts its stored state with. It exits at startup without one
`FERMIHDI_DMS_IDENTITY_KEY`	what the license payload is encrypted to. Has to be the one the license was issued against
`FERMIHDI_DMS_LICENSE_PAYLOAD`	the encrypted license. Optional: DMS-Core starts without it and admits no node until one is applied, through the WebUX or by setting this and running again
`FERMIHDI_DMS_FE_TOKEN`	shared between DMS-Core and the WebUX — what the WebUX's back end presents on the internal management API
`FERMIHDI_BOOTSTRAP_TOKEN_DMS_OPA_ENGINE`	the OPA engine registers with DMS-Core the way a node does, so its token follows the same per-node convention, with `FERMIHDI_BOOTSTRAP_TOKEN` as the fallback

tests/sandbox_dms/generate_certs_and_license.sh produces a complete set of these for a sandbox, and is the clearest description of what each one is and how they are signed.

Use ansible-vault or a secrets manager to get them into the environment. None of them are ever written to a host: the DMS services read PEM content straight from their environment and write it inside their own containers.

What `deploy.yml` works out for each container

From `cluster.yaml`	Becomes
`instances[].cpuset`	`MASTER_CPUSET` — the poll cores the process pins itself to
`instances[].container_cpuset`	the container's `cpuset_cpus` — overflow cores only
`roles[].dpdk`	absent means `FERMIHDI_FORCE_POSIX=1`
`instances[].storage` / `.nic` with `driver: vfio`	`/dev/vfio/<group>` device passthrough
`instances[].storage` with a `/dev/...` path	that block device, passed through
`images` / `registry.url`	the image reference
`networks`	`FERMIHDI_*_SUBNET`

The cpuset split is the invariant worth understanding before changing any of this: the container's cgroup must contain only the overflow cores. The runtime shim runs in the host scheduler but inherits that cgroup, and if a DPDK core is in it the shim will be scheduled there — profiling measured over 300,000 shim evictions in a 90-second window that way. isolcpus does not prevent it, because an explicit cgroup cpuset overrides isolcpus. So a DPDK instance with no container_cpuset is refused rather than deployed with the default of "every CPU on the box".

IOMMU group numbers are read from the host at deploy time rather than taken from cluster.yaml. The file records what discovery saw, which is the right thing for the validator to reason about, but group numbering can move under a kernel upgrade — and a container handed the wrong /dev/vfio node fails at SPDK attach looking like a driver fault. When the two disagree the run says so.

Both hugepage mounts go into every engine container: /dev/hugepages is the 1 GB pool SPDK pins its EAL hugedir to, and /dev/hugepages2M is the 2 MB pool DPDK's EAL needs a mount for. (docs/examples/ansible_example mounts /dev/hugepages as the 2 MB pool and a /dev/hugepages1G beside it. Those deployments set FERMIHDI_FORCE_POSIX=1 and use file-backed storage, so nothing there ever DMAs and the distinction never bites. It bites here.)

The DMS

dms.yml deploys it on the host whose cluster.yaml block declares a dms role — one host, because the file names one dms.url and every node is pointed at it.

Container	Is	Reached
`dms-ca`	signs every identity in the deployment	by DMS-Core, and by nothing else
`dms-core`	registration, node configuration, the control plane	by every node, on 6986
`dms-opa-engine`	the policy engine. Registers with DMS-Core the way a node does	internally
`dms-opa-adapter`	polls an OPA control plane that cannot reach in, and writes what it fetches into the directory the engine reads. Off by default	outbound only
`dms-webux`	the web UI and its back end	through `dms-traefik`
`dms-traefik`	routes to the WebUX and applies the forward-auth middleware	on 8081, loopback by default
`otel-collector`	where every node in the cluster sends traces and metrics	on 4317 and 4318
`sentry-relay`	scrubs events before they leave the deployment. Only when a DSN is set	internally

This is the one place the layer does not use host networking. DMS-CA and DMS-Core both listen on 6985 for their internal API and both serve observability on 8080; on a host network that is a collision, and on their own bridges it is the isolation the design asks for. So the services find each other by container name across three networks — fermihdi-dms-ca holds the CA and DMS-Core alone, fermihdi-dms-internal holds everything the UI path touches, fermihdi-observability holds everything that reports — and only what the outside has to reach is published to the host.

Three things are worth getting right before the first run:

DMS-Core's certificate has to carry the address the nodes use. dms.url is handed to every node verbatim, and the nodes verify what they connect to. A certificate whose SAN says dms-core and nothing else works inside the deployment's own network and fails every registration from outside it. dms.yml warns when dms.url does not name this host; it cannot see inside the certificate.

The WebUX has no authentication of its own. It trusts the X-Forwarded-* headers a forward-auth service in front of it sets. Until fermihdi_dms_forward_auth_url names one, dms-traefik routes to it with no middleware, which is why 8081 is published on the loopback address and not on every interface — reach it through an SSH tunnel, or put your SSO's forward-auth on the fermihdi-dms-internal network, name it, and then widen the bind address.

The OPA adapter registers separately. It is a node in the DMS's eyes, with its own name and its own bootstrap token (FERMIHDI_BOOTSTRAP_TOKEN_DMS_OPA_ADAPTER, falling back to the cluster-wide one), and it is the one service here that reads its trust bundle only as a file — there is no PEM environment variable for it, so the run writes the bundle to the host and mounts it. Give it fermihdi_dms_styra_api_url and FERMIHDI_STYRA_TOKEN or it registers and then idles, having nothing to poll.

Nothing is regenerated. Every key and certificate comes from the environment or from the files cluster.yaml names, on each run — so a certificate the DMS-CA renewed at run time is replaced by whatever those hold the next time this runs. Keep them the source of truth, or expect the run to undo the renewal.

The license and the certificates are not the same decision

A license supplied as DMS_LICENSE_PAYLOAD — which is what both license_file and the environment variable become — is decrypted, verified and stored, and that is all. Applying the same license through the WebUX does more: DMS-Core pulls the DMS-CA's ICA certificate and its own identity certificate out of it (handleLicense, cmd/dms-core/main.go), which is what the Day-0 bootstrap uses — the CA generates a key, posts a CSR, and waits for the ICA the license carries.

This layer deploys the pre-provisioned arrangement instead: the certificates are in place before the first container starts, which is why they are asked for separately from the license. If all you have is a license, deploy without the ICA and apply the license through the WebUX afterwards; fermihdi-configure says so when it is given one and not the other.

A license the DMS cannot parse is worse than no license: DMS-Core exits at startup rather than coming up unlicensed. validate-cluster reads the file and says so before it gets that far.

The dataset schema

schema_file is copied to the DMS host and mounted read-only over DMS-Core's active schema at /app/schemas/dataset_schema.json — the file /v1/dms/schema/active serves to the OPA engine, and which the image does not otherwise ship under that name.

It is not the per-deployment schema. The one Proxy and Ingester nodes are handed at registration comes from the deployment record, which the WebUX writes; this installs the deployment-wide one the policy engine reads.

What this does not deploy

The public front door: the FE Traefik that terminates TLS for the Proxy and the WebUX, the forward-auth service behind it, and the data explorer. docs/examples/ansible_example is the reference for those. It also deploys the DMS, from an older arrangement than this one — where it and this layer disagree, tests/sandbox_dms/compose.yml is what the DMS is actually run as, and is what dms.yml is built from.

What `cluster.yaml` does not carry

fermihdi-configure writes identity, placement and host preparation. The rest of what the containers need is not in the schema, and comes from group_vars/all.yml (or -e) instead:

Setting	Why it is not in the file
`fermihdi_kafka_brokers` / `_topics`	the ingester's message bus. No schema field; the run warns rather than starting an ingester with nothing to read
`fermihdi_graphql_proxies_path`	scaffolded on first deploy and never overwritten, so an operator's proxy list survives the next run
`fermihdi_extra_env`	per-role core splits (`TRANSPORT_CORES`, `STORAGE_CORES`, `HASH_CORES`, `KV_CORES`, `FILTER_CORES`) and anything else. Merged last, so it wins
`fermihdi_dms_forward_auth_url`	what authenticates the WebUX. Nothing in the file says who your SSO is
`fermihdi_sentry_dsn`	the upstream a Sentry relay would forward to. Set it and a relay is deployed; leave it and none is
`fermihdi_dms_enable_analysis`	whether opt-in telemetry may leave the deployment. Off unless said otherwise, and a decision rather than a fact about the cluster

fermihdi_otel_url and fermihdi_otel_exporter_endpoint are not in the file either, but neither has to be set: both are derived from the host running the DMS, because that is where the collector goes. Set them by hand for a collector that lives somewhere else. They are two different things and are easy to confuse — the first is a routing probe the C++ nodes use to work out which of their own interfaces is on the observability plane, and configures no exporter; the second is where the SDK actually sends traces and metrics.

networks: and images: are in the schema, but fermihdi-configure does not write them yet. Without networks: the plane CIDRs are left unset and each node falls back to its own interface selection — which is fine on a single-homed host and wrong on any other, so deploy.yml says so once per run. Without images: the reference is built from registry.url, the namespace in group_vars/all.yml, and the published image name.

Requirements

Ansible ≥ 2.14 on the machine you run this from, with community.docker:
```
ansible-galaxy collection install community.docker
```
python3 with PyYAML on the control machine (the inventory) and on every target (fermihdi-prepare). prepare.yml installs it on the targets.
SSH with sudo to every host. Connection details come from each host's address, ssh_user and ssh_key in cluster.yaml.
Docker on the targets. deploy.yml and dms.yml install docker.io and python3-docker; override fermihdi_docker_packages for a host that gets Docker from docker-ce.