name: fermihdi-hd-sandbox description: Install and run the self-contained FermiHDI HD sandbox on a single host — a full end-to-end HD instance (DMS, SCC, storage nodes, filter node, proxy, the SQL/GraphQL/MCP query services and synthetic flow generation) plus an optional benchmark suite, from the published download at downloads.fermihdi.io with every image from cr.fermihdi.io. The agent probes the host, asks the user every deployment question up front (version, image variant, storage node count, which drives to assign, hugepages and boot-loader changes, NICs, filter node, AI services), then asks what to test (browser tour, MCP/GraphQL/SQL query smoke test, benchmark), and runs everything unattended through command-line switches. Customer-facing; one host, no source checkout, no external PKI.

Run the FermiHDI HD sandbox

The sandbox is a complete HD deployment on one host for evaluation, demos and benchmarking. It ships its own CA and a licence minted when the archive was packaged, so it needs no external PKI and no source checkout. For a production, multi-host deployment use fermihdi-hd-install.

• Archive: https://downloads.fermihdi.io/sandbox-latest.tar.gz (pinned: https://downloads.fermihdi.io/<version>/fermihdi-hd-sandbox-<version>.tar.gz, each with a .sha256 next to it).
• Images: cr.fermihdi.io/hd/<service>:<tag>. For 2.0.5: 2.0.5 is the AVX-512 Rock, 2.0.5-avx2 the AVX2 Rock, and -ub2604 the Ubuntu 26.04 builds. The query helpers also pull alpine and python:3-alpine from Docker Hub.
• Licence: valid for 30 days from packaging. An archive older than that will not start; download it again.

How to run this skill

Work through the steps in order. Every script takes a command-line switch for every question it could ask, so run nothing that waits on a prompt, and never pipe answers into one.

1. Probe the host. Read-only.
2. Ask the deployment questions, all in one batch.
3. Show the plan: the exact commands, which ones need sudo, and what they change on the host. Wait for a yes.
4. Prepare the host and launch.
5. Ask the test questions, all in one batch.
6. Run the tests and report. Tear down as the user chose.

Hard rules:

• Never choose a drive for the user. Show the probe results and let them pick. An assigned drive's contents are overwritten.
• Never run sudo, bind a device to vfio-pci, change the boot loader or reboot unless the user said yes to that specific action. The scripts never reboot on their own; tell the user when a reboot is needed and wait for them.
• If the host runs other workloads, confirm before anything that stops containers. Launching the sandbox removes any earlier sandbox instance.

Use your question tool to ask when you have one; otherwise ask in one message and wait.

Step 1: probe (read-only)

Download and unpack first, because the probe uses the archive's own tools. Nothing here changes the host.

curl -fsSL https://downloads.fermihdi.io/sandbox-latest.tar.gz -o fermihdi-hd-sandbox.tar.gz
curl -fsSL https://downloads.fermihdi.io/fermihdi-hd-sandbox-2.0.5.tar.gz.sha256   # compare the hash
sha256sum fermihdi-hd-sandbox.tar.gz
tar -xzf fermihdi-hd-sandbox.tar.gz && cd fermihdi_sandbox

Then collect:

--list-storage --json reports every NVMe and SATA device with its type (nvme_vfio already bound to vfio-pci, nvme_kernel, sata), PCI address, block device, NUMA node, IOMMU group, size, model, in_use reasons (mounted, swap, held by LVM/RAID/device-mapper) and the value to pass to --sn-storage. Devices that are in use cannot be assigned.

Step 2: ask about the deployment

Ask all of these in one batch. Offer what the probe found as the default, and say why.

If any sandbox port is taken, or a sandbox is already running, say so here and ask what to do.

Step 3: plan, prepare, launch

Show the commands built from the answers, mark the sudo ones, and wait for a yes.

Prepare the host (one command; it also writes sandbox_hw_config.json and the compose override, so the launch asks nothing):

# tmpfs-only lab
sudo bash sandbox_host_setup.sh --num-sns 1 --storage tmpfs --hugepages-1g 0 \
     --bind-nvme none --bind-nic none --grub skip --persist yes --yes

# NVMe through SPDK for two nodes (values from --list-storage)
sudo bash sandbox_host_setup.sh --num-sns 2 --bind-nvme 0000:41:00.0,0000:42:00.0 \
     --sn-storage sn-1=0000:41:00.0 --sn-storage sn-2=0000:42:00.0 \
     --bind-nic none --grub apply --persist yes --yes

Add --numa-mode/--numa-node, --nic, --bind-nic and --dedicated-interfaces from the answers. If it prints "GRUB updated — please reboot" (exit status 0 after the boot-loader change), stop and ask the user to reboot. After the reboot, run the same command again with --grub print: it is idempotent, re-reserves the pools and re-checks the mounts.

What host setup does with hugepages:

• Sizes: 4 × 1 GB plus 512 × 2 MB per instance (every storage node and the SCC), split evenly across NUMA nodes. Override with --hugepages-1g and --hugepages-2m.
• Now: reserves both pools per NUMA node, and mounts /dev/hugepages and /dev/hugepages/nodeN at 2 MB and /dev/hugepages1G at 1 GB.
• At boot (--persist yes): fermihdi-sandbox-hugepages.service re-reserves the pages and re-creates the mounts before Docker starts. A managed block in /etc/fstab (nofail) mounts the same three paths. The 1 GB pool is reserved by the kernel command line (default_hugepagesz=2M hugepagesz=1G hugepages=N), which --grub apply writes through grubby on RHEL-family systems and through /etc/default/grub elsewhere. When a file in /etc/default/grub.d/ overrides it (Ubuntu cloud images do), it also writes 99-fermihdi-sandbox.cfg there, and it checks the regenerated grub.cfg for the arguments. default_hugepagesz stays 2M so /dev/hugepages remains the 2 MB pool.
• Checks: sudo bash sandbox_host_setup.sh --help; the summary at the end lists both pools, both mounts and what survives a reboot.

Verify before launching:

grep -E 'HugePages_(Total|Free)|Hugepagesize' /proc/meminfo
cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
grep hugetlbfs /proc/mounts     # /dev/hugepages pagesize=2M, /dev/hugepages1G pagesize=1024M

Pull and launch (no sudo):

bash pull_sandbox_images.sh --tag 2.0.5 --type rock
bash start_sandbox.sh --tag 2.0.5 --type rock --num-sns 1 --bloomrf active --yes

Launching takes a few minutes. It is ready at "Sandbox is fully launched!", and ingesting at "batches confirmed on sn-1". For NVMe storage nodes the launcher refuses to start if /dev/hugepages1G is not a 1 GB mount or has no free pages, and prints the fix.

If sandbox_hw_config.json does not exist, start_sandbox.sh takes the same answers itself: --storage, --sn-storage, --numa-mode, --nic, --dedicated-interfaces, --grub and --yes.

Rules while it runs:

• Relaunch the whole lab between runs. Restarting one service, the proxy especially, loses its DMS registration and everything behind it answers 502. Recovery is a full relaunch.
• Use --num-sns 1 to 3 for a lab that ingests. --standalone never receives its targets from the DMS, and ingest hangs while every service looks healthy.

Step 4: ask about the test

Ask in one batch, once the lab is up.

Step 5: run and report

Browser tour. Replace HOST with the host's address. The lab's stand-in authentication lets everyone in as admin, so keep these ports off untrusted networks.

Query smoke test. Each helper talks to its service on the sandbox's front-end network and writes a JSON result beside itself.

bash query_mcp.sh                       # MCP execute_hd_query tool, 10 rows
bash query_ipflow.sh                    # HD GraphQL, 100 rows
bash query_ipflow.sh --sql 'SELECT ip_proto, SUM(bytes) AS bytes FROM IPFLOW GROUP BY ip_proto'
bash query_sql.sh                       # SQL service: rows per protocol
bash query_sql.sh --sql 'SELECT COUNT(*) AS n, MIN(timestamp), MAX(timestamp) FROM {table}'

A pass is rows from all three and no errors in the GraphQL result. Flows keep arriving, so run the GraphQL and SQL aggregates back to back to compare their totals. --help on each helper lists its options.

Benchmark. It stops the running lab and runs its own. Relaunch the lab afterwards if the user wants it back.

bash run_benchmark.sh --tag 2.0.5 --type rock --num-sns 1 --num-batches 10 [--phase 2] \
     [--records-per-batch 60000] [--jumbo] [--concurrent] [--no-cleanup]

Report each phase's PASS or FAIL, "All batches confirmed", and the peak ingest and query rates. The JSON report is in tests/test_logs/benchmark_*.json.

Health. docker ps --format '{{.Names}} {{.Status}}' shows no restarting containers, and bash sn_storage_status.sh shows each storage node's fill. For no rows, hangs, a storage node that will not start, or Rock startup problems, use fermihdi-hd-troubleshoot. Most sandbox problems are host prep (hugepages, their mounts, VFIO) or the relaunch rule, not the engine.

Tear down, as the user chose:

bash reset_sandbox.sh --yes --keep-hw --keep-logs   # stop everything; keep hardware config and reports
bash reset_sandbox.sh --yes                         # full reset

Undo the host changes only if the user asks:

sudo systemctl disable --now fermihdi-sandbox-hugepages.service
sudo rm /etc/systemd/system/fermihdi-sandbox-hugepages.service /usr/local/sbin/fermihdi-sandbox-hugepages
sudo sed -i '/^# BEGIN FermiHDI sandbox hugetlbfs/,/^# END FermiHDI sandbox hugetlbfs/d' /etc/fstab
sudo rm /etc/sysctl.d/99-fermihdi-sandbox-aio.conf
# Kernel command line: /etc/default/grub.fermihdi-<timestamp>.bak holds the previous file,
# and remove /etc/default/grub.d/99-fermihdi-sandbox.cfg if it exists, then sudo update-grub;
# on RHEL-family systems: sudo grubby --update-kernel=ALL --remove-args="hugepagesz hugepages default_hugepagesz"

Drives bound to vfio-pci return to their kernel driver at the next reboot.