Skip to content

FermiHDI End-to-End Test Documentation

This describes a retired harness. run_e2e_tests.py and data_generator.py are no longer in the repository, and the image names and environment contract below predate the 2.x images — the DMS is two images rather than one, and nodes take their configuration from the DMS registration response rather than from a mounted config file. It is kept for the pipeline and chain-of-custody description, which still holds. The harnesses that run today are:

Test Exercises
tests/integration_test_full_pipeline Kafka → Ingester → SCC → SN → query via Proxy → test head
tests/integration_test_dms_pipeline registration, provisioning and the DMS side of the same path
tests/integration_test_sc_cluster a Storage Cluster on its own
tests/sandbox_dms the full deployment, and the benchmark suite

The end-to-end integration test (run_e2e_tests.py) validates the entirety of the FermiHDI Data Management stack, proving that configuration mappings, mTLS certificate bootstrapping, parsing ingress pipelines, and underlying storage deployments work flawlessly within a containerized environment.

This document describes exactly how the test is structured, how data is processed, and how the various containers are interconnected.

1. Test Overview

  1. Environment Setup: Tears down any lingering state and creates a dedicated Docker network (hd_e2e_net) on the 172.25.0.0/16 subnet.
  2. Data Generation: Executes data_generator.py locally to build an expected schema (test_dataset_schema.json), generate millions of rows grouped into CSV batches, and export baseline DuckDB validated target query answers.
  3. Container Orchestration: Instead of relying heavily on -v volume mounts which can present synchronization bugs or directory assumption bugs across platforms, the script extensively utilizes docker create, docker cp, and docker start commands to inject files immutably before booting a service.
  4. Cryptographic Verification: The data_generator.py now supports multiple hashing algorithms (defaulting to Hybrid XXH64). It calculates a continuous chain hash across all batches. Each batch's opening_hash and closing_hash are recorded in batch_mapping.csv, allowing the system to verify the "Chain of Custody" during ingestion and storage audits.
  5. Ingestion Triggering: After the PKI cluster binds securely, the orchestration script moves the newly generated .csv files into the hd_message_bus_ingester's input engine directories.
  6. Validation Run: Queries are forwarded directly against the Data Plane proxy to ensure they properly route payloads downstream spanning Storage Nodes (SN), and the Query Verifier assesses accuracy.

2. Container Topology and Configuration

The test environment provisions eight distinct Docker containers interconnected within the statically assigned 172.25.0.0/16 IP range.

2.1 DMS (Data Management Server)

  • Image: cr.fermihdi.io/hd/dms-core:2.0.2
  • IP: 172.25.0.10
  • Ports: 6986:6986 (Southbound API), 3000:3000 (Web UI)
  • Execution: sh -c "dms-ca & dms-core"

2.1.5 Observability Stack (Telemetry)

  • Image: otel/opentelemetry-collector-contrib:latest and getsentry/relay:latest
  • IPs: 172.25.0.11 (OTel), 172.25.0.12 (Sentry Relay)
  • Ports: 4317 (OTLP), 3000 (Relay)
  • Role: Managed dynamically by the DMS OpAMP supervisor on port 4320.
    • Note: Overrides the entrypoint to run the Certificate Authority sidecar concurrently to allow zero-touch provisioning.
  • File Injections:
    • dms-core.crt -> dms:/etc/fermihdi/x509/dms-core.crt
    • dms-core.key -> dms:/etc/fermihdi/x509/dms-core.key

2.2 FermiHDI Proxy

  • Image: cr.fermihdi.io/hd/proxy:2.0.2
  • IP: 172.25.0.20
  • Ports: 6987:6987 (Data Plane)
  • Environment Variables:
    • HD_PROXY_CONFIG=/tmp/config.json
    • HD_DATASET_SCHEMA=/tmp/test_dataset_schema.json
  • File Injections:
    • proxy_config.json -> proxy:/tmp/config.json
    • test_dataset_schema.json -> proxy:/tmp/test_dataset_schema.json
    • fermihdi_root_ca.pem -> proxy:/tmp/dms_ca.crt (The matched Root CA to verify dms-core's certificate)

2.3 Message Bus Ingester

  • Image: cr.fermihdi.io/hd/ingester:2.0.2
  • IP: 172.25.0.30
  • Startup Flags: -c /test_data/ingester_config.json
  • File Injections:
    • Host directory /test_data/ -> ingester:/test_data
    • Host directory /empty_dir/ -> ingester:/input

2.4 SCC (Storage Cluster Controller)

  • Image: cr.fermihdi.io/hd/scc:2.0.2
  • IP: 172.25.0.50
  • Environment Variables:
    • DMS_URL=https://172.25.0.10:6986

2.5 Storage Nodes (SN1, SN2, SN3)

  • Image: cr.fermihdi.io/hd/sn:2.0.2
  • IPs: 172.25.0.101, 172.25.0.102, 172.25.0.103
  • Environment Variables:
    • DMS_URL=https://172.25.0.10:6986
  • File Injections:
    • Host directory snX_drives/ -> snX:/drives (Contains an emulated.dat sparse file acting as a 2GB raw block device for DPDK testing)

2.6 Query Target (Verifier)

  • Image: cr.fermihdi.io/hd/test-head:2.0.2
  • IP: 172.25.0.200
  • File Injections:
    • Host directory /test_data/ -> query_target:/app/test_data

3. Network Flow & mTLS Execution Phase

When the containers spin up, they coordinate over the following strict security lifecycle:

  1. PKI Bootstrapping: Proxy, Ingester, SCC, and the SN instances natively utilize the DMS_URL=https://172.25.0.10:6986 configuration property. They will securely handshake using normal TLS validation verifying dms-core.crt against the root CAs provided to them.
  2. Registration and Singing: Nodes send an authenticated CSR to the DMS. The DMS uses its local dms-ca process internally to blindly sign those specific requests (in testing/demonstration) and hand back the signed End-Entity verification artifacts.
  3. Data Flow: The script actively executes docker cp to transport .csv files into ingester:/input/{f}. The Ingester polling loop instantly catches the file and converts the contents into the binary high-speed Data Plane formats and transmits it across the 172.25.0.0 subspace to the respective SCC and SN infrastructure reliably without test script intervention.

4. Hash-Chain Auditing

To ensure zero-tampering from generation to query, the batch_mapping.csv file contains two cryptographic linkage fields: * opening_hash: The state of the cryptographic chain before the first record of the batch was processed. * closing_hash: The state of the cryptographic chain after the last record was processed.

The XXH64_HYBRID scheme uses an 8-byte chain value with a 64-bit seed rotation function derived from SHA512 at the end of each batch. For long-term archival, the SHA512 algorithm can be selected using the --algo sha512 flag on the generator.