name: fermihdi-hd-dry-run description: Verify a data source (a Kafka topic, a Parquet/CSV/JSON/ORC/Avro/Protobuf/XML corpus, RabbitMQ, MQTT, S3) is set up correctly and its records actually map to a dataset schema, using the ingester's --dry-run mode. No HD deployment, no DMS, no SCC and no licence needed — this runs the published ingester image standalone. Use this instead of standing up a lab or sandbox just to test ingestibility.

Verify a data source with a dry run

hd_message_bus_ingester --dry-run reads, decodes and maps records exactly like a real ingest — same schema mapping, same batch-id handling, same per-field warnings — but never registers with a DMS, never connects to an SCC, and needs no licence. It's the fast, isolated way to answer "will this actually ingest?" before touching a real deployment. Needs only: the published cr.fermihdi.io/hd/ingester image, a copy of the source data (or reachable broker/bucket), and a dataset schema (fermihdi-hd-schema-build if one doesn't exist yet).

Step 1: get the pieces

Image: cr.fermihdi.io/hd/ingester:<tag> (no separate build or lab required).
Schema: a local file, or a schema already embedded in a config JSON's dataset_schema field. Dry run never fetches from a DMS — there is no DMS in this picture.
Data: for a file/S3 corpus, copy it into a scratch directory first. The file ingester deletes a source file once it's processed, dry run included — never point this at the only copy of anything.

Step 2: run it

docker run --rm \
  -v /path/to/corpus:/data:ro \
  -v /path/to/schema.json:/schema.json:ro \
  cr.fermihdi.io/hd/ingester:<tag> \
  --dry-run --ingression-type file --file-dir /data --schema /schema.json

Swap --ingression-type and its source flags for the real target: kafka with --kafka-brokers/--kafka-topic/--kafka-group, s3 with --s3-bucket/--s3-prefix/ --s3-region/--s3-endpoint (plus S3_ACCESS_KEY_ID/S3_SECRET_ACCESS_KEY env vars — never a CLI flag for these), rabbitmq with --rabbitmq-url, mqtt with --mqtt-address/--mqtt-topic. Every flag the real ingester takes for a source works the same way in dry run.

Auto-exit controls — dry run is a bounded check, not a standing service, so it stops itself instead of running forever:

Flag / env	Default	What it does
`--dry-run-idle-exit-sec` / `FERMIHDI_DRY_RUN_IDLE_EXIT_SEC`	`5`	Stop after this many seconds with no new record — including when nothing ever arrived at all (a misconfigured source fails fast, not silently forever). `0` disables auto-exit: run until Ctrl-C, for watching a live Kafka topic as long as wanted.
`--dry-run-max-records` / `FERMIHDI_DRY_RUN_MAX_RECORDS`	`0` (unlimited)	Stop after this many records — a quick smoke test against a huge corpus without waiting for the whole thing.

Step 3: read the summary

Every run ends with a block like this, whether it stopped itself or was interrupted:

=== DRY RUN SUMMARY ===
Source              : file
Duration            : 12.4 s
Records read        : 1970000
Batches formed      : 33 (reopened: 0)
Would-send to SCC   : 1180000 packets, 6904850000 bytes (nothing was actually sent)
Mapping warnings:
  enum values defaulted        : 0
  array elements truncated     : 0
  time deltas clamped          : 0
  chess values unmapped        : 12
  uuid values unmapped         : 0
  uuid values generated        : 0
  time-reference source missing: 0
No records were sent to any Storage Cluster Controller. No licence was required.

How to read it:

Records read = 0, exit code 1: the source produced nothing at all — check the connection details (broker reachability, bucket/prefix, directory path) before looking at the schema at all.
reopened > 0: records arrived out of batch order — for a file/S3 source with an unrounded batch_id_field, this means the corpus isn't pre-sorted by that field (dry run doesn't fix ordering for you, same as a real ingest); for a rounded time field it usually means clock skew or a corpus spanning a gap.
Any mapping-warning counter above zero: real, actionable signal. A steady trickle across a large corpus (well under 1%) is often tolerable and expected (an occasional chess move this codebase's codebook hasn't cracked yet, say); a counter climbing in lockstep with records read means a systematic mismatch — a field name that doesn't match the schema under any naming convention, a genuinely wrong type, or a source using a different unit/format than the schema declares.
A clean run (all warnings zero, reopened: 0, records read > 0): the source and schema are compatible. This is the strongest evidence this codebase has for "will this actually ingest" short of doing it for real.

What this replaces

Before this mode existed, the only way to check a schema against real data was standing up a disposable sandbox or lab and running a real ingest into it — see fermihdi-hd-schema-build's validation step and fermihdi-hd-migrate/ fermihdi-hd-ingest's source setup, all of which now point here first. Reach for a real sandbox (fermihdi-hd-sandbox) only when the question has moved past "is the source readable and schema-compatible" to something dry run can't answer — actual query behavior, storage-cluster capacity, or end-to-end timing.

Not a separate image