name: fermihdi-hd-migrate description: Load an existing corpus of Parquet, ORC, Avro, Protobuf, XML, CSV, JSON or JSONL files into HD as a one-time bulk load, or migrate an existing database's data into HD via export-and-load, CDC streaming, or a direct SDK feed. For an ongoing/continuous ingestion pipeline instead of a one-time load, use fermihdi-hd-ingest. Needs a dataset schema first — see fermihdi-hd-schema-build.

Bulk-load or migrate data into HD

This is for moving data that already exists somewhere else into HD once — a corpus of files, or another database's contents. For setting up a pipeline that keeps ingesting going forward, use fermihdi-hd-ingest instead; the two skills use the same mechanics but this one is about getting a fixed body of existing data in, safely, and knowing when it's done.

Needs a dataset schema first. If one doesn't exist yet, use fermihdi-hd-schema-build before anything here — a schema built here from scratch with no review is the single biggest risk to a bulk load, since record_len becomes very hard to change once a cluster is running (see "Production schema registration" below).

Step 1: pick the path

Step 2a: file corpus → HD

The file ingester watches a directory and reads every file that lands in it, in any of: CSV, JSON, JSONL/NDJSON, Parquet, ORC, Avro, Protobuf, XML. Protobuf needs a compiled descriptor set (protoc --descriptor_set_out --include_imports) plus the message type name; XML needs the record element name only when records aren't the root's direct children.

Copy the corpus into the drop directory — never point the ingester at the original files. The file ingester deletes a source file as soon as it's acknowledged. This has cost real data before; treat it as a hard rule, not a suggestion.

Configure ingression_type: file with a directory and sort_by. If the schema's batch_id_field is a rounded time field, the ingester sorts and computes the batch id for you automatically before sending. If it's a raw field value (not rounded), no automatic sort happens at all — the files must already be written in the order that field increases, or records will scatter across batches and the storage cluster will reject late arrivals for an already-closed batch. Check which form the schema uses before assuming batching is automatic.

Memory: sorting a file (only needed for the rounded-time-field case) holds the whole file in memory. A real ~69GB/19.6M-row file has driven RSS past 60GB doing this. Split a bulk corpus into chunks sized to the host's RAM rather than handing the ingester one huge file.

Record size: the widest record a dataset may declare is 8433 bytes; the ingester refuses to start above that. Batches close by batch_close_timeout_mode: dynamic (default — closes after the longest span seen so far in the batch field), manual (a fixed timeout, default 300s), or off (only a later batch closes it).

Step 2b: export a database, then run 2a

pg_dump/COPY (Postgres), mysqldump or a proper SELECT ... INTO OUTFILE-style export (MySQL), or any ETL step that lands Parquet/CSV/JSONL — then treat the export as the corpus in 2a. There is no live database connector built into HD; this export step is unavoidable for a one-time snapshot migration.

Step 2c: continuous replication via CDC

HD has no native CDC connector. The real path is a Kafka Connect source connector (a Debezium CDC connector, or a plain JDBC source connector) publishing into the topic HD's Kafka ingester reads. Configure ingression_type: kafka with brokers/topic/group_id on the HD side; the Kafka Connect layer is entirely the customer's own infrastructure. No SASL/auth env vars exist for the Kafka or MQTT ingestion sources in this codebase today — if the broker requires authentication, that's a real gap to flag rather than something to configure around.

Step 2d: a direct SDK feed

For a one-time scheduled pull with no file or broker hop at all: a small program using the SDK's FermiHDIIngestClient (C++ or Python bindings) registers as an HD_CLIENT node and sends batches directly over FHDWP. This is the right choice when the source is queryable on a schedule and staging it as files or a broker topic is unnecessary overhead.

Naming — the corpus doesn't need to match the schema's exact spelling

Field lookup tolerates any naming convention (snake_case, camelCase, PascalCase, kebab-case) against the schema's own field names — an exact match is tried first, a normalized fallback second, and two source keys that could both match the same schema field with neither an exact hit are skipped rather than guessed. A corpus doesn't need a rename pass before ingest just because its column names don't match the schema's convention.

Step 3: production schema registration — read before touching a live cluster

The sandbox's own provisioning scripts write a schema directly into lab config; a real deployment registers a schema through DMS-Core's management API instead (POST /v1/dms/schema/update, with instance_id when more than one deployment exists). If record_len changes on a fleet that's already running, this is refused (409) unless teardown_acknowledged: true is set — and setting it destroys stored data. Never set that flag without the user's explicit, informed confirmation; if a schema needs a record_len change on a cluster that already holds data the user cares about, stop and make sure they understand what they're agreeing to before proceeding.

Step 4: watch it land

Health and metrics on port 8081 (/healthz, /metrics). Watch records-in, batches-closed, and ack lag from the storage cluster as the primary signal a bulk load is progressing and finishing cleanly, not just that the process is still running.

What's actually proven vs. still scaffolding

File-source ingest of a real multi-gigabyte Parquet corpus, including the batch-id and naming-convention handling above, is confirmed working end-to-end. The S3-compatible object-store source is wired up (bucket/prefix/region/endpoint, credentials via S3_ACCESS_KEY_ID/ S3_SECRET_ACCESS_KEY env vars, never in the DMS-distributed config) but has not been exercised against a real S3-compatible target — treat it as code-complete, not load-tested. RabbitMQ and MQTT sources exist in the code and are documented but weren't exercised recently either.

Troubleshooting

Records silently missing from batches, "already closed" batch errors, or a stall mid-load — see fermihdi-hd-troubleshoot. Set up a schema first with fermihdi-hd-schema-build if none exists. For an ongoing pipeline instead of a one-time load, switch to fermihdi-hd-ingest.