name: fermihdi-hd-schema-build description: Build an HD dataset schema from an existing database's DDL (CREATE TABLE text, any dialect) or from a corpus of files (CSV, JSON, JSONL, Parquet, ORC, Avro, Protobuf, XML), using the hosted hd-schema-generator API, then check it for real ingestability against sample data before a real ingest is attempted. Use before fermihdi-hd-ingest or fermihdi-hd-migrate when no HD dataset schema exists yet.

Build and validate an HD dataset schema

Produces an HD dataset schema (schemas/hd_dataset_schema.json's format) from something that isn't one yet — another database's tables, or a pile of files nobody wrote a schema for — and checks it before it meets real data. Once a schema exists, hand off to fermihdi-hd-ingest (set up the pipe) or fermihdi-hd-migrate (a one-time bulk load).

What actually exists, and what doesn't

• hd-schema-generator is a stateless API, not a database connector and not a file parser. It turns DDL text into a schema. It does not connect to a live database and does not read CSV/Parquet/etc. files itself — for a file corpus, you build the input by hand (Step 2b).
• It has no sample-data validation. Its /v1/validate endpoint checks a schema's own internal structure and byte layout — never a schema against real records. The real check against actual data is fermihdi-hd-dry-run (Step 5) — the ingester's own --dry-run mode, run standalone from the published image, no lab or sandbox needed.
• Endpoint: https://hd-schema-generator.fermihdi.io (hosted, stateless, rate-limited, no sign-in — reach it directly with curl, no local build). If the target HD instance is on a private network the public endpoint can't reach, the generator also runs self-hosted as a DMS add-on component (source checkout, docker compose up --build, serves on :8080) — ask the user which applies before defaulting to the public URL. It is not published as a cr.fermihdi.io image; a self-hosted run is always from source.

Step 1: pick the starting point

Step 2a: from an existing database's DDL

Get the CREATE TABLE text from the source — pg_dump --schema-only, MySQL SHOW CREATE TABLE, a CQL DESCRIBE TABLE, or hand-written ANSI SQL. Multiple tables in one call are fine.

curl -sX POST https://hd-schema-generator.fermihdi.io/v1/parse \
  -H 'Content-Type: application/json' \
  -d '{"dialect": "postgresql", "source": "<CREATE TABLE ...>"}'

dialect is one of ansi_sql, postgresql, mysql, cql. Returns a LogicalModel (columns, inferred types, key roles) plus findings for anything ambiguous. HTTP 422 means the DDL parsed but yielded zero columns — check the dialect and the DDL text, not the tool. Review the model with the user before continuing; a mis-inferred column type here becomes a wrong byte layout downstream.

Step 2b: from a corpus of files (no source DDL)

There is no endpoint for this — build a LogicalModel JSON object yourself from the corpus's own metadata, then skip straight to /v1/generate:

• Parquet/ORC/Avro carry their own typed schema — read it directly (e.g. a Parquet file's embedded schema, pyarrow.parquet.read_schema() or equivalent) rather than sniffing values.
• CSV/JSON/JSONL have no embedded types — infer them from a real sample of rows (numeric vs. string vs. timestamp-shaped, array vs. scalar), not just the header names.
• Protobuf — the .proto message definition is the schema; there is nothing to infer.
• XML — infer from the record element's real attribute/child shape across a sample, the same way as CSV/JSON.

Name each column and give it the closest HD field type (§3) before calling /v1/generate.

Step 3: generate the HD schema

curl -sX POST https://hd-schema-generator.fermihdi.io/v1/generate \
  -H 'Content-Type: application/json' \
  -d '{"model": <LogicalModel from 2a or 2b>}'

Returns a candidate HD dataset schema with byte layout already assigned (the only layout strategy is declared-field-order packing — it does not reorder fields for you), plus proposals for HD-specific concepts it inferred: dataset_type, time_index_field, batch_id_field, compressed_time_fields, enum detection. Proposals are never auto-applied. Review each one with the user, then re-call with {"model": ..., "overrides": {...}, "confirmed": true} to lock them in.

HD field types, for judging the proposals: unsigned/signed integers (uint8..int256), float/double, boolean, numeric (numeric_precision ≤38 digits → 16 bytes, ≤76 → 32 bytes), string/bytes (caller-sized; string is UTF-32 by default — 4 bytes/char — unless string_encoding: ascii, 1 byte/char), blob (a 64-byte object-store key; requires the dataset to have a batch_id_field), ipv4/ipv6/timestamp/chess_move (semantic aliases over uint32/uint128/uint64), uuid (16 bytes, optional v6/v7 bit-relabelling), chess_fen (32 bytes packed), chess_move_cc (2 bytes, chess.com's own lookup code). An array object on a bytes field needs element_type, max_elements, element_width, count_bytes, and its total span must equal count_bytes + max_elements × element_width exactly.

batch_id_field — a dataset schema needs a way to group records into batches (a closed batch is never reopened by the storage cluster): a rounded time field (round_batch_id_field: true + time_index_interval/_unit, rounds down from epoch), a raw field's own value (records must already arrive grouped by it — no ingester-side sort happens for this form), or nothing at all (the ingester's own clock). This choice belongs in the proposals review, not left to the default.

There is no written schema-design-optimization guide in this codebase — no doc says when to prefer enum over a raw integer, how field order affects anything, or when array/blob beats a fixed-width field. The rules of thumb below are this skill's own judgment, not a cited platform doc — say so if the user asks where they come from:

• A bounded, small set of repeating string/int values (status codes, categories) → enum, not a raw wide field.
• A field the SN's BloomRF filter node would usefully index (selective, queried often) benefits from staying a plain fixed-width type rather than living inside an array/blob, which the filter can't see into.
• Prefer a rounded time field for batch_id_field whenever the corpus has one — it's the only form that gets automatic sort-on-ingest; a raw-value batch key demands the source already arrives grouped by it.
• Reach for blob/overflow only for genuinely variable-length or rarely-read data; it costs an extra KV round trip per read.

Step 4: validate

curl -sX POST https://hd-schema-generator.fermihdi.io/v1/validate \
  -H 'Content-Type: application/json' \
  -d '{"schema": <the schema from step 3>}'

Always returns HTTP 200 — read the valid field and findings, don't trust the status code. This checks structure (against the published hd_dataset_schema.json meta-schema) and physical layout (byte-span overlaps/gaps, record_len arithmetic, type-width mismatches) only — it has not looked at a single real record yet.

Step 5: the check /v1/validate can't do — a dry run against real data

Use fermihdi-hd-dry-run: point the published ingester image's --dry-run mode at a copy of the real corpus (or the live source) with this schema — no lab, no sandbox, no DMS, no SCC, no licence. It reports mapping failures, dropped array elements, clamped values and unmapped enum codes as real counters, not silent drops, and a clean run is the closest thing this codebase has to "the schema validates against the corpus." Reach for a full fermihdi-hd-sandbox deployment only if the question moves past schema/source compatibility into real query or cluster behavior.

Handoff

Once the schema passes Step 5: fermihdi-hd-ingest to wire up an ongoing pipeline, or fermihdi-hd-migrate for a one-time bulk load. Registering the schema on a real (non-sandbox) deployment goes through DMS-Core's schema-update API — see fermihdi-hd-ingest §"Production schema registration" for the exact call and its data-destroying gotcha before changing a schema on a cluster that already holds data.