FermiHDI HD Wire Protocol — Network & MTU Guide
The FermiHDI HD Wire Protocol (FHDWP) is a Layer 5 binary format used exclusively on the Data Plane (port 6987) for the unidirectional, high-speed transfer of record batches between FermiHDI HD nodes. This guide covers FHDWP's MTU constraints, the rationale behind the default packet size, and the conditions under which Jumbo Frame mode may be safely enabled.
MTU Constraints
FHDWP enforces a strict Maximum Transmit Unit to prevent IP fragmentation across all supported network topologies, including heavily encapsulated environments such as MPLS and Metro Carrier Ethernet.
| Mode | MTU (bytes) | Condition |
|---|---|---|
| Standard | 1200 | Default for all HD instances |
| Jumbo Frame | 8500 | Admin-elected, instance-wide, network must be verified |
The 1200-byte standard MTU is intentionally conservative relative to the Ethernet standard of 1500 bytes. This headroom absorbs encapsulation overhead from:
- VXLAN (Docker overlay / Kubernetes Flannel/Calico/Cilium): +50 bytes
- IPIP (Calico in IPIP mode): +20 bytes
- MPLS labels: +4–12 bytes per label
- IPsec ESP (encrypted data planes): +60–70 bytes
Starting from 1200 bytes ensures that even triple-encapsulated paths remain below the Ethernet 1500-byte boundary, eliminating fragmentation entirely.
[!IMPORTANT] The FHDWP MTU is an instance-wide setting. All nodes in an HD instance must use the same MTU. Mixing standard and jumbo MTU nodes within a single instance will cause packet loss and data corruption.
Why Larger Packets Are Not the Answer
It is tempting to raise the FHDWP MTU to increase throughput. However, fragmentation has a disproportionate impact on FHDWP:
- FHDWP payloads consist only of whole records. A record may not span two packets (except in the defined multipart case). A fragmented packet that loses one IP fragment therefore causes the loss of an entire record — or forces a multipart reassembly path that defeats zero-copy delivery.
- TCP retransmission at the L4 layer will eventually recover fragmented packets, but the retransmit timeout adds milliseconds of jitter to what is otherwise a sub-millisecond ingest pipeline.
- Fragmentation in the network is silent: routers fragment without notifying the sender, and the FHDWP receiver cannot distinguish a fragmented reassembly from normal delivery.
The performance gain from larger packets (fewer syscalls, less header overhead) does not justify the reliability risk in production environments where every intermediate hop is not under your administrative control.
Jumbo Frame Support: Docker & Kubernetes
Jumbo Frame mode (FHDWP MTU = 8500 bytes) requires an end-to-end MTU of at least 8550 bytes (8500 bytes payload + FHDWP headers + IP/TCP headers + any encapsulation). Every hop from sender to receiver must be configured consistently. The table below reflects the real-world support landscape as of 2025.
| Environment | Jumbo Frame Support | Notes |
|---|---|---|
| Docker bridge (single host) | ✅ Configurable | Set "mtu": 9000 in /etc/docker/daemon.json. Host NIC must also be set: ip link set <nic> mtu 9000. Works reliably on bare metal. |
| Docker overlay (Swarm mode) | ⚠️ Possible, complex | VXLAN adds 50 bytes of encapsulation overhead. Physical NICs must be ≥ 9050 bytes MTU. A single misconfigured intermediate switch silently drops oversized frames. |
| Kubernetes + Flannel (VXLAN) | ⚠️ Possible, complex | Same VXLAN 50-byte overhead as Docker overlay. Physical MTU must be ≥ 9050. Flannel does not propagate MTU changes automatically to running pods; restart required. |
| Kubernetes + Calico (IPIP mode) | ✅ Best CNI option | Calico has explicit jumbo frame support. IPIP adds only 20 bytes. Configure mtu in the FelixConfiguration CRD. All nodes must agree. |
| Kubernetes + Calico (eBPF / no encap) | ✅ Excellent | No encapsulation overhead at all if BGP peering is configured. Physical MTU = pod MTU. Highest throughput option. |
| Kubernetes + Cilium (eBPF) | ✅ Good | Cilium honours the kernel NIC MTU and propagates it to pods automatically via CNI config. Jumbo frames work if the physical infrastructure supports them. |
| Kubernetes + Weave | ⚠️ Possible, complex | VXLAN-based with 50-byte encapsulation overhead. Same caveats as Flannel. |
| AWS (any CNI) | ⚠️ Limited | Enhanced Networking (ENA) supports up to 9001-byte MTU, but only between instances in the same placement group on the same host. Cross-AZ traffic is capped at 1500 bytes. |
| GCP (any CNI) | ❌ Not supported | GCP's VPC enforces a 1460-byte MTU ceiling on all VM network interfaces. Jumbo frames are not available. |
| Azure (any CNI) | ❌ Not supported | Azure's virtual network stack caps MTU at 1500 bytes for all VM SKUs. Jumbo frames are not available. |
| Bare-metal (physical Ethernet) | ✅ Full support | Configure switch ports to jumbo mode and set NIC MTU to 9000 on all hosts. No encapsulation overhead. Recommended environment for Jumbo Frame mode. |
[!WARNING] Cloud environments (AWS, GCP, Azure) almost never provide reliable end-to-end jumbo frame support. Even where the hypervisor NIC supports larger frames, cloud load balancers, VPN gateways, and inter-AZ routers typically silently re-fragment or drop oversized packets. Do not enable Jumbo Frame mode on cloud-hosted HD instances without explicit network-level verification.
Enabling Jumbo Frame Mode
Jumbo Frame mode is an admin-elected, instance-wide setting. It must be consistent across all nodes in the HD instance before any FHDWP traffic flows.
Pre-flight Checklist
Before enabling, verify the entire data path:
# 1. Verify the host NIC MTU on each SN / SCC host:
ip link show <interface>
# Expected: mtu 9000
# 2. Verify end-to-end path MTU from SN to SCC (no fragmentation):
ping -M do -s 8472 <scc_ip>
# 8472 = 8500 FHDWP MTU - 28 bytes (IP + ICMP headers)
# If this succeeds without fragmentation, the path supports jumbo frames.
# 3. For Docker bridge deployments, verify daemon MTU:
docker network inspect <network_name> | grep -i mtu
# 4. For Kubernetes, verify pod MTU matches node MTU:
kubectl exec -it <sn_pod> -- ip link show eth0
Configuration
Set the following in the HD instance configuration before bringing up any nodes:
{
"data_plane": {
"mtu_mode": "jumbo",
"mtu_bytes": 8500
}
}
[!CAUTION] Changing the MTU mode on a running instance requires a full cluster restart. In-flight FHDWP packets use the old MTU and will be misinterpreted by nodes running the new MTU. There is no rolling upgrade path for MTU changes.
Summary
| Decision | Recommendation |
|---|---|
| Default deployment (cloud, Docker, K8s) | Use standard 1200-byte MTU — no action required |
| Bare-metal cluster with controlled switch infrastructure | Jumbo Frame mode may be enabled after full pre-flight verification |
| Mixed environment (some nodes bare-metal, some cloud) | Not supported — instance MTU must be uniform |
| Performance optimisation via larger MTU | Fragmentation risk outweighs throughput gains; pursue hash-chain or pipeline optimisations instead |
See also: Architecture Overview | SN Configuration Guide