FFORGE//RS
← Roadmap

distributed systems / LEVEL 4

A million one-row inserts

EST.50 MIN
01

THEORY / RETRIEVAL

What to restore

  • Batch rows by both row count and byte budget
  • Sort output for the chosen analytical locality
  • Keep the highest version for a replayed event identity

Analytical stores reward batches

Tiny synchronous inserts create excessive parts and coordination overhead. Producers or Kafka consumers should batch within bounded latency, row and byte budgets. Backpressure must propagate when ClickHouse maintenance or merges cannot keep up.

ORDER BY is a physical access decision

A MergeTree ordering key should follow the dominant filters and locality, while partitioning should stay coarse enough to avoid a huge partition count. Replacing-style deduplication is asynchronous, so query correctness cannot blindly assume duplicates disappeared immediately.

CHECKPOINT

Why are thousands of one-row inserts usually worse than bounded batches for a MergeTree workload?

ISOLATED RUST 1.96
src/lib.rsEDIT

02 / IMPLEMENTATION

Implement the contract

Prepare bounded insert batches. Deduplicate by (tenant_id, event_id), retaining the highest version; sort by tenant_id, timestamp_ms, event_id; then split by max_rows and max_bytes. Reject zero limits or any row that cannot fit by itself.

Initializing editor…
CLOUD SANDBOXnetwork off · 256 MB · 12 s
1 / 64 KB
OUTPUT
Runner is waiting for a submission.