Architecture overview

Thadam is a single-process, single-binary edge runtime running exactly one pipeline per process. Its defining structural decision is capture-first: raw messages are written to a durable log and flushed before anything is decoded, filtered, or acknowledged. Everything downstream of that log consumes it independently.

One picture

                thadam process (one pipeline, one data directory)
  ┌───────────────────────────────────────────────────────────────────┐
  │ MQTT liveness task ──► SourceDriver: next_batch, submit,          │
  │ (owns protocol client)│              ordered ack FIFO             │
  │                       │        │ RawBatch (byte permits)          │
  │                       ▼        ▼                                  │
  │               WAL writer OS thread ── group commit ──► CommitTicket│
  │                       │                                           │
  │                       ▼                                           │
  │               ┌────────────┐                                      │
  │               │  Capture   │    read cursors                      │
  │               │  WAL       ├──────────┬──────────┬─────────►      │
  │               └────────────┘          │          │                │
  │                       ▼               ▼          ▼                │
  │               Chain (sink P): router → decode → transforms ──►    │
  │                       │        SegmentDriver (Parquet per dataset)│
  │                       │ rejects  │        │ finalize → cursor     │
  │                       ▼          ▼        ▼ commit → eligible     │
  │               Quarantine      Segment catalog ──► Shipper ──► S3  │
  │                                                               (M1)│
  │               Chain (sink C): … ──► BatchDriver (accumulate,      │
  │                                        retry, bisect) ──► ClickHouse
  │                                                                 │
  │               Supervisor · maintenance · admin HTTP · metrics    │
  └───────────────────────────────────────────────────────────────────┘

The same diagram in mermaid form is in the repository README and in ARCHITECTURE.md section 2.

One message, end to end

1. The MQTT client delivers a message. The source stamps ingest_time and queues it.
2. The SourceDriver pulls a batch and submits it to the WAL writer; the commit ticket goes into an ordered FIFO.
3. The WAL writer appends the batch as one frame, group-commits according to the fsync policy, and resolves the ticket.
4. The SourceDriver acknowledges the broker in receive order, and only now. Filtering or sink failure later can never affect acknowledgement.
5. Each sink’s chain reads the committed WAL from its own cursor: route → decode → dataset shaping → transform steps → the sink driver.
6. The segment sink writes one Parquet writer per dataset. A finalized file advances the sink’s watermark; the runtime commits the cursor durably; only then is the file eligible for upload.
7. The shipper uploads eligible files with deterministic keys and create-if-absent semantics, then records the upload in the segment catalog.

Because capture is first and chains read behind, a misconfigured filter leaves the raw WAL intact (fix the config, rewind the cursor, replay within the retained window), a slow sink never blocks capture or its sibling sinks, and the on-disk format never depends on the in-memory event model.

Milestone boundary

The roadmap in milestones (roadmap):

• M1 (MVP v0.1): MQTT source, capture WAL, local Parquet segments, S3-compatible shipping, replay, chaos verification.
• M2: ROS 2 (MCAP/rosbag2 offline ingestion first).
• M3 (P1): ClickHouse sink, after the M1 reliability gate.

The crates

The workspace is split by dependency weight and change rate, with the dependency rules enforced by scripts/check-deps.sh and cargo deny bans:

thadam-runtime knows connectors only through thadam-core traits; Arrow and protocol libraries never appear in core APIs. The graph and the rules are in ARCHITECTURE.md section 9.

Delivery discipline

Guarantees are stated per hop and only as strongly as the evidence allows; the details get their own page: Delivery guarantees.