Distillation

["23dda6fb1d3728caff630d0ca8e56d94","a26d8aaeac0165aab40f6abefe27ea7e"]

<observations> Date: July 30, 2026 * 🔴 (16:47) User clarified architecture: server listens on a specific UDP port; all consumers send data toward it. User's expectation is "we should not be missing events" because consumers are still running * 🟡 (16:47) User floated hypothesis: the volume drop is "because we are filtering out a bunch of duplicates" (i.e., caused by the reprocess flag) * 🟡 (16:47) User asked: is the drop solely caused by duplicate filtering via the flag, or could there be another reason for the drop * 🟡 (16:47) Assistant confirmed: yes, likely the flag's mechanism — without it, the UDP server gets one signal per delivery including Kafka re-deliveries/retries/re-processing that escape the in-memory dedup cache (1-hour TTL on `ev:{project_id}:{event_id}`). Turning on the flag IS turning on duplicate suppression, and drop magnitude reflects prior duplicate rate * 🟡 (16:47) Assistant enumerated OTHER mechanisms that could amplify or be the actual cause of the drop: 1. **`inline_save_event=True` (5ae4736ff76, Jun 15, 2026)** — taskbroker passthrough runs `preprocess_event` inline; exceptions caught and re-raised as `Retriable`. Persistent failures → DLQ → `event_accepted` never fires. Check `metrics.timer("ingest_consumer.process_event")` and DLQ topic 2. **`Organization.DoesNotExist` at processors.py:246** — silent drop if project's org was deleted. Look for "Organization does not exist" warnings in consumer logs 3. **UDP packet loss** — UDP isn't reliable; if server is overrun or network drops packets, that's a separate loss path. Compare `event_accepted` signal firings against UDP-server arrivals * 🟡 (16:47) Assistant proposed verification steps: - Temporarily disable the flag on one consumer pod and watch volume — if it spikes back, flag is the culprit - Instrument the receiver to log/count `event_accepted.send_robust()` fires per process_event run. Compare that count to UDP-server-arrival count * 🟡 (16:47) Assistant diagnostic logic for verification: if both confirm real signal-firing drop → flag's duplicate suppression is the cause. If receiver fires ~N signals but UDP gets only a small fraction → look at UDP path. If receiver also fires few → dig into why `process_event` isn't reaching line 313 (cache failures, processing failures, organization drops) * 🟡 (16:47) Assistant offered two follow-up actions: (a) look at where `event_accepted`