Session 0rWXqG93IDSI

Trace of event_accepted.send_robust() in src/sentry/ingest/consumer/processors.py:313-316

The call sits inside the try block at line 179, so it's the last thing that runs in process_event() before the cache.set dedup write. Two failure modes: not called, or called-but-throws.

Conditions where it is NOT called

All early returns run before line 313 — none of them emit event_accepted:

| # | Location | Condition | |---|----------|-----------| | 1 | processors.py:131 | Duplicate event — cache.get(deduplication_key) returns non-None | | 2 | processors.py:144 | Killswitch store.load-shed-pipeline-projects matches | | 3 | processors.py:150 | orjson.loads(payload) raises (JSON is malformed) — this is OUTSIDE the try block, so it propagates unhandled and is NOT wrapped as Retriable | | 4 | processors.py:175 | Killswitch store.load-shed-parsed-pipeline-projects matches | | 5 | processors.py:188 | reprocess_only_stuck_events=True and event not in processing_store | | 6 | processors.py:196 | reprocess_only_events_not_in_nodestore=True and event IS in nodestore (new in #120133) | | 7 | processors.py:246 | Organization.DoesNotExist for the project's organization | | 8 | anywhere in try block | Any exception (storage save, cache.set, preprocess, etc.) — caught at line 317, re-raised as Retriable. KeyErrors are re-raised directly without the Retriable wrapper. | | 9 | feedback path | data["type"] == "feedback" and is_in_feedback_denylist(project.organization) is True — track_outcome is emitted but the message falls through to the post-save code at line 308+. Wait — this DOES still emit event_accepted because the denylist branch is inside the try and execution continues. Re-check: yes, after the if/else the code falls through to cache.set and event_accepted.send_robust(). So denylisted feedback events DO emit the signal. (Not a NOT-called case.) |

Conditions where it may FAIL or ...

The most likely culprit is d7fcb07bc42 (Jul 21) — the reprocess_only_events_not_in_nodestore flag.

That PR added a new early-return at line 196:

if reprocess_only_events_not_in_nodestore:
    with start_span(op="nodestore.exists", name="nodestore.exists"):
        node_id = Event.generate_node_id(project_id, event_id)
        if nodestore.backend.get(node_id) is not None:
            return

If this flag is enabled on the main ingest consumers (not just reprocess tools), every event whose body is already in nodestore exits before event_accepted.send_robust() runs. Since most events arriving at the consumer are reprocesses / duplicates by Kafka, the result looks exactly like a slow volume drop matching the rollout timeline.

Verify:

# Check consumer config / launch flags for the flag
grep -r "reprocess-only-events-not-in-nodestore" --include="*.py" --include="*.yaml" --include="*.toml"

Also check the Sentry options / env vars on the consumer processes.

Second-likeliest: 5ae4736ff76 (Jun 15) — inline_save_event=True in taskbroker passthrough mode (#117580). When preprocess_event runs inline, any exception there is caught at line 317 and re-raised as Retriable, so the message is retried; if it keeps failing, it goes to DLQ and event_accepted is never emitted. Before that PR, those exceptions were absorbed by Celery. Check if the taskbroker passthrough path is enabled and if preprocess_event is throwing — metrics.timer("ingest_consumer.process_event") and the kafka_consumer retry/DLQ metrics in your dashboards will show this.

Less likely but worth checking:

• Your own @receiver(event_accepted) handler — if "another service" is rate-limiting or down, send_robust swallows the receiver exception (returns it in responses) but the signal still fires. Instrument the receiver to log if it's failing.
• Organization.DoesNotExist (line 246) — if any of the source projects have been deleted, those ...

How it causes the issue

The flag was added in #120133 specifically for reprocess tooling — scripts that want to re-run save_event for events that failed before and were deleted from the processing store. The check skips anything that's already in nodestore, because those events are already done:

# processors.py:192-196
if reprocess_only_events_not_in_nodestore:
    with start_span(op="nodestore.exists", name="nodestore.exists"):
        node_id = Event.generate_node_id(project_id, event_id)
        if nodestore.backend.get(node_id) is not None:
            return  # ← exits before line 313

In normal ingest traffic, Kafka is at-least-once. Events that the user's processor "saw" before are sitting in nodestore. So the moment the flag flips on for the consumer(s) running their processor, the early-return at line 196 short-circuits the signal for every event that already made it through, and the receiver only fires for genuinely new traffic.

Result: event_accepted signal volume collapses, but the events themselves are still being ingested normally — they just never reach the user's processor.

Why it doesn't fire event_accepted

The signal sits at line 313 after the reprocess check. The semantics the PR author encoded are: "if we decided not to re-save, the event is not 'accepted' from this consumer's POV." That's fine for reprocess tooling but wrong for any side-effect that wants to mirror all traffic.

Proper fix

Three options, ordered by preference:

1. Don't enable the flag on the consumer fleet that runs the user's processor. This is the intended use of the flag — it should be on a dedicated reprocess consumer (e.g. events-reprocessing), not on the main ingest consumer (events). Check the consumer config:

# on the consumer host / configmap / sentry options
grep -r "reprocess_only_events_not_in_nodestore" sentry.conf.py sentry/

If it's set globally, scope it to the reprocess consumer only. ...

Then Option 1 is the fix. The flag needs to come off whatever consumer is running your receiver. Find it:

Flag is wired as a CLI arg in src/sentry/consumers/__init__.py:247:

["--reprocess-only-events-not-in-nodestore", "reprocess_only_events_not_in_nodestore"],

So this is per-process startup, not a Sentry option. Check whichever consumer(s) run your event_accepted receiver:

# on the consumer host
ps aux | grep -E "events\.ingest|events\.reprocessing" | grep -o "reprocess-only-events-not-in-nodestore\|reprocess-only-stuck-events"

# or via the runbook
ps -ef --forest | grep -i 'sentry run\|consumer'

If --reprocess-only-events-not-in-nodestore is on the same consumer class that runs your processor (likely events.ingest-consumer), drop it. The factory in src/sentry/ingest/consumer/factory.py:91-94 actually errors if you try to combine it with --reprocess-only-stuck-events, so the two are mutually exclusive reprocess knobs.

Once you remove the flag, restart the consumer and your processor will start receiving event_accepted for events already in nodestore again.

Your intuition is right that the flag mainly skips events already in nodestore. The volume drop comes from the events that slip past the in-memory dedup (the cache.get check at line 121) but aren't really "new."

Mechanics:

A process_event call fires event_accepted only if it runs to the end of the try block. There are two filters that prevent it from running:

1. Dedup cache (processors.py:117-131) — cache.get(deduplication_key) returns the event_id if it was processed in the last hour.
2. reprocess_only_events_not_in_nodestore flag (processors.py:192-196) — checks if the body is in nodestore.

In a healthy steady state, the dedup cache catches most "already-processed" events within an hour. The reprocess flag is a second filter for events that get past it. So on the main ingest consumer, the flag's impact should be small.

The scenario where the drop is large is when your processor is attached to a consumer where most events are already in nodestore by design — i.e., a reprocessing consumer like events-reprocessing. There, the events arriving at the consumer are re-deliveries of events that were already saved in the past. Without the flag, every one of those fires event_accepted (relying on the dedup cache to keep things sane). With the flag, almost none of them do, because their bodies are in nodestore.

So the question to verify is: which consumer is your receiver attached to?

If it's on events.ingest-consumer (the main one), the flag's impact should be modest and the drop you're seeing probably has another cause. If it's on events-reprocessing or a similar consumer with the flag enabled, then the drop is the expected behavior — and your processor is simply attached to the wrong consumer for forwarding new accepted events. ...

Likely yes — that's the mechanism. The flag's job is to filter out events that have already been accepted (body in nodestore). Without it, your UDP server gets one signal per delivery, including Kafka re-deliveries, retries, and re-processing attempts that escape the in-memory dedup cache (the 1-hour TTL on ev:{project_id}:{event_id}).

So "we used to get a bunch more events and now we get a trickle" matches: turning on this flag is exactly turning on duplicate suppression, and the magnitude of the drop reflects how many duplicates your consumer was emitting before.

That said, a couple of other mechanisms could amplify it or be the actual cause:

1. inline_save_event=True (5ae4736ff76, Jun 15). When the taskbroker passthrough runs preprocess_event inline, any exception there is now caught and re-raised as Retriable. Persistent failures → message lands in DLQ → event_accepted never fires. Check metrics.timer("ingest_consumer.process_event") and your DLQ topic for reprocess-mode consumers.

2. Organization.DoesNotExist (processors.py:246). Silent drop if the project's org was deleted. Look for Organization does not exist warnings in the consumer logs.

3. UDP packet loss. UDP isn't reliable — if your server is overrun or the network drops packets, that's a separate loss path. Compare event_accepted signal firings (instrument the receiver to count) against what the UDP server actually receives. If they match, the loss is upstream; if signal firings exceed UDP arrivals, your forwarder is dropping.

To verify it's the flag and not one of the others:

• Temporarily disable the flag on one consumer pod and watch the volume — if it spikes back, the flag is the culprit.
• Instrument the receiver to log/count event_accepted.send_robust() fires per process_event run. Compare that count to your UDP-server-arrival count.

If both confirm a real signal-firing drop, then yes — the flag's duplicate suppression is the cause. ...

I've traced the receiver and there's nothing suspect locally:

getsentry/receivers/orbital.py:22-46

@event_accepted.connect(weak=False)
def notify_orbital(ip, data=None, **kwargs):
    if udp_socket is None:
        return
    try:
        result = geo_by_addr(ip)
    except Exception:
        return
    ...
    udp_socket.sendto(dumps(data).encode("utf-8"), udp_addr)

It connects with weak=False so it stays alive regardless of module GC. Each event_accepted fire calls geo_by_addr(ip) and sends one UDP packet. If geo lookup fails, it silently returns — so a geo dataset hiccup wouldn't cause a sustained drop weeks long.

Loading chain (where it gets activated):

getsentry/models/__init__.py:67   →   import getsentry.receivers
                                     └─ getsentry/receivers/__init__.py:8
                                        └─ from .orbital import *
                                           └─ @event_accepted.connect(...)

So it activates in any getsentry Django process — including the ingest consumers.

No recent changes to orbital.py, receivers/__init__.py, or models/__init__.py in the last 2 months (git log --since="2 months ago" returns empty for those paths). The last meaningful change was 22243ea6ab (Mar 2023) — graceful handling for the single-tenant config, irrelevant to volume.

Conclusion: the drop is not in the receiver or its module load. It's strictly upstream — fewer event_accepted.send_robust() calls reaching line 313 in process_event. That confirms the earlier diagnosis: the reprocess_only_events_not_in_nodestore flag is suppressing the signal for events already in nodestore, and since process_event only runs to completion when the signal fires, the orbital UDP sender never gets a chance to send those events.

Quick ways to verify before touching config:

1. ...