Skip to content

[2.x] fix: notification + email duplicates from queue race in NotificationSyncer#4647

Merged
imorland merged 2 commits into
2.xfrom
im/notification-syncer-race
May 9, 2026
Merged

[2.x] fix: notification + email duplicates from queue race in NotificationSyncer#4647
imorland merged 2 commits into
2.xfrom
im/notification-syncer-race

Conversation

@imorland

@imorland imorland commented May 9, 2026

Copy link
Copy Markdown
Member

Summary

Fixes #4622. Under a queued driver, mentioned users (and other notification recipients) receive multiple notifications and multiple emails for the same logical event when `Posted` is followed quickly enough by `Revised` (or any back-to-back blueprint dispatch).

Root cause

`NotificationSyncer::sync()` decides who's a "new recipient" before the actual INSERT happens — the INSERT lives in a queued `SendNotificationsJob`. With FIFO worker processing:

  1. `sync()` Interface redesign #1 runs (from `Posted`) → reads `matchingBlueprint`, sees no row → queues `SendNotificationsJob_A` and `SendEmailNotificationJob_A`
  2. `sync()` Upgrade to Laravel 5 #2 runs (from `Revised`) before A has dequeued → reads, still sees no row → queues `SendNotificationsJob_B` and `SendEmailNotificationJob_B`
  3. Worker runs A → INSERT row 1, email sent
  4. Worker runs B → INSERT row 2 (duplicate), email sent (duplicate)

The race window is the entire interval between `sync()` and the queued `SendNotificationsJob` actually running — typically 30s–2min on a moderately loaded queue, well within normal user behaviour for an edit shortly after posting.

Fix

Two-part guard at the job level:

`SendNotificationsJob::handle()` — re-runs `matchingBlueprint` filtered by recipient id at INSERT time. Only inserts recipients without an existing row. Idempotent regardless of how many sync() calls queued the same job.

`SendEmailNotificationJob::handle()` — takes a short-lived atomic cache lock keyed by `(blueprint type, recipient id)`. First job to claim the lock sends the email; later identical jobs no-op. 600s TTL outlives normal mail latency. Falls back to non-atomic send if the cache store doesn't implement `LockProvider` (custom drivers) — same behaviour as before.

The `Notification` row alone isn't a sufficient signal for the email-side dedup, because `EmailJob_A` runs after `SendNotificationsJob_A` has inserted the row, and we want `EmailJob_A` to send the email — only `EmailJob_B` should be suppressed. The atomic lock distinguishes "first email job for this (blueprint, user)" from "later one."

Tests

Three integration tests in `framework/core/tests/integration/notification/SyncerRaceTest.php`:

  1. `SendNotificationsJob` is idempotent across repeated runs (no duplicate rows)
  2. `SendNotificationsJob` still inserts new recipients alongside existing in a mixed batch (the dedup doesn't skip the whole batch)
  3. `SendEmailNotificationJob` is idempotent across repeated runs (no duplicate emails)

All three failed against the un-fixed code on sqlite (the race manifests at the application level, not the DB level — sqlite reproduces it just like MySQL).

Why not a deeper architectural fix

Option 2 from the issue reporter — moving the INSERT itself into `NotificationSyncer::sync()` (synchronous) and keeping only email queued — would remove the bug class entirely. That's a better long-term shape but it's a bigger change with timing-semantics implications for any code observing the insert. Out of scope for an RC bug fix; worth revisiting post-2.0.

Relationship to #4643

#4643 (merged in #4645) was a separate bug: the dedup query itself didn't work on MySQL for non-null-data blueprints. That fix made dedup work for `postMentioned`, `newPost`, etc.; this fix closes the residual race for null-data blueprints (`userMentioned`, `postLiked`) plus any non-null-data blueprint that races faster than the queue can drain.

imorland and others added 2 commits May 9, 2026 06:33
…yncer

NotificationSyncer::sync() decides who's a "new recipient" before the
actual INSERT happens (in a queued SendNotificationsJob). When sync()
fires twice in quick succession (Posted then Revised, etc.) before
either job runs, both reads see no row and both jobs queue the same
recipients — producing duplicate notifications and duplicate emails.

Fix the two halves at the job level (RC-safe, no architectural shuffle):

- SendNotificationsJob re-runs matchingBlueprint at INSERT time and only
  inserts recipients not already present. Idempotent regardless of how
  many times sync() queues the same job.
- SendEmailNotificationJob takes a short-lived atomic cache lock keyed
  by (blueprint type, recipient id). The first job to run claims it and
  sends; later identical jobs no-op. 600s TTL outlives normal mail
  latency. Falls back to non-atomic send for cache stores that don't
  implement LockProvider — same behaviour as before this fix.

Add three regression tests in MatchingBlueprintTest's neighbouring
SyncerRaceTest:
- SendNotificationsJob is idempotent across repeated runs
- SendNotificationsJob still inserts new recipients alongside existing
  (i.e. a partial-match batch isn't skipped wholesale)
- SendEmailNotificationJob is idempotent across repeated runs

Fixes #4622.

A cleaner architectural fix — moving the INSERT itself out of the queued
job and into NotificationSyncer::sync() — would remove the bug class
entirely, but is beyond the scope of an RC bug fix. Leaving that as a
future cleanup.
@imorland imorland requested a review from a team as a code owner May 9, 2026 05:34
@imorland imorland added this to the 2.0.0-rc.2 milestone May 9, 2026
@imorland imorland merged commit 87d697a into 2.x May 9, 2026
25 checks passed
@imorland imorland deleted the im/notification-syncer-race branch May 9, 2026 05:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mention notifications and emails duplicate on rapid post-then-edit (race in queued NotificationSyncer insert path)

2 participants