Debugging event ID collisions in production Meta CAPI

A Slack message landed at 6 a.m. Pacific: ROAS dropped 28 percent overnight. The CAPI rebuild had been live for two days and looked clean for the first eighteen hours. Then Meta's 48-hour dedup window started catching up to the doubles, and every Purchase event we had been double-counting collapsed to one. The team thought the implementation had broken. It had not. The events were firing fine. The keys that were supposed to make them the same event were colliding silently, and BigQuery had been quietly logging the evidence the whole time.

This is the postmortem on that incident, the three collision shapes I now look for first, the BigQuery queries that surface them, and the production fix that held.

The incident

A Shopify DTC mid-rebuild, Q2 2024. Server-side CAPI had cut over two days earlier. The architecture matched the pattern from the tracking gap rebuild: GTM web plus GTM server on Stape, custom loader domain, eleven server-side event tags, SHA-256 event_id dedup. Match quality posted at 9.0 in the first twelve-hour verification window. The dedup rate showed 91 percent in Events Manager. Everything looked correct.

Eighteen hours in, the dedup rate quietly slid to 76 percent. By hour thirty, Meta's 48-hour dedup window was actively collapsing the doubles we had been producing, and the Ads Manager ROAS chart inverted. The team thought the rebuild had broken. It had not. Server events were returning 200 and browser fires were hitting Meta just fine; the collision sat at the key layer, not the transport layer.

I pulled BigQuery open and three queries told me everything I needed to know inside ten minutes. The events were duplicating because the keys that were supposed to identify them were drifting, and the drift had three distinct shapes underneath it.

Timeline

Times in UTC, recorded from the incident log.

t-48h: A refactor lands in the GTM web container. One line moves the hashing helper from a shared utility to an inline function inside the Purchase tag. The hash output is supposed to be byte-identical to the server's. Nobody re-tests with side-by-side Test Events because the unit test for the helper still passes.

t0: Cutover deploy goes live. First-hour dedup rate posts at 91 percent. Match quality at 9.0.

t+18h: Dedup rate drops to 76 percent. Nobody is watching the dedup tab specifically; the Overview tab still looks green.

t+30h: Meta's 48-hour window starts catching the earliest doubles. The collapse is invisible in Ads Manager for the first few hours because the data refreshes on a delay.

t+44h: ROAS chart in Ads Manager inverts. Team Slack lights up. Someone screenshots the chart. The hypothesis: the rebuild broke. We pause spend on the affected campaigns.

t+48h: I open BigQuery and run the first collision query. Two thousand orders, three thousand four hundred Purchase events. The pattern is immediate. I run two more queries. Root cause confirmed across three axes.

t+52h: Pinned the generator. Redeploy clean. Side-by-side Test Events shows one row per event again.

t+96h: Dedup rate settles at 92 percent and holds. Match quality settles at 9.1.

Root cause: three shapes of event ID collision

One incident. Three collision shapes underneath it. Each one would have been enough to drop the dedup rate on its own. All three were active at the same time, and the BigQuery queries below isolated them in that order.

Shape one: same event_id firing twice from different sources. The Shopify native Meta integration was still active for Purchase events while the GTM server container was also firing Purchase events. Both produced an event_id. Neither knew about the other. Meta saw two events with two different keys for the same purchase, could not dedup them, and counted both. The rebuild plan had assumed Shopify's native integration was disabled. It was not. Somebody had toggled it back on during a Meta sales-rep walkthrough a week earlier and nobody documented the change.

Shape two: browser/server timing race. The browser hashed window.ShopifyAnalytics.meta.checkout.order_id and the server hashed webhook.order_id from the orders/paid webhook. In testing these were the same string. In production, on a small but growing fraction of orders, the browser was reading a hydrated cart object from a persisted localStorage cache rather than the freshly painted analytics object. The two values drifted by a single character or by an underscore, and the SHA-256 outputs were entirely different. Two events, no dedup, both flagged low-quality.

Shape three: hash drift across systems. Klaviyo's Meta integration was active in parallel, sending its own Purchase event with its own event_id. Klaviyo's payload used the bare numeric Shopify order id (5841726209). GTM was using the GraphQL global id format (gid://shopify/Order/5841726209). Both hashed to valid SHA-256 strings, both reached Meta, neither matched. Two more events per purchase, on top of the two we already had from shapes one and two.

For some orders we were sending four Purchase events to Meta with four different keys. The 48-hour dedup window was the only thing standing between us and a dedup rate near zero, and even the window could not collapse all four into one. It collapsed the closest pair and counted the rest.

The BigQuery queries that surface duplicates

The most useful surface for this kind of incident is not Meta. It is your own warehouse. The CAPI logs in Events Manager show you per-event diagnostics one event at a time. BigQuery shows you the collision pattern across thousands of events at once, which is the only view that tells you whether the duplicate is a rare race or a structural bug.

For this to work you need server-side CAPI events landing in BigQuery as their own table, alongside the Shopify orders table. The wiring pattern is the same one I lay out in the warehouse rebuild for DTC analytics teams. Once the events table is there, three queries get you to the root cause faster than any vendor dashboard.

Query 1: count distinct event_id per order_id. This is the collision indicator. A healthy CAPI setup shows one distinct event_id per order_id for the Purchase event, plus one matching browser fire that shares the same key. Anything more than two is a collision.

select
  order_id,
  count(distinct event_id) as distinct_keys,
  count(*) as total_fires,
  array_agg(distinct event_source) as sources
from `dtc_warehouse.capi_events`
where event_name = 'Purchase'
  and event_time >= timestamp_sub(current_timestamp(), interval 48 hour)
group by order_id
having distinct_keys > 1
order by distinct_keys desc, total_fires desc
limit 50;

In a clean setup this returns zero rows. Mine returned about eighteen hundred. The distinct_keys column went up to four. The sources array was the second tell: every offending row showed three or four different sources contributing to the collision.

Query 2: pivot by event_source, find which sources are double-firing. Once you know there are too many keys, the next question is which integrations are producing them. This query rolls collisions up by source so you can see whether the issue is two integrations both firing or one integration firing inconsistently.

with pairs as (
  select
    order_id,
    event_source,
    count(*) as fires_per_source,
    count(distinct event_id) as keys_per_source
  from `dtc_warehouse.capi_events`
  where event_name = 'Purchase'
    and event_time >= timestamp_sub(current_timestamp(), interval 48 hour)
  group by order_id, event_source
)
select
  event_source,
  count(distinct order_id) as orders_touched,
  sum(fires_per_source) as total_fires,
  sum(case when keys_per_source > 1 then 1 else 0 end) as orders_with_internal_drift
from pairs
group by event_source
order by total_fires desc;

This was the query that exposed shape one and shape three at the same time. The output showed Shopify native, GTM server, and Klaviyo all listed as event sources for Purchase, each touching most of the same orders. orders_with_internal_drift flagged GTM specifically: even within that single source, some orders were producing two distinct keys, which pointed at the browser/server race.

Query 3: time-delta between collisions inside the 48-hour window. The third query measures how far apart the colliding events landed in time. This matters because Meta's dedup window is 48 hours, and events that arrive seconds apart are collapsed cleanly while events that arrive hours apart sometimes are not. The pattern in the deltas tells you whether your collisions are a hot-path race (sub-second) or a stale-cache problem (minutes to hours).

with ordered_events as (
  select
    order_id,
    event_id,
    event_source,
    event_time,
    row_number() over (
      partition by order_id
      order by event_time
    ) as fire_position
  from `dtc_warehouse.capi_events`
  where event_name = 'Purchase'
    and event_time >= timestamp_sub(current_timestamp(), interval 48 hour)
),
deltas as (
  select
    order_id,
    event_id,
    event_source,
    event_time,
    fire_position,
    timestamp_diff(
      event_time,
      lag(event_time) over (partition by order_id order by fire_position),
      second
    ) as seconds_since_prior
  from ordered_events
)
select
  order_id,
  event_source,
  event_id,
  fire_position,
  seconds_since_prior
from deltas
where order_id in (
  select order_id
  from ordered_events
  group by order_id
  having count(distinct event_id) > 1
)
order by order_id, fire_position
limit 200;

The output split cleanly. Most collisions landed within two seconds (browser and server racing on the same purchase). A second cluster landed at thirty to ninety seconds (Klaviyo's CAPI fired on its own delayed schedule). A third cluster landed at four to six minutes, which turned out to be the persisted-cart hydration issue: the customer hit the thank-you page, the browser pushed a stale cached order id, and the server-side webhook fired separately on the orders/paid event a few minutes later.

Three queries surfaced three different collision shapes, each one taking about ninety seconds to write because the schema was already in BigQuery; the rest of the work was reading the output.

What we changed

Five specific diffs, not platitudes.

First, we pinned the event_id generator. The hash inputs are now produced server-side from a deterministic ordering of fields: order_id_global_id + event_name + timestamp_floor_minute. The browser reads the pre-computed SHA-256 string from a data-event-id attribute on the thank-you page rather than hashing independently. One source of truth, no drift possible. The browser cannot disagree with the server because the browser does no hashing at all.

Second, we killed Shopify's native CAPI for every event GTM was already covering. The native integration is convenient when GTM is not in the picture, but running both is the cleanest way to produce the collisions in shape one. The fix is one toggle in Shopify admin, but it required a written decision in the repo so the next person who walks through the Meta admin does not turn it back on.

Third, we disabled Klaviyo's Meta integration for Purchase. Klaviyo continued to fire its own conversion events for its email attribution, but those went to Klaviyo's own pixel, not to Meta CAPI. The collision pattern in shape three vanished within an hour of toggling that off.

Fourth, we added a daily collision report. Query 1 above runs as a scheduled BigQuery query every twenty-four hours, writes the result to a capi_collision_log table, and pages the analytics channel if the row count exceeds a threshold. The threshold is currently set to 0.5 percent of the prior day's order count, which catches structural breaks early without paging on the long-tail of legitimate edge cases.

Fifth, we added a contract test for hash inputs. Any change to the file that defines the input fields, their order, or their normalization fails CI. The test reads a pinned set of test orders, generates the hashes, and diffs them against a checked-in fixture. A future refactor of the helper, the kind that started this incident, can no longer ship without explicit acknowledgement that the hashes will move.

What I would do differently

Three things, with the benefit of hindsight.

I would have written the collision queries before the rebuild went live. The three queries above are not complex. They take about an hour to write the first time and ten minutes to copy into a new repo after that. Having them ready means the first response to any dedup anomaly is a SQL run, not a Slack thread. The incident lasted fifty-two hours partly because the team's first instinct was to look at Meta's surface, where the data is one event at a time.

I would have treated BigQuery as the source of truth from day one of the rebuild. Meta's Events Manager is fine for spot checks, but the dedup chart aggregates over forty-eight hours and hides the structural cases under the noise of legitimate single-source events. The warehouse view, where I can see every event with its source, key, and timestamp on one row, is the only view that shows me what is actually happening. The pattern in the match quality audit covers the same warehouse-first instinct for a different metric.

I would have wired the dedup rate into the same observability surface as the conversion rate. A separate Meta dashboard is a separate context switch, and the team rarely flips between them under load. A unified surface that shows conversion rate, ROAS, and dedup rate in the same chart would have caught this within two hours of the rate slipping below 90 percent. The lag was almost entirely about which screen was open.

FAQ

Sources and specifics

• Incident occurred Q2 2024 at a Shopify DTC mid-rebuild. The architecture context is the same as the rebuild referenced in the opening section.
• Pre-incident dedup rate: 91 percent. Mid-incident: 76 percent. Post-fix: 92 percent settled at 96-hour mark. Match quality settled at 9.1 out of 10.
• Meta's documented dedup window is 48 hours from first event arrival; events with matching event_id and event_name within that window collapse to one count.
• BigQuery schema assumed by the queries: a capi_events table with columns for order_id, event_id, event_name, event_source, and event_time.
• The browser/server hash drift in shape two was traced to a persisted localStorage cart hydration; the fix was to remove client-side hashing entirely. The companion mechanics for purchase event hashing are in the purchase event hash walkthrough.
• The Klaviyo collision in shape three is consistent with the patterns catalogued in the six-shape CAPI failure tour; see also the Consent Mode v2 collision-layer notes for the consent-layer interactions that often sit alongside this kind of drift.
• For a Shopify-specific dedup wiring walkthrough that prevents these collisions in the first place, the dedup walkthrough for Shopify Pixel and CAPI covers the GTM and Liquid setup end to end.
• Operators who want a sanity check before they get to in-incident debugging can run the CAPI Leak Report; the fourteen-check diagnostic includes a dedup-rate check and surfaces collision-prone configurations early.