What GA4 does and doesn’t do

Google Optimize shut down on 30 September 2023. GA4 can’t split traffic or change a page, so something else has to:

  1. Assign each visitor to a variant, and keep them there.
  2. Tell GA4 which variant each visitor saw.

The exposure event

Google’s developer documentation specifies one event for this: experience_impression, with one parameter, exp_variant_string. The value names the tool, the experiment and the variant, joined by hyphens:

gtag('event', 'experience_impression', {
  // TOOL-EXPERIMENT-VARIANT: who assigned it, which test, which arm
  exp_variant_string: 'RIG-HERO1-B',
});

Send it on every page load where the experiment runs, for the control too: a control visitor without the event is invisible.

Assignment has to be sticky

A visitor who sees A today and B tomorrow belongs to neither. Draw once and keep it:

// One draw per visitor and experiment, kept, so a returning visitor
// sees the same variant. Runs before the page is shown.
function variantFor(experiment, variants) {
  const key = 'exp_' + experiment;
  let v = localStorage.getItem(key);
  if (!variants.includes(v)) {
    v = variants[Math.floor(Math.random() * variants.length)];
    localStorage.setItem(key, v);
  }
  return v;
}

const v = variantFor('HERO1', ['A', 'B']);
if (v === 'B') document.querySelector('h1').textContent = 'Find your best version';
gtag('event', 'experience_impression', { exp_variant_string: 'RIG-HERO1-' + v });

Apply the variant before the page shows, or visitors see the original flash first.

Make the variant a dimension

Admin → Data display → Custom definitions → Create custom dimension:

  • Dimension name: Experiment variant
  • Scope: Event
  • Event parameter: exp_variant_string

It only fills from the moment it exists: register it before the test starts.

Read the result in Explorations

Compare users, not sessions: each visitor was assigned once.

  1. In Explore, start a Free form exploration.
  2. Build one user segment per variant: users who triggered experience_impression where Experiment variant exactly matches RIG-HERO1-A, and the same for B.
  3. Apply both segments. Add Total users and your key event’s count (or Purchase revenue) as metrics.
  4. Conversion rate per variant is converting users divided by total users, per segment.

Where the GA4 interface misleads

Good enough to watch a test, not to decide one:

  • Thresholds. With Google signals on, GA4 withholds rows with few users. A small variant can drop out of a report entirely.
  • Sampling. Explorations are sampled once a query covers more events than the property’s quota (10 million for standard properties). Long tests on busy sites hit it.
  • Modelled users. With consent mode, GA4 can estimate visitors who declined analytics cookies. Estimates aren’t observations, and they weren’t assigned to anything.
  • Conversions before exposure. A user segment counts every conversion the user ever made in the date range, including ones before they first saw the test.
  • Visitors in both variants. Someone who cleared cookies, or switched devices while logged in, can match both segments.
  • No statistics. GA4 shows two conversion rates. It doesn’t say whether the difference is larger than chance.

When to move to BigQuery

As soon as the decision matters. The BigQuery export (Admin → Product links → BigQuery links) is free and holds every raw event, unsampled. A/B test analysis in BigQuery gives the query.

Or let agents run it

Riggle’s AI agents build, assign and measure every change in your own analytics, against one goal, and keep getting better on their own.

Sources