Article

How Amplitude checks whether AI-suggested product changes help

Amplitude Wave’s experiments show how to assess AI-suggested changes, interpret a reported 160% lift and separate shipping from proven customer benefit.

Editorial illustration for How Amplitude checks whether AI-suggested product changes help: a document represents the research briefing. Not documentary evidence.

Amplitude announced paid access to Wave for Growth and Enterprise customers on 12 November 2026. Its AI agent investigates product usage and proposes improvements, giving product managers and software-business owners another way to decide what deserves development time.

According to its documentation, Wave links customer evidence to proposed changes, human review and measured outcomes. An OpenAI case study published the same day follows a change to Amplitude’s own suggestion cards. The practical question is what the resulting numbers establish about customer benefit.

From skipped cards to a reviewed change

On Amplitude’s AI-agent homepage, suggestion cards offered ready-made prompts. OpenAI describes Wave examining 100 session replays—recordings of product interactions. In that sample, 80 sessions involved typed questions and 12 a card selection. That investigation sample does not establish the size of the later experiment.

Wave proposed tailoring cards to recent charts, dashboards or the page a user had come from. The plan preserved manual typing and generic cards when personal context was unavailable. According to the case study, the team reviewed the proposal, handed implementation to a coding agent and ran an A/B test, comparing the changed experience with an unchanged control group.

What the reported 160% lift measures

Amplitude’s more detailed account, published on 9 November, dates the original experiment to July. It reports 2.43 suggestion sends per 100 exposed users with personalized cards, against 0.94 with generic cards, and calls the result provisional, requiring further validation. These are company-reported results, not independently audited findings.

Using those rounded figures, the calculation is:

  • Absolute difference: 2.43 − 0.94 = 1.49 additional suggestion sends per 100 exposed users.

  • Relative increase: 1.49 ÷ 0.94 × 100 ≈ 159%, consistent with the company’s roughly 160% figure.

The unit matters: these are sends, not additional unique users. Amplitude’s measurement documentation distinguishes event counts per exposed user from the share of people who convert. More card-driven requests could reflect a different way of starting a task; they do not by themselves demonstrate more completed work, higher retention or extra revenue.

The published accounts do not disclose the experiment’s group sizes, raw sends or confidence intervals. They therefore do not allow readers to assess the precision of the reported lift.

The same Amplitude account also describes an onboarding change with activation rates of 24.61% in the changed experience and 17.75% in control. Despite the higher number, the company calls that result statistically inconclusive. It remains unresolved—not a confirmed improvement and not proof of no effect.

What to require before calling a change successful

For a team applying this approach, a useful review has four parts:

  1. Check the underlying evidence. Open the cited recordings, analytics and feedback. Wave’s guidance treats small replay samples as clues to investigate, not proof that a problem is widespread. A plausible recommendation still needs a testable explanation.

  2. Define the customer outcome before building. Choose what would count as useful progress, which users count as exposed and what must not worsen. Microsoft’s experimentation guidance recommends success measures, guardrails—outcomes to protect—and data-quality checks. For suggestion cards, a proposed companion measure would be successful task completion, not just card use.

  3. Approve the behavior and confirm delivery. Agree on checks showing that the change works as intended. Then confirm that users actually receive it. Wave’s lifecycle documentation says its Shipped status can follow completed development work; that status alone does not confirm deployment to the live product.

  4. Keep the result attached to the original question. Record what was measured and whether it supports a decision. Amplitude’s analysis guidance explains that an uncertainty range spanning no difference leaves the effect unresolved. Do not turn that into a win because implementation is finished.

The responsibility split is a separate decision. RohitAI’s Holiday Extras analysis examines how business specialists can help build with AI while engineers retain technical responsibilities.

Access, cost and the time needed to learn

As of 12 November, Wave requires a Growth or Enterprise plan and is a paid offering. Amplitude’s pricing page lists those plans as custom-priced; it does not establish a standalone Wave price. Ask for a Wave-specific quote and included usage rather than assuming the platform’s free AI features include it.

Amplitude’s readiness guidance also calls for meaningful analytics and session-replay data, a business goal, and product and engineering owners able to act on recommendations. The workflow depends on those foundations as well as the agent.

For a smaller software business, the delivery deadline and the evidence deadline may differ. Amplitude’s experiment-planning guidance explains that detecting smaller effects needs larger samples. Low traffic can therefore mean a longer wait to learn, even when a change is quick to build.

Start with one customer journey and an explicit decision: what evidence would justify keeping, revising or withdrawing the change? A functioning feature answers whether the team built it correctly. Outcome measurement answers whether it helped.