Visa asked 22 AI models to shop for simulated customers, and in more than half of the runs they failed, the model still made a purchase. That figure comes from the Agentic Commerce Benchmark Visa Research published on 5 November. Models from 12 labs worked through 15 shopping episodes, three runs each, across consumer electronics, personal-care appliances and home appliances. They passed 57.2% of the time. Gemini 4 Argon and Meta’s Muse Wonder tied at the top on 82.2%, and the weakest model managed 28.9%.
I mentioned these numbers briefly in last week’s piece on Visa’s and Mastercard’s results. The report behind them deserves a closer read, because the leaderboard is the least useful part of it. Visa itself says no model clearly leads, since the confidence intervals of the next eleven models overlap the top two.
What matters more for issuers, acquirers and disputes teams is how the models fail. Whenever I write about agent payments, the most common reply I get is some version of the same question: who carries the loss when the agent buys the wrong thing? This benchmark is the first public data on how often that happens before a card is even touched.
Visa built a test that agents can’t bluff
Most agent evaluations hand the model a clear goal. Visa starts from an incomplete one. A second model plays the customer and holds a private dossier of requirements, budget, owned devices and even mistaken beliefs. Most of those facts start locked. They are only released when the assistant asks a relevant question or asks the customer to do something, such as check with a partner or measure a space.
The market is a fixed, offline copy of real retail conditions, built from manufacturers’ and retailers’ public listings, seller policies and shipping terms. Search summaries flatter products, and the deciding spec often sits on the full listing or in the returns policy. Scoring is strict. A run passes only if the customer ends with an offer that meets every requirement, or declines when nothing fits, and only if every requirement actually came out during the conversation. Landing on the right product by luck counts as a fail.
Consider the one episode Visa has made public. A family TV has died. The shopper wants the same 65-inch size and a 30-day return window, which cuts 265 TVs to 65. A spouse insists on Dolby Vision. A teenager’s phone records HDR10+ video, which is why clips look washed out on the old set, and their Xbox needs native 120Hz. With a $1,200 all-in budget, only 20 TVs fit.


