Addressing Gen AI’s Quality-Control Problem

Idea in Brief The Opportunity Generative AI has immense potential for automating the creation of content that enhances customer engagement and increases sales. The Challenge Gen AI has a tendency to produce unreliable outputs, such as hallucinations and irrelevant information. Traditional quality-control methods—such as human reviews and stand-alone testing tools—are expensive and can handle only a fraction of gen AI’s output. The Solution Amazon’s Catalog AI system shows how these hurdles can be overcome. It automatically detects and blocks unreliable data, improves itself using feedback from quality checks and experiments, and integrates A/B testing into its workflow to scientifically evaluate the effectiveness of gen AI content. For all the enthusiasm around generative AI, there’s a hurdle that is limiting its adoption: the technology’s tendency to make things up, leave things out, and create so many possibilities that it is hard to figure out which ones will be effective. For that reason, the vast majority of companies now employ human reviews and stand-alone testing tools or services to address generative AI’s deficiencies. However, both of those quality-control methods are expensive, and they can handle only a fraction of generative AI’s total output. Amazon has developed a better approach for its massive product-catalog operation: a generative AI–based system, named Catalog AI, that detects and blocks unreliable data; produces ideas for new product pages and tests their effectiveness; and improves itself automatically using feedback from quality checks and experiments. It can create and test tens of millions of hypotheses annually versus the thousands that most human-based systems can. Although many organizations are struggling to see a financial return on their AI investments, Amazon’s Catalog AI project is already creating measurable value. As of this writing, 8% of its suggestions have had a positive impact on sales revenue. One of us (Stefan) knows of other businesses whose online experiments have had higher success rates (from 10% to 20%). But those companies relied on people to generate hypotheses, and their systems for testing them were not nearly as automated—which means that they generated and tested far fewer hypotheses than Amazon did. What’s more, the ability of the Amazon system to automatically improve ensures that its yield rate will increase over time. Although Amazon considers the system, which was launched in 2023, a work in progress, we believe that it is far enough along that managers at other organizations can benefit from learning about how Amazon performs quality control on AI-generated content at scale. The Deficiencies of the Traditional Approach Amazon’s online catalog contains hundreds of millions of products that are sold to customers worldwide. Millions of product listings are added and edited daily. The data—images, titles, descriptions, and recommendations—must be complete, accurate, and appealing so that shoppers can find the products they are seeking quickly. Moreover, returning shoppers expect to see a familiar layout with images, product titles, descriptions, and buy buttons that are found easily and load quickly. The stakes are high: When product information is incomplete, irrelevant, or just plain wrong, customers don’t complete transactions or they return products that didn’t meet their expectations. In either case, Amazon loses money and customer trust. To ensure data quality, Amazon has traditionally drawn upon the expertise of thousands of employees who specialize in managing product listings. It also operated hundreds of machine-learning (ML) models, each optimized for a product category (such as shirts or TVs) and a layout component (such as product titles or descriptions). The specialists worked with the models to add or subtract information, identify inaccuracies, consolidate information, translate text into different languages, and incorporate data from third-party sources. These traditional ML models have limitations: They are best for smaller, structured datasets, and scaling them across product categories is costly. An ML model trained on, say, shirts cannot be cost-effectively used for TVs, or any other non-shirt product. By contrast, large language models (LLMs) are trained on large datasets and work across product categories. By replacing ML models with LLMs, Amazon simplified its technology infrastructure (fewer models) and organization (fewer functional specialists) and reduced costs. Ensuring Reliability of the New Approach For the first few weeks after Catalog AI was launched, about 80% of its results were unreliable. The system made up things, omitted information, or gave advice that customers weren’t interested in. For instance, it said that an electric pump had 15 horsepower when that information wasn’t available. Similarly, when asked about a sofa’s material, the model provided information about the material used in the frame rather than on the seats, which is what most customers were interested in. To remedy such quality issues and test the effectiveness of possible changes, Amazon took the following four steps. 1. Conduct an Audit To track progress, an organization needs to know its system’s baseline performance. In manufacturing, this is accomplished by evaluating a process during a stable period and using that information to determine control limits. Amazon had the LLM generate thousands of product pages that were already known. Human auditors then compared the LLM’s pages with the known information, scored their reliability, and analyzed the root causes of any poor performance. This led to a number of fast improvements, which we will describe next. 2. Deploy Guardrails A “hallucination”—a false or inaccurate output presented as if it were true—typically happens when a model comes to a conclusion that isn’t based on the input data. One way to improve reliability and avoid hallucinations is to constrain the LLM so that it produces outputs derived only from data specific to the business, not from general information on the web or data sources unrelated to the business. But there is a trade-off: The more freedom the LLM has to access both external and internal data, the more new ideas the system can explore, modify, and test. For instance, drawing from information it obtains from the web, the LLM may suggest that paper plates are not dishwasher safe. Putting too many constraints on an LLM’s inputs reduces its ability to make such inferences. For that reason, instead of limiting its LLM’s inputs, Amazon decided to install three other types of constraints. Simple rules. One approach to ensuring reliability is to instruct the system to reject content that doesn’t adhere to rules. A rule might be that a number that describes weight has to be followed by units—kilograms or pounds, for example. Amazon created a rule that Catalog AI must reject suggestions that make immaterial changes from the current listing (for example, switching the product style from “contemporary” to “modern”). Simple rules also dictate the page layout so that customers get a consistent experience throughout the site. Statistical profiles. Guardrails function like statistical process control (SPC) limits in factories. When process variables fall outside those limits, an alarm is triggered and the root cause of the problem is investigated. To create such guardrails for generative AI models, companies can use data on current products to determine if the output falls within an expected range. For example, an LLM might generate product information about a table that is sold by a third-party vendor. Information about typical tables sold in Amazon’s online store are used to generate the guardrails. When the product information created by the LLM falls outside the control limits, the LLM is questioned by another LLM. In some cases, the first LLM can recognize its own mistakes when it is prompted to explain the reason for its information. AI checking AI. One example is what we just described. But it is not possible to have rules or guardrails that cover every possible output from AI. The second generative AI system can deal with these. Amazon uses generative AI to look for problems that generative AI produced. The first LLM, the content generator, is trained to generate hypotheses; the second, the content reviewer, is trained to check the former’s output. They are connected and automatically engage in conversations drawing from their different background knowledge. Guillaume Kurkdjian For example, Amazon uses an LLM to detect inconsistencies across product pages—such as ensuring that the color in the product title matches the image color. If a discrepancy is identified, changes to a product’s information are automatically blocked. The generative AI system can be asked questions by the content reviewer, such as “Why is the new product page better than the existing page?” That forces the content generator to analyze the output and possibly abandon its earlier suggestion. To increase the rigor of reliability checks, Catalog AI can connect to internal and external LLMs that are trained on different data. Because the LLMs are trained on different information, they find different problems. For example, if the generator LLM makes a reasoning error and miscalculates product volume, the reviewer LLM—trained on a different dataset—is likely to detect the error and can block it. After a hypothesis from generative AI has passed all quality checks, it is then released to an experimentation platform, where it can be evaluated for effectiveness: Will the proposed change increase sales revenue or units sold? 3. Test the Effectiveness Companies need to find efficient ways to evaluate which of the many ideas produced by generative AI are effective. Previously, Amazon’s catalog specialists created rules and algorithms that automatically approved, designed, and improved the page layouts that they thought were most effective. That approach had several limitations: In creating those rules and algorithms, specialists included some untested assumptions, and they used testing that wasn’t always cost-effective or automated. Moreover, traditional market research (for example, focus groups and surveys) could be misleading because what customers said didn’t always match what they did. That made it very difficult to predict customers’ preferences. A Booking.com executive told one of us (Stefan): “We see evidence every day that people are terrible at guessing. Our predictions of how customers behave are wrong nine out of 10 times.” To learn which changes resonate with customers, the Catalog AI team integrated A/B testing into the system’s workflow. Scientifically testing the extremely high volume of output of generative AI requires an experimentation infrastructure: instrumentation (to record such things as clicks, mouse hovers, and event times), data pipelines, and data scientists. Several third-party tools and services make it easy to try experiments, but to scale things up a company must tightly integrate the capability into its workflow. At Amazon, the infrastructure is fully automated: All product-page changes proposed by Catalog AI are subjected to A/B tests. It runs a controlled experiment that compares two (or more) possibilities: “A” (the control, or champion) is the current product listing, and “B” (the challenger) is a modification of that information generated by AI to serve a distinct purpose—for example, to improve customer conversion rates. Users are randomly assigned to these experiences, and composite metrics are computed and compared. The metrics should align with strategic goals; the best short-term metrics also predict long-term outcomes. (See “The Surprising Power of Online Experiments,” HBR, September–October 2017.) In a recent experiment involving a skincare product, Catalog AI generated a challenger description that focused on key benefits; the champion’s description, by contrast, contained a long list of features and their benefits. While the champion delved into details of how the features produced benefits like unclogging pores, improving skin texture, and reducing redness, the much shorter version proposed by AI simply listed the product’s primary benefits: smoother, hydrated skin and reduced wrinkles. The shorter version increased sales significantly among a randomly selected customer group. Amazon’s generative AI–based system can create and test tens of millions of hypotheses annually versus the thousands that most human-based systems can. Other experiments revealed that the LLM was omitting some benefits from product titles. For instance, AI’s removal of the words “glowing skin” from the product title—it changed “Foaming Body Wash Mango Butter for Glowing Skin, 13.5 Ounce (Pack of 4)” to “Mango Butter Foaming Body Wash, 13.5 Ounce, 4 Pack”—had a negative impact on sales. Overall, the experiments have shown that about 40% of AI-generated content that passes Amazon’s reliability checks either improves primary performance metrics, such as shopper conversion into sales, or has no impact, positive or negative. The other 60%, however, yields significantly negative results. That content is typically not published in the company’s catalog. 4. Create a Learning System An ideal quality system should be a learning system that continuously improves with little or no human involvement. And Amazon’s system generates data that improves its LLM’s performance, making it more effective at challenging assumptions about what customers like or dislike. However, the company’s scientists found that it still pays to occasionally involve humans to generate better learning data. For example, human investigations of negative experiments sometimes find and correct flaws in the LLM. In one investigation, a team found that the LLM mistakenly wrote “no warranty” as the default whenever no information was provided. As Catalog AI improves, however, human involvement will become confined to system design and infrastructure decisions. Here are the components needed when building a learning system. Customer model. To accelerate feedback, companies can develop a composite metric that models customer preferences. Although strictly speaking, a customer model isn’t a requirement for building a learning system, it makes the system run much faster, provided that it is accurate. A model allows a company to get nearly instant feedback by running virtual experiments. Similar to running crash simulations of cars on a computer rather than smashing physical prototypes, testing AI output can be done much faster because real customers are no longer involved. Designing a metric that responds the same way as a typical customer requires thoughtful consideration. The company must determine which data to include and needs to provide validation through many customer experiments. Catalog AI uses a customer model along with live tests, the Catalog Data Quality (CDQ) metric. It includes information from rules, statistical profiles, and reliability checks. Eventually the CDQ will replace most tests involving real customers, which will accelerate the system’s learning. Multivariate experiments. Along with A/B tests, Amazon’s system employs multivariate experiments—more elaborate experiments that provide deeper insights into how multiple variables (for example, text, color, and images) interact or determine the best design choices. Amazon’s algorithms find patterns and investigate them via multivariate experiments on products whose web pages enjoy a high level of customer traffic and whose impact on primary performance metrics is statistically significant. The experiments that yield the most significant results (those that have more than a 1% positive impact on some primary metrics and less than 1% negative impact on others) are reviewed by data scientists for patterns and errors. Given the volume of customer traffic and transactions on Amazon’s marketplace, even small changes across millions of product listings can translate into billions of dollars in additional revenues. Companies with much less website traffic will have different thresholds for implementing a change than Amazon does. Although the goal is for Catalog AI to teach itself automatically, people contribute to its learning process now. Some experiments isolate effects and create anecdotes that can be shared between the team that manages Catalog AI and data scientists. For example, the Catalog AI Titles team may prompt the LLM to generate product titles with varying lengths and different information content: Brand X Plant Protein Powder, vanilla, 22g protein, 20 servings (75 characters) Brand X Plant Protein Powder, vanilla, 22g protein, non-GMO, vegan, no gluten, dairy, or soy, 20 servings (120 characters) Brand X Plant Protein Vanilla 22g protein, non-GMO, vegan, no gluten, dairy, or soy, no artificial flavors, synthetic coloring, preservatives, or additives, 20 servings (180 characters). Experiments determine which of the three product titles is most effective, and that is then used to train the LLM. By generating better training data across product categories, the quality of Catalog AI’s output has improved significantly: Now 80% of Catalog AI’s output passes the initial quality checks. Concept tests. When Catalog AI was launched, it generated millions of changes to listings that had to be tested on customers. Ideally, each change would have been tested in a stand-alone experiment, but many product listings don’t generate enough customer traffic to achieve the critical sample sizes needed. (To achieve statistical confidence, the smaller the expected effect of a change, the greater the number of observations needed to differentiate it from background noise.) When customer traffic isn’t robust enough—fewer than one million views—the experimentation platform automatically tests AI-generated content by batching similar product listings into single-concept tests to discover insights. A special algorithm batches content, sometimes from thousands of AI-generated listings, by product categories. (For products that get more than one million views, a stand-alone experiment is run.) The batches are as small as possible while meeting statistical-sample-size requirements. AI generates thousands of variants for improving the description of, say, coffeemakers, which are combined into a single-concept experiment. The experiments can challenge assumptions about customer preferences that have never been tested. For example, one experiment challenged a long-held assumption by specialists that had been built into ML models: that shoppers preferred a strong contrast between a white background and the displayed product. The concept test explored how images enhanced by AI would impact customer engagement. Thousands of AI-generated pages had suggestions for backgrounds that showed products in the shopper’s environment. These suggestions were batched in a concept test, which had a control with a white background and a model wearing a black thermal top. The image being tested added an enriched background, placing the model wearing the top in an indoor setting to help customers visualize the product in everyday use. The experiment found that adding the enriched background increased sales, and the change was implemented for hundreds of basic fashion products. Increase the Effectiveness of AI Projects Once quality systems are in place, they can do more than manage the quality of the AI-generated output; they can also help direct resources to the ongoing AI projects that have the highest ROI. Typically the business groups conducting these initiatives submit overly optimistic estimates of their anticipated financial returns, which, in turn, decide the allocation of specialists and computing resources. (In fact, 41% of 2,770 business leaders in six industries surveyed by Deloitte in 2024 said their organizations struggle to define and measure the impact of their generative AI efforts.) When, instead, the quality system tests the effectiveness of new projects with customers, leaders then can rely on hard evidence to decide the best way to allocate resources. It also allows leaders to revise the allocation more frequently in response to ongoing tests. For example, such a system allows a company to learn from prototypes—to quantify the impact of a project’s effort on subsets of customers—before it moves ahead with a broader rollout. Improvements to Catalog AI often start as prototypes, and they sometimes compete against each other. Teams run experiments and use the results to advocate for additional resources to scale their prototypes, and the winners are eventually rolled out. Amazon’s Project Amelia, its generative AI assistant for sellers, is gradually being made available to more sellers while the model’s performance is being monitored and customer feedback is incorporated. Experimentation can also help with managing AI infrastructure trade-offs, such as between model performance and computational cost. Increasing the size of LLMs may provide only marginal benefits, and they can take very long to train. By conducting experiments, Amazon has found that some smaller models outperform larger ones because they require fewer resources, can be retrained more frequently, and have faster response times to customer inquiries. Other companies can conduct similar experiments to decide the best path forward for themselves on questions like whether to develop in-house LLMs or use third-party models. . . . Decades ago, Harvard Business School professor David Garvin wrote about the sustainable advantage that quality systems gave to firms (see “Quality on the Line,” HBR, September–October 1983). In his study of the manufacturers of room air conditioners, he found that the failure rates of the products made by the highest-quality manufacturers were between 500 and 1,000 times less than those made by the lowest-quality ones. Achieving high quality, he concluded, came down to having a comprehensive quality-control system. Garvin’s lesson still holds true in the age of AI.

Addressing Gen AI’s Quality-Control Problem

Summary.

Partner Center