<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Trusted AI]]></title><description><![CDATA[Trusted AI examines the governance and observability required to build AI systems that scale. Views expressed are my own.]]></description><link>https://trustedai.recodework.com</link><image><url>https://substackcdn.com/image/fetch/$s_!iiXZ!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf42fa95-926b-4942-903f-bc3fe1ff1dd2_1280x1280.png</url><title>Trusted AI</title><link>https://trustedai.recodework.com</link></image><generator>Substack</generator><lastBuildDate>Sat, 21 Nov 2026 10:11:32 GMT</lastBuildDate><atom:link href="https://trustedai.recodework.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Jon Knisley]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[trustedai@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[trustedai@substack.com]]></itunes:email><itunes:name><![CDATA[Jon Knisley]]></itunes:name></itunes:owner><itunes:author><![CDATA[Jon Knisley]]></itunes:author><googleplay:owner><![CDATA[trustedai@substack.com]]></googleplay:owner><googleplay:email><![CDATA[trustedai@substack.com]]></googleplay:email><googleplay:author><![CDATA[Jon Knisley]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Your AI Is Already Regulated. More Rules Start January 1.]]></title><description><![CDATA[How existing law reaches AI today, and the evidence file that answers regulators, plaintiffs and auditors]]></description><link>https://trustedai.recodework.com/p/your-ai-is-already-regulated-more</link><guid isPermaLink="false">https://trustedai.recodework.com/p/your-ai-is-already-regulated-more</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Fri, 20 Nov 2026 18:30:34 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!yMER!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!yMER!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yMER!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg 424w, https://substackcdn.com/image/fetch/$s_!yMER!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg 848w, https://substackcdn.com/image/fetch/$s_!yMER!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!yMER!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!yMER!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg" width="1456" height="1369" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1369,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:314830,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/224968759?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!yMER!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg 424w, https://substackcdn.com/image/fetch/$s_!yMER!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg 848w, https://substackcdn.com/image/fetch/$s_!yMER!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!yMER!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b515ea8-195a-420e-8d3a-908195aabf83_4182x3931.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In July 2025, the Massachusetts attorney general settled with Earnest Operations for $2.5 million. The state alleged that Earnest&#8217;s AI underwriting models disadvantaged Black, Hispanic and non-citizen applicants. The models considered the average default rate at each applicant&#8217;s college, which the state argued led to denials and less favorable terms for Black and Hispanic borrowers. Another rule excluded applicants without at least a green card. Earnest denied the allegations and admitted no fault.</p><p>Massachusetts pursued the case without an AI-specific law, relying instead on the state&#8217;s consumer protection act and the federal Equal Credit Opportunity Act and Regulation B. The state asserted that Earnest did not test its models for disparate impact and that its adverse action notices did not provide applicants with specific reasons for denial.</p><p>The settlement requires Earnest to implement AI governance procedures, conduct annual fair lending tests, remove certain variables from its models, and report compliance to the state.</p><p>While several state AI rules take effect on January 1, most AI enforcement so far has relied on existing consumer protection, civil rights, privacy and professional licensing laws. These laws require no new rulemaking and remain effective while legislatures revise AI statutes and companies challenge them in court.</p><h2>What Changes on January 1</h2><p>California&#8217;s rules have a firm effective date. Under the California Consumer Privacy Act (CCPA), businesses using automated decision technology for significant consumer decisions must comply by January 1. Significant decisions include lending, housing, education, employment and health care. Before using such systems, businesses must provide a plain-language notice explaining the system&#8217;s purpose and the consumer&#8217;s rights. Consumers may opt out, with exceptions that include when a real human appeal is available. They may also request an explanation of how the system reached its result. The rules apply to systems that replace or substantially replace human judgment, so companies retaining human decision-makers may be exempt. Risk assessments for ongoing processing are due by the end of 2027.</p><p>Colorado&#8217;s replacement law also takes effect on January 1, in a simplified form compared to the 2024 statute. The legislature removed the duty of care against algorithmic discrimination, the risk management program and impact assessments. The law now requires notice and recourse. Companies using automated decision technology to materially influence consequential decisions must notify consumers in advance. If a decision is adverse, the company has 30 days to explain the decision and the system&#8217;s role. Consumers may request correction of inaccurate personal data and, where commercially reasonable, meaningful human review. Lenders already providing adverse action notices under federal credit law may satisfy the explanation requirement through those notices.</p><p>The two states set different thresholds. California&#8217;s rules apply to systems that replace or substantially replace human decision-making, so a reviewer with authority to change the outcome may exempt a system. Colorado&#8217;s law covers any system that materially influences a decision, which is a lower threshold. For example, a screening tool whose scores inform a recruiter&#8217;s decision may be outside California&#8217;s rules but within Colorado&#8217;s.</p><p>Enforcement in Colorado remains pending. xAI sued the state in April over the original law, and the Justice Department joined the case, marking its first intervention against a state AI statute. Under an April 27 order, the attorney general agreed not to enforce the law or any replacement, including investigations, until 14 days after the court rules on xAI&#8217;s request for a preliminary injunction. xAI must file this request within 28 days after the attorney general finalizes the implementing rules. The attorney general proposed rules in August, released a revised draft in October, held a hearing on October 26, and must finalize them by January 1. The law takes effect January 1, but enforcement will wait for the final rules and a court decision.</p><p>Other January laws have narrower scope. New York&#8217;s RAISE Act and Illinois SB 315 apply only to developers of the largest frontier models, excluding most companies that purchase or deploy AI. California&#8217;s new Right to Human Customer Service Act is broader. Starting January 1, businesses with over $500 million in national revenue serving California customers may not present a customer service chatbot as a person. They must disclose when a customer is interacting with AI if a reasonable person could think otherwise, and must provide a way for customers to reach a human agent.</p><p>Colorado&#8217;s Chatbot Safety Act, HB 26-1263, also takes effect January 1. It applies to operators of conversational AI services, defined as AI systems open to the public that primarily imitate human conversation in text, visuals, or audio. Operators must disclose the AI nature of the service at a user&#8217;s first interaction each day and whenever requested. For continuous use, the notice must repeat at least every three hours or remain visible. Operators must estimate user ages, implement protections for minors, maintain protocols for suicide and self-harm prompts, and avoid presenting outputs as services from licensed professionals. Reporting to the attorney general begins July 1, 2027. The attorney general enforces the act, with a $5,000 civil penalty per violation, and each non-compliant output counts as a separate violation. The act excludes tools mainly for developers or internal business use, and assistants limited to transactions; draft rules further clarify these exclusions. Companies with public-facing assistants should assess compliance before January. The xAI enforcement order applies to the original AI act and its replacement, not this act.</p><p>Federal actions have not altered the timeline. A December 2025 executive order directed the Justice Department to challenge state AI laws in court, and in March the White House asked Congress to preempt those it deems burdensome. Congress has not acted. The leading bill, the FRONTIER Act, would preempt new state rules on frontier developers regarding catastrophic-risk transparency, third-party audits, and incident reporting, but would preserve generally applicable laws and state rules for companies deploying and using AI. In September, the House Energy and Commerce Committee chair declined to commit to a vote this year. That month, 26 state attorneys general urged Congress to regulate frontier AI and prevent preemption of state AI laws.</p><p>For most companies, the new compliance requirements due in January are straightforward.</p><ul><li><p>Identify all systems that materially influence consequential or significant decisions about California or Colorado residents.</p></li><li><p>Provide the required notice for each system and establish a process for human review.</p></li><li><p>Establish a process to explain adverse outcomes within 30 days and correct underlying data as needed.</p></li><li><p>Request necessary information from each vendor to support required notices. Colorado mandates that developers provide this information.</p></li><li><p>If your revenue exceeds $500 million and you offer a chatbot to California customers, disclose its AI nature wherever a customer might mistake it for a person, and ensure access to a human agent.</p></li><li><p>If you have a public-facing assistant serving Colorado users, review it against the Chatbot Safety Act&#8217;s definitions and exclusions, and comply with the act&#8217;s requirements if applicable.</p><p></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p></li></ul><h2>The Laws Already Working</h2><p>Every state has a statute against unfair or deceptive business practices, allowing attorneys general to take action without an AI-specific law. Attorneys general have used these and related statutes throughout 2026. In July, Troutman Pepper Locke reported that enforcement relied on established consumer protection, licensing, privacy, and advertising laws rather than new AI statutes. The 26 attorneys general who wrote to Congress in September stated they are holding frontier AI labs accountable under existing state law.</p><p>Deceptive practices statutes are only one tool. Attorneys general also enforce civil rights and privacy laws, often launching investigations before clear standards exist. Some states have developed dedicated resources. For example, Pennsylvania established an AI task force in February to prevent chatbots from posing as licensed medical professionals, three months before suing Character.AI.</p><p>Federal regulators have refined their enforcement approach. In December 2025, the FTC set aside its earlier order against the writing tool Rytr because the complaint did not allege that Rytr itself made deceptive claims or that anyone used it to post false reviews. The agency continues to pursue cases involving false claims about AI capabilities. DLA Piper counted 13 such cases since 2024 as of May. In March, the FTC settled with Air AI, alleging it misled small businesses about earnings and refunds for its AI business tools. The order bans Air AI and its owners from marketing business opportunities and imposes an $18 million judgment, mostly suspended due to inability to pay; they owe $50,000. In August, the FTC finalized orders against Cox Media Group and two partner firms, alleging they falsely claimed their &#8220;Active Listening&#8221; product used smart device conversations to target ads. The three firms will pay $930,000 and remain under FTC orders for 20 years.</p><p>General laws are more effective than AI-specific statutes because they do not require determining whether a system qualifies as AI. Deceptive claims to consumers are unlawful regardless of whether they come from a model or a salesperson. These laws cannot be preempted, as the leading federal bill preserves generally applicable state law. Many also allow private plaintiffs to sue, so cases can proceed even if an agency shifts its priorities.</p><h2>Where General Law Reaches AI</h2><p>Recent cases involve four types of exposure, and each corresponds to an area a company can control.</p><h4>What companies claim and what their systems say</h4><p>In September 2024, Texas set a precedent when its attorney general alleged that Pieces Technologies, a Dallas-based generative AI provider for hospitals, violated the state&#8217;s deceptive trade practices law by advertising a critical hallucination rate below 0.001 percent. The state&#8217;s investigation found this figure was likely inaccurate. At least four major Texas hospitals used the company&#8217;s products. Pieces denied wrongdoing and signed an assurance of voluntary compliance without payment. Texas described the agreement as the first of its kind for a generative AI company in healthcare.</p><p>Since then, scrutiny has shifted from marketing materials to statements made directly by AI systems. In May, Pennsylvania&#8217;s State Board of Medicine sued Character Technologies under the Medical Practice Act after a state investigator, posing as a patient, found a chatbot named Emilie that allegedly claimed to be licensed to practice psychiatry in Pennsylvania and provided an invalid license number. The board is seeking an injunction. In June, Florida&#8217;s attorney general filed a ten-count suit against OpenAI and its chief executive, including four counts under the state&#8217;s deceptive and unfair trade practices act, seeking civil penalties of up to $10,000 per willful violation.</p><p>In July, the FTC introduced a new theory. Its proposed policy statement indicated that covertly influencing an AI system&#8217;s answers is likely deceptive, as companies have promised their systems strive for accuracy. The statement also warned that exaggerating a system&#8217;s accuracy can itself be deceptive.</p><p>Negligence law reinforces this principle. In 2024, a Canadian tribunal held Air Canada liable after its website chatbot incorrectly told a customer they could claim a bereavement fare after travel, contrary to company policy. Air Canada argued it was not responsible for the chatbot&#8217;s information, but the tribunal found the chatbot was part of the website and that Air Canada failed to ensure its accuracy.</p><p>For companies deploying AI, liability extends beyond advertising. Both a sales deck promising a specific accuracy rate and a support bot providing incorrect refund terms are company statements, and regulators will require substantiation for each.</p><h4>Who AI systems screen out</h4><p>The hiring case drawing the most attention is Mobley v. Workday. Derek Mobley alleges that Workday&#8217;s AI screening tools rejected applicants based on age, race, and disability. In May 2025, the court allowed his age discrimination claim to proceed as a nationwide collective action. In June, Judge Rita Lin largely denied Workday&#8217;s motion to dismiss newer claims and allowed claims under California&#8217;s Fair Employment and Housing Act to proceed, since plaintiffs allege Workday designs and operates the tools from California. The ruling finds the allegations sufficient to proceed, but that reasoning could extend California law to out-of-state applicants. Workday denies the claims and asserts its customers control hiring decisions, but the court has not accepted this as grounds to dismiss Workday from the case.</p><p>The federal government has deprioritized enforcement based on disparate impact, following an April 2025 executive order instructing agencies to focus less on practices that disproportionately affect protected groups. While this shifts agency priorities, underlying statutes and private lawsuits remain. In contrast, states have strengthened enforcement. New Jersey&#8217;s December 2025 rules codify disparate impact liability under its Law Against Discrimination and explicitly cover automated employment decision tools. Companies using vendor systems must take reasonable steps to ensure compliance. The Massachusetts settlement with Earnest relied on the same theory.</p><h4>What AI systems hear</h4><p>California&#8217;s Invasion of Privacy Act, enacted in 1967, requires consent from all parties before recording or intercepting a conversation and allows statutory damages of $5,000 per violation. Plaintiffs now invoke this law against AI systems that listen. In August, a federal judge allowed wiretap, privacy, and Illinois biometric claims to proceed against Otter.ai, whose notetaker joins and transcribes video meetings. Plaintiffs alleged Otter retained recordings and used them to train its models, making it plausible Otter acted as an outside eavesdropper rather than a tool for the meeting host. In October, a federal judge dismissed claims by Home Depot callers regarding Google&#8217;s AI customer service product, finding they did not plausibly allege Google listened without consent, and sent three other callers to arbitration. The Home Depot callers had until November 12 to amend their claims.</p><p>Privacy law also covers data used to train AI models. In January 2025, Texas sued Allstate and its analytics subsidiary Arity under the state&#8217;s comprehensive privacy law, marking the first such suit by a state attorney general. The state alleges the companies collected driving data from over 45 million people through embedded software in other apps, without notice or consent, and used it for insurance pricing.</p><p>Recent rulings focus on whether individuals were aware an AI system was listening and whether the vendor used recordings for its own purposes. Companies can address both matters proactively through clear disclosures and contract terms.</p><h4>What companies tell investors</h4><p>Securities law also applies to public companies&#8217; AI-related claims. The SEC brought its first AI-washing cases in March 2024 against investment advisers Delphia and Global Predictions, and its first case against a public company, Presto Automation, in January 2025. The SEC&#8217;s fiscal 2027 examination priorities, released October 23, instruct examiners to review how investment advisers and broker-dealers use AI. On October 30, the SEC&#8217;s enforcement director stated the division will examine public companies that may overstate AI capabilities or omit material information about AI use or risks. Shareholders have filed several securities fraud suits this year against companies and their officers over AI claims.</p><p>An earnings call describing a product as AI-driven requires the same substantiation as a marketing claim, and officers making such statements may be named in lawsuits.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/your-ai-is-already-regulated-more?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/your-ai-is-already-regulated-more?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Govern the Decisions and the Claims</h2><p>Most AI inventories list models and systems, but the cases above hinge on four additional factors. An effective inventory for managing exposure should record these for every AI in use.</p><ul><li><p><strong>The decision the AI influences.</strong>&nbsp;Lending, hiring, housing, insurance, health care, and education face the most scrutiny under civil rights law and new California and Colorado regulations. A tool that drafts marketing copy presents different risks than one that ranks job applicants, even if both use the same model.</p></li><li><p><strong>The claims made about the AI.</strong>&nbsp;Every statement about accuracy or human review matters, wherever it appears, including websites, sales decks, customer contracts, earnings calls, and filings.</p></li><li><p><strong>The data the AI collects.</strong>&nbsp;The inventory should record what the system gathers, from whom, with what notice, and whether a vendor retains or uses it for training.</p></li><li><p><strong>Responsibility for statements.</strong>&nbsp;Each customer-facing system should have a designated owner for its statements, similar to how an individual owns the content of a published policy.</p></li></ul><p>Each points to a control that can be implemented.</p><p>Claims require supporting documentation. This file should include the test behind each figure, the date conducted, the system version, and be updated by the owner when the model changes. The FTC&#8217;s orders against Cox Media Group illustrate the consequences of unsupported claims. The agency alleged the firms sold an AI listening product that did not perform as advertised, and the orders prohibit misrepresenting their services.</p><p>Outcomes require scheduled testing. Companies using AI for consequential decisions should compare results across protected groups before launch and after any significant change. The Massachusetts settlement requires an annual fair lending test. Documentation of changes made in response is as important as the test results themselves.</p><p>Consent is required before recording begins. This involves disclosing at the start of a call or meeting that AI recording or analysis will occur, and specifying in contracts whether the vendor may retain recordings or use them for its own models. The Otter and Google rulings differed on these issues.</p><p>Vendor contracts should specify who is responsible for testing the system and obtaining consent. These provisions do not transfer liability. New Jersey&#8217;s rules hold companies using vendor systems responsible for taking reasonable steps to comply, and the Workday case shows that vendors can remain parties to lawsuits alongside their customers. Colorado&#8217;s new law requires developers to provide deployers with necessary information for notices, which should be detailed in the contract.</p><p>Customer-facing output requires monitoring. A designated reviewer should regularly check chatbot communications about prices, refunds, eligibility, and qualifications, with a prompt process for corrections. Pennsylvania&#8217;s investigator identified the chatbot in question by searching the platform for &#8220;psychiatry,&#8221; as any user could.</p><p>These controls are not solely the AI team's responsibility. Marketing and legal own the claims. The business owner and compliance manage outcome testing, while the privacy team handles consent, and procurement manages vendor terms. AI governance ensures each responsibility is clearly assigned.</p><p>The same records address most January rule requirements. The decision inventory can help prepare a California pre-use notice. Colorado&#8217;s 30-day explanation requires documentation of how the system reached its result, and the required human review is consistent with the process for discrimination claims. Companies maintaining these records can meet new state laws by preparing notices from existing information.</p><h2>One Lender, Four Exposures</h2><p>Consider a consumer lender with approximately $2 billion in annual revenue and customers in California and Colorado. Its governance committee has approved four AI applications in the past two years. The company and figures are illustrative.</p><p>Begin with the resume screener purchased from an HR software vendor, which ranks applicants for underwriting roles. This tool influences hiring decisions, invoking federal and state discrimination laws, as well as new California and Colorado rules for applicants. The vendor claims to have validated the tool, as reflected in the contract. However, this validation still holds the lender accountable for its hiring practices, and the Workday case demonstrates that vendors may also be implicated. The lender must conduct dated tests comparing pass rates by age and race before launch and after each vendor update, and provide the required notices for both states. No current notice regime covers this use, and hiring remains the area with the most private AI litigation.</p><p>The credit model is more established. Lenders already issue adverse action notices under federal credit law, and Colorado accepts these notices to fulfill its explanation requirement. California&#8217;s rules, along with state fair lending and consumer protection laws, also apply. The Massachusetts case illustrates the implications. The lender must run outcome tests by group, document variables that influence denials, and ensure adverse action notices accurately reflect the model&#8217;s decisions.</p><p>The voice agent answers customer calls, summarizes them, and suggests next steps to staff handling complex cases. Because it records calls, California&#8217;s wiretap law applies to calls with Californians. With revenue exceeding $500 million, the lender also falls under California&#8217;s new chatbot law. The agent must disclose it is AI when callers might mistake it for a person and must provide access to a human representative. The vendor&#8217;s standard terms permit using call recordings to improve its models, as in the Otter case. The lender needs a disclosure script, an amended contract, and records confirming callers reached a human upon request.</p><p>Finally, the website states that the lender&#8217;s AI approves loans in minutes and that an expert reviews every decision. Regulators consider both statements as factual claims. The second statement matters under California&#8217;s rules, which depend on the extent of human involvement. The lender must provide data on approvals completed within minutes and a signed description from the responsible individual detailing the expert review process.</p><h2>The Evidence You Should Expect to Provide</h2><p>Eventually, a company will be asked to demonstrate how its AI operates. This request may come from a state attorney general, a plaintiff&#8217;s lawyer, an auditor or a large customer&#8217;s due diligence team. These requests often overlap. A company that can provide the following materials within a week will address most inquiries, regardless of the requester.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!cZXO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cZXO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png 424w, https://substackcdn.com/image/fetch/$s_!cZXO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png 848w, https://substackcdn.com/image/fetch/$s_!cZXO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png 1272w, https://substackcdn.com/image/fetch/$s_!cZXO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!cZXO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png" width="1456" height="858" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:858,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:181048,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/224968759?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!cZXO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png 424w, https://substackcdn.com/image/fetch/$s_!cZXO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png 848w, https://substackcdn.com/image/fetch/$s_!cZXO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png 1272w, https://substackcdn.com/image/fetch/$s_!cZXO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09ca6724-4107-459c-a36b-c6e7820670d4_2220x1308.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Companies should start preparing these materials before January 1. The file will satisfy most requirements under the new California and Colorado rules and address initial questions in any post-complaint inquiry.</p><h2>The Risk in Waiting</h2><p>During the winter, attention will focus on the January deadlines and the preemption debate. These issues may persist for years. Companies that view them as the primary AI legal risks may delay necessary work until the statutes are resolved.</p><p>However, enforcement trends suggest otherwise. Recent cases have relied on a Massachusetts consumer protection statute, a Pennsylvania medical licensing law, federal and California employment laws, and a 1967 wiretap statute. The leading preemption bill in Congress does not affect these laws. They apply to any company using AI in decisions about individuals or in customer interactions.</p><p>A company unable to demonstrate what its systems decide or communicate, or the basis for its claims, faces legal exposure now, regardless of future legislative or judicial outcomes.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/your-ai-is-already-regulated-more/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/your-ai-is-already-regulated-more/comments"><span>Leave a comment</span></a></p><div class="callout-block" data-callout="true"><p>Trusted AI is written for executives, board members and senior practitioners building, funding or overseeing AI programs. Have a topic you want covered? Reply to any issue or reach out on LinkedIn.</p></div><h2>Sources</h2><ul><li><p><a href="https://www.mass.gov/news/ag-campbell-announces-25-million-settlement-with-student-loan-lender-for-unlawful-practices-through-ai-use-other-consumer-protection-violations">Massachusetts Attorney General, AG Campbell Announces $2.5 Million Settlement With Student Loan Lender for Unlawful Practices Through AI Use</a></p></li><li><p><a href="https://bankingjournal.aba.com/2025/08/mass-ag-reaches-settlement-with-earnest-operations-for-2-5m-over-ai-lending-bias/">ABA Banking Journal, Mass. AG Reaches Settlement With Earnest Operations Over AI Lending Bias</a></p></li><li><p><a href="https://todaysgeneralcounsel.com/?p=95641">Today&#8217;s General Counsel, Massachusetts Settles AI Bias Case With Student Loan Firm</a></p></li><li><p><a href="https://www.cppa.ca.gov/meetings/materials/adt_regulations.pdf">California Privacy Protection Agency, Automated Decisionmaking Technology Regulations Overview</a></p></li><li><p><a href="https://www.alston.com/en/insights/publications/2025/10/ccpa-cybersecurity-audits-admt-risk-assessments">Alston &amp; Bird, CCPA Cybersecurity Audits, ADMT and Risk Assessments</a></p></li><li><p><a href="https://secureprivacy.ai/blog/cpra-admt-regulations-2026-what-california-businesses-must-do">Secure Privacy, CPRA ADMT Regulations: What California Businesses Must Do</a></p></li><li><p><a href="https://www.finnegan.com/en/insights/articles/colorado-replaces-landmark-ai-act-an-overview-of-the-new-sb-26-189-framework.html">Finnegan, Colorado Replaces Landmark AI Act: An Overview of the New SB 26-189 Framework</a></p></li><li><p><a href="https://www.crowell.com/en/insights/client-alerts/colorado-hits-reset-on-ai-regulation-sb-26-189-repeals-and-reenacts-the-colorado-ai-act">Crowell &amp; Moring, Colorado Hits Reset on AI Regulation</a></p></li><li><p><a href="https://www.mcdermottlaw.com/insights/colorado-ai-law-in-flux-comprehensive-replacement-bill-signed-after-federal-court-blocks-predecessors-enforcement/">McDermott Will &amp; Schulte, Colorado AI Law in Flux</a></p></li><li><p><a href="https://www.nortonrosefulbright.com/en/knowledge/publications/de3ad9de/xai-sues-doj-intervenes-enforcement-of-colorado-ai-act-suspended">Norton Rose Fulbright, X.AI Sues, DOJ Intervenes, Enforcement of Colorado&#8217;s AI Act Suspended</a></p></li><li><p><a href="https://www.jenner.com/en/news-insights/client-alerts/blue-wave-will-reshape-state-tech-and-ai-regulation-in-2027">Jenner &amp; Block, State Tech and AI Regulation in 2027</a></p></li><li><p><a href="https://calawyers.org/?p=1295179">California Lawyers Association, Privacy Law Legislative Update</a></p></li><li><p><a href="https://www.quarles.com/newsroom/publications/please-hold-for-no-longer-than-fifteen-minutes-new-customer-service-requirements-for-ai-chatbots">Quarles, New Customer Service Requirements for AI Chatbots</a></p></li><li><p><a href="https://gc.ai/blog/state-ai-laws">GC AI, State AI Laws in 2026: SB 53, SB 942, and Preemption</a></p></li><li><p><a href="https://www.mlex.com/employment-authority/labor/articles/2455662/white-house-pushes-congress-to-override-state-ai-laws">MLex, White House Pushes Congress to Override State AI Laws</a></p></li><li><p><a href="https://www.theregreview.org/2026/10/07/knapp-the-hidden-costs-of-ai-preemption/">The Regulatory Review, The Hidden Costs of AI Preemption</a></p></li><li><p><a href="https://www.regulatoryoversight.com/2026/10/26-state-ags-urge-congress-to-regulate-frontier-ai-and-preserve-state-authority/">Troutman Pepper Locke, 26 State AGs Urge Congress to Regulate Frontier AI and Preserve State Authority</a></p></li><li><p><a href="https://www.pymnts.com/news/artificial-intelligence/2026/state-attorney-general-expand-ai-enforcement-using-traditional-consumer-protection-laws/">PYMNTS, State AGs Expand AI Enforcement Using Traditional Consumer Protection Laws</a></p></li><li><p><a href="https://ailawsuittracker.com/state-ai-enforcement/">AI Lawsuit Tracker, State AI Enforcement</a></p></li><li><p><a href="https://www.hklaw.com/en/insights/media-entities/2026/09/the-federal-trade-commissions-current-approach-to-ai-regulation">Holland &amp; Knight, The Federal Trade Commission&#8217;s Current Approach to AI Regulation</a></p></li><li><p><a href="https://www.troutman.com/insights/false-advertising-liability-ftc-scrutiny-and-competitive-litigation-in-the-age-of-ai-marketing.html">Troutman Pepper Locke, False Advertising Liability, FTC Scrutiny, and Competitive Litigation in the Age of AI Marketing</a></p></li><li><p><a href="https://www.ftc.gov/news-events/news/press-releases/2026/03/air-ai-its-owners-will-be-banned-marketing-business-opportunities-settle-ftc-charges-company-misled">Federal Trade Commission, AI and Its Owners Will Be Banned From Marketing Business Opportunities</a></p></li><li><p><a href="https://www.ftc.gov/news-events/news/press-releases/2026/08/ftc-finalizes-orders-cox-media-group-two-other-firms-settling-charges-they-deceived-customers-about">Federal Trade Commission, FTC Finalizes Orders With Cox Media Group and Two Other Firms</a></p></li><li><p><a href="https://ppc.land/cox-media-group-faces-20-years-of-ftc-oversight-over-fake-ai-listening-ads/">PPC Land, Cox Media Group Faces 20 Years of FTC Oversight</a></p></li><li><p><a href="https://tech-insider.org/?p=18893">Tech Insider, FTC Fines Cox Media $930K, Imposes 20-Year AI Oversight</a></p></li><li><p><a href="https://www.aipolicydesk.com/blog/ftc-ai-enforcement-actions-2026">AI Policy Desk, FTC AI Enforcement Actions 2026</a></p></li><li><p><a href="https://www.dechert.com/knowledge/re-torts/2026/6/Pennsylvania-Sues-Character-ai-Over-Chatbot-That-Posed-as-Licensed-Psychiatrist.html">Dechert, Pennsylvania Sues Character.ai Over Chatbot That Posed as Licensed Psychiatrist</a></p></li><li><p><a href="https://amp.insurancejournal.com/news/east/2026/05/06/868729.htm">Insurance Journal, Pennsylvania Sues Character AI, Says Chatbot Poses as Doctors</a></p></li><li><p><a href="https://ppc.land/pennsylvania-sues-character-ai-for-ai-bot-posing-as-licensed-psychiatrist/">PPC Land, Pennsylvania Sues Character.AI for AI Bot Posing as Licensed Psychiatrist</a></p></li><li><p><a href="https://www.washingtonexaminer.com/news/justice/4589444/florida-ag-sues-openai-sam-altman-deceptive-endanger-people/">Washington Examiner, Florida Attorney General Sues OpenAI and Sam Altman</a></p></li><li><p><a href="https://ppc.land/florida-sues-openai-and-sam-altman-over-chatgpt-public-safety-risks/">PPC Land, Florida Sues OpenAI and Sam Altman Over ChatGPT Public Safety Risks</a></p></li><li><p><a href="https://www.spencerfane.com/?p=42114">Spencer Fane, FTC Proposes New Policy on AI Accuracy</a></p></li><li><p><a href="https://www.fisherphillips.com/print/v2/content/41157/discrimination-lawsuit-over-workdays-ai-hiring-tools-can-proceed-as-class-action-6-things.pdf">Fisher Phillips, Discrimination Lawsuit Over Workday&#8217;s AI Hiring Tools Can Proceed as Class Action</a></p></li><li><p><a href="https://www.mlex.com/employment-authority/wage-hour/articles/2492344/workday-can-t-knock-calif-law-claims-out-of-ai-bias-suit">Law360 via MLex, Workday Can&#8217;t Knock Calif. Law Claims Out of AI Bias Suit</a></p></li><li><p><a href="https://www.fmglaw.com/?p=42012">Freeman Mathis &amp; Gary, Mobley v. Workday Ruling</a></p></li><li><p><a href="https://www.seyfarth.com/news-insights/new-executive-order-directs-federal-agencies-to-deprioritize-disparate-impact-what-employers-need-to-know-now.html">Seyfarth, New Executive Order Directs Federal Agencies to Deprioritize Disparate Impact</a></p></li><li><p><a href="https://www.consumerfinancialserviceslawmonitor.com/2025/12/new-jersey-adopts-disparate-impact-rules-under-lad-with-broad-reach-across-housing-lending-employment-and-other-fields-with-specific-guidance-on-ai/">Consumer Financial Services Law Monitor, New Jersey Adopts Disparate Impact Rules Under LAD</a></p></li><li><p><a href="https://www.hklaw.com/en/insights/publications/2026/01/nj-attorney-general-preserves-disparate-impact-theory-of-lending">Holland &amp; Knight, NJ Attorney General Preserves Disparate Impact Theory of Lending</a></p></li><li><p><a href="https://consentpixel.com/?p=6978">ConsentPixel, CIPA&#8217;s Expanding Frontier: From Phone Taps to Pixels to AI</a></p></li><li><p><a href="https://www.bairdholm.com/blog/core-privacy-claims-against-ai-note-taking-platform-survive-dismissal/">Baird Holm, Core Privacy Claims Against AI Note Taking Platform Survive Dismissal</a></p></li><li><p><a href="https://www.sheppard.com/insights/blogs/when-ai-takes-notes-court-allows-privacy-claims-against-otterai-to-proceed">Sheppard Mullin, When AI Takes Notes: Court Allows Privacy Claims Against Otter.ai to Proceed</a></p></li><li><p><a href="https://techjacksolutions.com/?p=99652">TechJack Solutions, Federal Judge Dismisses CIPA Claims Against Google&#8217;s AI Customer Service Product</a></p></li><li><p><a href="https://www.texasattorneygeneral.gov/news/releases/attorney-general-ken-paxton-sues-allstate-and-arity-unlawfully-collecting-using-and-selling-over-45">Texas Attorney General, Attorney General Ken Paxton Sues Allstate and Arity</a></p></li><li><p><a href="https://www.wilmerhale.com/en/insights/blogs/wilmerhale-privacy-and-cybersecurity-law/20250121-texas-ag-brings-first-ever-lawsuit-under-a-state-comprehensive-privacy-law">WilmerHale, Texas AG Brings First Ever Lawsuit Under a State Comprehensive Privacy Law</a></p></li><li><p><a href="https://blog.promise.legal/sec-ai-disclosure-10k-filings/">Promise Legal, SEC AI Disclosures in 10-K Filings</a></p></li><li><p><a href="https://www.dandodiary.com/?p=28795">The D&amp;O Diary, AI, the SEC and the 2026 Reporting Season</a></p></li><li><p><a href="https://www.debevoise.com/insights/publications/2026/11/sec-enforcement-director-signals-continued-scrutin">Debevoise &amp; Plimpton, SEC Enforcement Director Signals Continued Scrutiny of Corporate AI Disclosures</a></p></li><li><p><a href="https://www.hklaw.com/en/insights/publications/2026/08/operation-ai-comply-2-years-later-continued-enforcement">Holland &amp; Knight, Operation AI Comply 2 Years Later</a></p></li></ul><ul><li><p><a href="https://www.mccarthy.ca/en/insights/blogs/techlex/moffatt-v-air-canada-misrepresentation-ai-chatbot">McCarthy T&#233;trault, Moffatt v. Air Canada: A Misrepresentation by an AI Chatbot</a></p></li><li><p><a href="https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/">American Bar Association, BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot</a></p></li><li><p><a href="https://www.procopio.com/resource/california-admt-privacy-regulations">Procopio, California ADMT Privacy Regulations</a></p></li><li><p><a href="https://www.hunton.com/privacy-and-information-security-law/cppa-finalizes-ccpa-regulations-on-automated-decision-making-technology-risk-assessments-and-cybersecurity-audits">Hunton Andrews Kurth, CPPA Finalizes CCPA Regulations on Automated Decision-Making Technology</a></p></li><li><p><a href="https://www.beneschlaw.com/insight/where-ai-regulation-stands-today-part-2-state-attorneys-general-as-the-primary-ai-enforcers/">Benesch, State Attorneys General as the Primary AI Enforcers</a></p></li><li><p><a href="https://decrypt.co/366833/pennsylvania-sues-character-ai-chatbot-posing-licensed-psychiatrist">Decrypt, Pennsylvania Sues Character.AI Over Chatbot Posing as Licensed Psychiatrist</a></p></li><li><p><a href="https://www.ftc.gov/system/files/ftc_gov/pdf/Rytr-Order.pdf">Federal Trade Commission, Order Reopening and Setting Aside the Rytr Decision and Order</a></p></li><li><p><a href="https://www.dlapiper.com/en-us/insights/publications/2026/05/ftc-ai-washing-action-underscores-enforcement-in-business-to-business-context">DLA Piper, FTC AI-Washing Action Underscores Enforcement in the Business-to-Business Context</a></p></li><li><p><a href="https://coag.gov/ai/">Colorado Attorney General, Automated Decision-Making Technology and Chatbot Safety Rulemaking</a></p></li><li><p><a href="https://therecord.media/frontier-act-ai-bill-house-brett-guthrie">The Record, Guthrie on the FRONTIER Act</a></p></li><li><p><a href="https://ag.ny.gov/sites/default/files/letters/federal-ai-regulation-letter-2026.pdf">New York Attorney General, Letter to Congressional Leadership on Federal AI Regulation</a></p></li><li><p><a href="https://www.fiercehealthcare.com/ai-and-machine-learning/texas-ag-pieces-technologies-settle-allegations-inaccurate-generative-ai">Fierce Healthcare, Texas Attorney General and Pieces Technologies Settle Allegations of Inaccurate Generative AI Advertising</a></p></li><li><p><a href="https://blogs.duanemorris.com/classactiondefense/2026/06/24/california-federal-court-grants-in-part-and-denies-in-part-workdays-motion-to-dismiss-in-mobley-v-workday/">Duane Morris, California Federal Court Grants in Part and Denies in Part Workday&#8217;s Motion to Dismiss</a></p></li><li><p><a href="https://openclassactions.com/news/google-contact-center-ai-wiretap-lawsuit-dismissed.php">Open Class Actions, Google AI Call-Center Lawsuit Dismissed</a></p></li><li><p><a href="https://www.mayerbrown.com/en/insights/publications/2026/11/sec-division-of-examinations-announces-fiscal-year-2027-exam-priorities">Mayer Brown, SEC Division of Examinations Announces Fiscal Year 2027 Exam Priorities</a></p></li><li><p><a href="https://regulations.ai/regulations/RAI-US-CA-AB16090-2026">Regulations.ai, California Customer Service Chatbot Law (AB 1609)</a></p></li><li><p><a href="https://leg.colorado.gov/bills/HB26-1263">Colorado General Assembly, HB26-1263 Conversational Artificial Intelligence Service Operator Requirements</a></p></li><li><p><a href="https://policyrisk.com/state-bill/CO-HB26-1263-2026A">Policy Risk, Colorado HB26-1263 Analysis</a></p></li><li><p><a href="https://www.wiley.law/alert-Colorado-Proposes-Detailed-Rules-Implementing-New-ADMT-and-Chatbot-Laws">Wiley, Colorado Proposes Detailed Rules Implementing New ADMT and Chatbot Laws</a></p></li><li><p><a href="https://www.joneswalker.com/en/insights/blogs/ai-law-blog/the-colorado-ai-law-you-prepared-for-is-gone-and-the-rulemaking-comment-window-c.html">Jones Walker, The Colorado AI Law You Prepared For Is Gone</a></p></li></ul>]]></content:encoded></item><item><title><![CDATA[The Metric Is Not Cost Per Token]]></title><description><![CDATA[AI unit economics and what you pay for each task it gets right]]></description><link>https://trustedai.recodework.com/p/the-metric-is-not-cost-per-token</link><guid isPermaLink="false">https://trustedai.recodework.com/p/the-metric-is-not-cost-per-token</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Sat, 31 Oct 2026 21:34:37 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!kOzq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kOzq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kOzq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg 424w, https://substackcdn.com/image/fetch/$s_!kOzq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg 848w, https://substackcdn.com/image/fetch/$s_!kOzq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!kOzq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!kOzq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1719818,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/222950879?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!kOzq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg 424w, https://substackcdn.com/image/fetch/$s_!kOzq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg 848w, https://substackcdn.com/image/fetch/$s_!kOzq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!kOzq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0245dc3-6faa-4cdf-979d-02cf1bb155e9_6000x4000.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In April, Uber reported that they'd already blown their 2026 AI coding tool budget, just four months into the year. The company actively encouraged engineers to use the tools and had leaderboards to track usage. By May, they struggled to link spend to new product features. By June, they set a $1,500-per-employee-per-month cap on each agentic coding tool, introduced a spend dashboard, and required approval to go above it.</p><p>Microsoft implemented similar measures in August. Leadership told engineers that each division now operates under a token budget, with each engineer's spending visible. Guidance indicated that engineers typically spend a few hundred to several thousand dollars per month on tokens. The company sought &#8220;more impact per token.&#8221;</p><p>Both companies have large, skilled engineering teams and can report token spending precisely. However, their reports do not show the cost or return of a completed task. Because vendors bill by token, most AI cost reports focus on tokens, which only measure input to a call. Systems track outcomes separately for tickets, claims, orders, and payments, and few reports connect costs to results.</p><p>The gap widens as agents handle more tasks. A low-cost call that fails and requires retries or human intervention can ultimately cost more than a higher-priced call that succeeds on the first attempt, yet token-based reports may favor the former. The more accurate metric is the total cost per successful task, including all associated expenses. Calculating this requires a full cost breakdown and a clear definition of success, which often changes how alternatives are ranked.</p><h2>Why the Bill Keeps Rising</h2><p>The cost of a fixed level of AI continues to decline. Gartner projects that by 2030, inference on a trillion-parameter model will cost providers over 90% less than in 2025. However, providers are expected to pass on only part of these savings, and total inference spending is still expected to increase.</p><p>Increased volume explains this gap. Gartner estimates that agents use 5 to 30 times more tokens than chatbots for the same task and expects agent workflow inference costs to rise more than fivefold by 2028. Agents plan, call tools, review results, self-check, and retry, sending larger contexts through the model at each step. Reasoning models also add tokens for their internal processing, which providers bill as well.</p><p>McKinsey&#8217;s research on agentic coding finds that about 60% of an agent task&#8217;s cost comes from refining answers through repeated checking and correction. Other cost drivers include context accumulation during long sessions, defaulting simple tasks to advanced models, inconsistent step counts due to vague instructions, and additional calls from orchestration when agents transfer work.</p><p>Token price alone does not reflect these factors. Even if the rate drops by half in a year, monthly invoices can double if each task consumes more tokens. Microsoft and Uber experienced this in their engineering teams, where coding agents run extended sessions on large codebases and engineers often run multiple agents simultaneously.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>The Forecast Starts in the Pilot</h2><p>Most AI budgets begin with a pilot, typically using a few hundred selected cases and often funded by vendor credits, with engineers closely monitoring each run. In production, the system faces higher volumes, more complex inputs, retries, longer contexts, safety checks on every call, and large-scale monitoring. These conditions don't exist during the pilot.</p><p>McKinsey&#8217;s research quantifies this increase. In a May 2026 survey, AI spending rose nearly fourfold as companies expanded from single use cases to broader adoption, and 93% of respondents reported exceeding their AI budgets. One bank&#8217;s projected AI spend grew from $4 million dollars annually to over $25 million.</p><p>Finance teams see these impacts in their forecasts. In an April 2026 Mavvrik study, only 11% could forecast AI spend within 10%, down from 15% the previous year. For 62%, unexpected AI costs influenced business decisions, 40% escalated cost surprises to the board, and 25% delayed or canceled AI projects.</p><p>Forecasts based on pilot token counts reflect pilot conditions, not production realities. A reliable forecast starts with the cost per completed task on live traffic, multiplied by expected task volume, and includes all relevant costs.</p><h2>The Rest of the Bill</h2><p>The token invoice is the only clearly labeled AI cost, but it represents just a fraction of total system expenses.</p><p>A working AI system pulls data through pipelines and stores it in a vector database. It incurs transfer fees to move data between clouds and regions. It runs behind a gateway and orchestration layer, sends calls to a second model for evaluation and safety checks, and writes a trace for each step. People review a sample of its work and complete the cases it fails. Engineers keep its prompts and tools current as models change.</p><p>These costs often surprise finance teams more than token charges. In a 2025 survey by Mavvrik, 56% cited data platforms, and 52% cited network charges as sources of AI cost surprises. Model token and API charges ranked fifth at 37%. </p><p>Monitoring also incurs high costs. Cisco&#8217;s 2026 State of Observability survey found that 57% of companies now spend more on telemetry than on the systems being monitored. Agents contribute by generating many steps per task, with full prompts and outputs increasing data volume. Teams can reduce costs by keeping concise, structured records of key actions and capturing full content only for sample runs, preserving necessary evidence while lowering expenses.</p><p>Some AI-related expenses are not tracked as AI costs. McKinsey estimates that 20% to 30% of AI spending is unaccounted for, often hidden in software subscriptions with bundled AI features. Mavvrik found that only 42% of engineering organizations include developer AI tools in their cost reports.</p><p>In systems where people handle failed cases, human labor is often the largest expense. Each AI failure requires manual completion, and McKinsey includes this rework in agent operating costs. In the billing desk example, the cost of resolving failed tickets is 10x the token bill.</p><h2>Cost Per Successful Outcome</h2><p>The key metric is total cost across all attempts in a given period, divided by the number of outcomes that meet the quality standard. McKinsey&#8217;s July research refers to this as &#8220;the completed business outcome.&#8221;</p><p>The numerator includes everything used to reach the outcome.</p><ul><li><p>Tokens for every attempt, including retries and reasoning</p></li><li><p>Retrieval, storage and network charges</p></li><li><p>Evaluation and safety calls, plus monitoring</p></li><li><p>A share of the engineering time that keeps the system current</p></li><li><p>Human review, and the human time spent finishing what the AI failed</p></li></ul><p>The denominator includes only outcomes that meet the quality bar. For support tickets, this means the billing system reflects the change and the customer does not return with the same issue within a set period, such as seven days. An insurance claim must pay the correct amount and pass audit. The system should generate outcome labels such as resolved, escalated, abandoned, or failed at the end of each run. Since agents may report success on work that later fails, outcome labels should not rely solely on agent reports.</p><p>The quality bar ensures the metric remains accurate. Klarna shifted most support chats to an AI assistant in 2024 and reported significant savings. But in May 2025, the company told Bloomberg that cost had become &#8220;a too predominant evaluation factor,&#8221; which reduced quality and led to renewed hiring of support staff. Measuring only lasting resolutions aligns quality and cost, so a declining resolution rate increases the cost per successful outcome.</p><p>Research on agents supports this approach. A Princeton study found that coding agents with similar accuracy could differ in cost by nearly two orders of magnitude. It recommended reporting cost alongside accuracy and optimizing both together in agent evaluations.</p><p>Vendors are beginning to price based on outcomes. Fin, the support agent from the company formerly known as Intercom, charges $0.99 per outcome. In June 2026, Salesforce introduced a help agent that bills $2 per resolution and nothing for unresolved contacts. Because each vendor defines resolution differently, buyers should establish their own definitions and data for invoice verification before contracting.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/the-metric-is-not-cost-per-token?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/the-metric-is-not-cost-per-token?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>One Billing Desk, Three Designs</h2><p>Consider a software company whose finance team processes 100,000 tickets monthly. About 60% are routine tasks like invoice copies and plan changes, while the remaining 40% involve more complex issues such as disputed charges and proration errors across multiple accounts. Before AI, the company handled each ticket manually at a loaded cost of $8, totaling $800,000 per month.</p><p>The company tested three designs.</p><ul><li><p>Top model on every ticket</p></li><li><p>Cheaper model on every ticket</p></li><li><p>A router that sent routine tickets to the cheaper model and hard ones to the top model</p></li></ul><p>The following figures are illustrative. They assume list prices of $5 and $25 per million input and output tokens for the top model, and $1 and $5 for the cheaper model, with half of input tokens served from cache at one-tenth the price. The cheaper model requires more tokens and retries, especially for complex tickets, resolving only half as many as the top model. Each failed AI ticket is handled manually at $8. Additional AI costs, including retrieval, tracing, evaluation, quality review, and engineering upkeep, total approximately $38,000 per month per design.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gKVX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gKVX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png 424w, https://substackcdn.com/image/fetch/$s_!gKVX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png 848w, https://substackcdn.com/image/fetch/$s_!gKVX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png 1272w, https://substackcdn.com/image/fetch/$s_!gKVX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gKVX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png" width="1456" height="542" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:542,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:218477,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/222950879?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!gKVX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png 424w, https://substackcdn.com/image/fetch/$s_!gKVX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png 848w, https://substackcdn.com/image/fetch/$s_!gKVX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png 1272w, https://substackcdn.com/image/fetch/$s_!gKVX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d802563-79a2-4509-903a-d998019c7cd2_1768x658.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Each approach costs less than half of the previous $800,000 monthly spend. Token price rankings are the reverse of cost per resolved ticket. Design B appears most efficient by token report, at $0.84 per million tokens and a $7,300 token bill, but uses nearly twice as many tokens as Design A (8.7 billion versus 4.4 billion) due to more retries on complex tickets. Design B resolves 13,200 fewer tickets monthly than Design A, requiring manual intervention that adds about $105,600 per month, over five times Design A&#8217;s total token bill.</p><p>The token bill represents a small portion of total costs in every design. In Design A, it accounts for less than 8% of the monthly total. Other AI costs are twice as high, and manually resolving failed tickets costs 10x more.</p><p>Partial cost measures can be misleading. If a team divides only AI costs by AI resolutions and excludes manual rework, the results are about $0.70 per resolution for Design C, $0.72 for Design B, and $0.74 for Design A. This approach favors the router and the cheaper model over the top model.</p><p>The decision between Design A and Design C depends on the cheaper model&#8217;s resolution rate for routine tickets. At 86% versus 88% for the top model, the router costs about $5,600 more per month. At 87%, costs break even. If the cheaper model matches the top model on routine tasks, the router saves about $3,900 per month. Determining this requires a month of outcome labels on live traffic, segmented by ticket type.</p><h2>The Controls and the Data Behind Them</h2><p>McKinsey reports that about one-third of surveyed companies have reduced AI spending by 20% to 30% through active tuning. These methods rely on data that links each call to its cost and outcome, a connection most companies still haven't established.</p><h4>Tie every call to an owner and an outcome</h4><p>Tag each model call with the task, system, owning team, and the outcome label set at the end of the run. OpenTelemetry conventions for generative AI already include token counts and model names, so most additional effort involves business-specific fields. These fields allow cost reports to attribute costs to specific tasks.</p><h4>Route by difficulty</h4><p>Assign each task to the least expensive model that meets its quality standard. The RouteLLM study showed a trained router reduced costs by over half while maintaining about 95% of the stronger model&#8217;s quality on a standard benchmark. </p><p>Gartner also advises using smaller models for routine tasks and reserving advanced models for complex reasoning. In the billing desk example, the router was cost-effective only when the cheaper model resolved routine tickets at rates close to the top model, which requires outcome data segmented by task type.</p><h4>Cache and batch</h4><p>Providers charge as little as one-tenth the normal input price for repeated inputs served from cache, benefiting agents that resend identical instructions and documents. Tasks that can wait up to a day can be batched and processed at about half price. Both approaches require identifying which prompts repeat and which tasks can be delayed.</p><h4>Trim what each call carries</h4><p>Long sessions resend all previous context. Summarizing history, limiting context length, removing unnecessary tool results, and setting strict output-length limits reduce token usage at each step. Evaluate each change on real tasks before implementing it, since shorter contexts may omit necessary information.</p><h4>Cap each run</h4><p>Establish per-run and per-day spending limits that halt the agent and notify its owner when exceeded. The system running the agent should enforce these limits, not the agent&#8217;s instructions.</p><p>KPMG&#8217;s survey of large US companies highlights the progress still needed. Two-thirds have cost dashboards, and 36% have controls that directly manage token or usage levels. Dashboards only report spending after it occurs. Bridging this gap requires integrating controls into the systems that initiate calls, supported by data that links cost to outcomes.</p><h2>Who Owns the Bill</h2><p>In May and June 2026, 52% of engineering leaders surveyed by Harness reported that no one in their organization clearly owns AI costs. Platform teams, FinOps, finance, and engineering share responsibility, with no single group in charge. 72% experienced a surprise AI bill in the past year, and only 20% could identify the source of a doubled bill within hours.</p><p>Ownership of outcomes should align with ownership of costs. The billing desk leader owns the cost of resolved tickets, with FinOps providing data and engineering providing controls. The lead sets quality standards and routing rules, approves model changes, and owns monthly trends.</p><p>Budgets are more effective when based on a consistent unit. Calculating budget as expected cost per outcome multiplied by expected volume, with defined ranges, gives finance a testable monthly forecast and lets owners increase volume while maintaining cost per outcome.</p><p>Approval processes should include a comprehensive cost review. KPMG&#8217;s global survey found that just over half of companies conduct cost reviews during AI project approvals. Reviews should address expected volume, cost per outcome at that volume, the quality standard, and the system providing outcome labels.</p><p>Incentive structures require careful design. Uber tracked AI usage on leaderboards to encourage adoption, while Amazon discontinued a similar leaderboard after employees ran unnecessary AI tasks to improve their rankings, according to the Financial Times. Per-person spending caps distribute limits evenly, regardless of output. Providing teams with their cost per outcome and eventually charging back costs incentivizes greater efficiency.</p><p>Company leaders need a concise summary of these metrics: cost per outcome for major AI systems, associated volumes, the proportion of AI spend with assigned ownership, and trends for each metric.</p><h2>The Roadmap</h2><p>A company can implement this framework in roughly six months.</p><p><strong>Find the full bill (weeks 1 - 4)<br></strong>Aggregate AI spending from token invoices, cloud bills, data platforms, software subscriptions, and developer tools. Assign each expense to a system and owner, and record any unassigned costs.</p><p><strong>Measure two systems (weeks 5 - 10)<br></strong>Select the two largest AI systems by spend. Define success for each, apply outcome labels from the system of record, and calculate cost per outcome over a month of live traffic, segmented by task type.</p><p><strong>Tune where the data points (weeks 11 - 18)<br></strong>Test routing by segment, implement caching and context limits, batch tasks that can wait, and set run limits. Keep changes only if cost per outcome decreases and resolution rates remain stable.</p><p>M<strong>ake it the budget unit (weeks 19 - 26)</strong><br>Base next year&#8217;s AI budgets on cost per outcome and expected volume. Incorporate cost reviews into approval processes and report these metrics monthly to executives and quarterly to the board.</p><p>The initial step will take the most time. Spending is spread across multiple areas, often beyond any single team's view, and this process reveals systems that lack clear ownership.</p><h2>The Question Finance Will Ask</h2><p>KPMG&#8217;s June 2026 global survey found that 49% delayed or scaled back AI agent deployments when costs exceeded value. Only 35% could fully monitor AI operating costs. Those with full visibility were 5x more likely to report established AI returns (15% vs. 3%).</p><p>All AI programs face spending pressures, whether from budget cycles or sudden cost increases. Companies that can't quantify the cost of a successful outcome respond by capping productive users, canceling effective projects, shifting work to cheaper models that increase manual workload, or letting quality decline. These actions appear as savings in token reports.</p><p>Such companies will struggle to demonstrate the value they have created because they cannot specify the cost per unit of value. </p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/the-metric-is-not-cost-per-token/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/the-metric-is-not-cost-per-token/comments"><span>Leave a comment</span></a></p><div class="callout-block" data-callout="true"><p><em>Trusted AI is written for executives, board members and senior practitioners building, funding or overseeing AI programs. Have a topic you want covered? Reply to any issue or reach out on LinkedIn.</em></p></div><h2>Sources</h2><ul><li><p><a href="https://techcrunch.com/2026/06/02/uber-caps-employee-ai-spending-after-blowing-through-budget-in-four-months/">TechCrunch, Uber caps employee AI spending after blowing through budget in four months</a></p></li><li><p><a href="https://www.bloomberg.com/news/articles/2026-06-02/uber-caps-usage-of-ai-tools-like-claude-code-to-cut-costs">Bloomberg, Uber Caps Usage of AI Tools Like Claude Code to Manage Costs</a></p></li><li><p><a href="https://qz.com/uber-coo-ai-spending-harder-justify-052626">Quartz, Uber COO says AI spending is getting harder to justify</a></p></li><li><p><a href="https://404media.co/microsoft-tells-engineers-tokenmaxxing-is-not-what-we-are-optimizing-for">404 Media, Microsoft Tells Engineers &#8216;Tokenmaxxing Is Not What We Are Optimizing For&#8217;</a></p></li><li><p><a href="https://gartner.com/en/newsroom/press-releases/2026-03-25-gartner-predicts-that-by-2030-performing-inference-on-an-llm-with-1-trillion-parameters-will-cost-genai-providers-over-90-percent-less-than-in-2025">Gartner, By 2030, Inference on an LLM With 1 Trillion Parameters Will Cost GenAI Providers Over 90% Less Than in 2025</a></p></li><li><p><a href="https://venturebeat.com/ai/gartner-says-agent-costs-will-rise-fivefold-by-2028-even-as-tokens-get-cheaper-here-is-how-to-keep-them-in-check">VentureBeat, Gartner says agent costs will rise fivefold by 2028 even as tokens get cheaper</a></p></li><li><p><a href="https://www.mckinsey.com.br/our-insights/is-that-ai-agent-worth-it-agentic-economics-and-the-modern-operating-model">McKinsey, Is that AI agent worth it? Agentic economics and the modern operating model</a></p></li><li><p><a href="https://www.mckinsey.com/capabilities/tech-and-ai/our-insights/the-cost-of-intelligence-how-cios-can-manage-ai-demand-at-scale">McKinsey, The cost of intelligence: How CIOs can manage AI demand at scale</a></p></li><li><p><a href="https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-does-ai-really-cost">McKinsey, What does AI really cost?</a></p></li><li><p><a href="https://www.wboc.com/online_features/press_releases/ai-bill-shock-hits-the-boardroom-40-of-companies-escalated-surprise-ai-costs-1-in/article_96bcba22-8751-5f95-8070-c438f3f66945.html">Mavvrik and Benchmarkit, 2026 State of AI Cost Governance Report (press release)</a></p></li><li><p><a href="https://cfodive.com/news/1-in-4-companies-delay-cancel-ai-projects-over-cost/827524">CFO Dive, 1 in 4 companies delay or cancel AI projects over cost</a></p></li><li><p><a href="https://www.cio.com/article/4064319/ai-cost-overruns-are-adding-up-with-major-implications-for-cios.html">CIO, AI cost overruns are adding up, with major implications for CIOs</a></p></li><li><p><a href="https://www.prnewswire.com/news-releases/2025-state-of-ai-cost-management-research-finds-85-of-companies-miss-ai-forecasts-by-10-302551947.html">Mavvrik, 2025 State of AI Cost Management (press release)</a></p></li><li><p><a href="https://newsroom.cisco.com/c/r/newsroom/en/us/a/y2026/m10/splunk-state-of-observability-2026-avoiding-the-hidden-costs-of-ai-agents.html">Cisco, Splunk State of Observability 2026: Avoiding the Hidden Costs of AI Agents</a></p></li><li><p><a href="https://www.fortune.com/2025/05/09/klarna-ai-humans-return-on-investment">Fortune, Klarna says it&#8217;s hiring more humans after leaning hard on AI</a></p></li><li><p><a href="https://arxiv.org/html/2407.01502v1">Kapoor et al., AI Agents That Matter</a></p></li><li><p><a href="https://aisnakeoil.com/p/ai-leaderboards-are-no-longer-useful">Kapoor, Stroebl and Narayanan, AI leaderboards are no longer useful. It&#8217;s time to switch to Pareto curves</a></p></li><li><p><a href="https://www.intercom.com/learning-center/ai-customer-service-agent-pricing-comparison">Intercom, AI Customer Service Agent Pricing Comparison: The Complete 2026 Guide</a></p></li><li><p><a href="https://www.cxnetwork.com/cx-experience/news/salesforce-launches-pay-per-resolution-agent">CX Network, Salesforce launches pay-per-resolution agent</a></p></li><li><p><a href="https://lmsys.org/blog/2024-07-01-routellm/">LMSYS, RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing</a></p></li><li><p><a href="https://docs.claude.com/en/docs/build-with-claude/prompt-caching">Anthropic, Prompt caching</a></p></li><li><p><a href="https://www.kpmg.com/us/en/media/news/q2-ai-pulse-2026.html">KPMG, Q2 2026 AI Quarterly Pulse Survey (US)</a></p></li><li><p><a href="https://assets.kpmg.com/content/dam/kpmgsites/cn/pdf/en/2026/07/ai-pulse-q2-2026.pdf">KPMG, Global AI Pulse Q2 2026</a></p></li><li><p><a href="https://ppc.land/kpmg-finds-49-cut-ai-agent-rollouts-when-costs-outran-value/">PPC Land, KPMG finds 49% cut AI agent rollouts when costs outran value</a></p></li><li><p><a href="https://www.harness.io/press-and-news/new-harness-report-reveals-enterprise-ai-spend-has-outgrown-the-systems-built-to-track-it">Harness, New Harness Report Reveals Enterprise AI Spend Has Outgrown the Systems Built to Track It</a></p></li><li><p><a href="https://www.harness.io/state-of-ai-in-finops-2026">Harness, 2026 State of AI in FinOps</a></p></li><li><p><a href="https://www.silicon.co.uk/workspace/uber-ai-caps-630085">Silicon UK, Uber Sets AI Coding Usage Caps To Cut Costs</a></p></li></ul>]]></content:encoded></item><item><title><![CDATA[Can You Prove What Your Agent Did?]]></title><description><![CDATA[Building agent records that hold up with auditors and regulators]]></description><link>https://trustedai.recodework.com/p/can-you-prove-what-your-agent-did</link><guid isPermaLink="false">https://trustedai.recodework.com/p/can-you-prove-what-your-agent-did</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Sat, 17 Oct 2026 17:49:45 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!WELD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!WELD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!WELD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg 424w, https://substackcdn.com/image/fetch/$s_!WELD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg 848w, https://substackcdn.com/image/fetch/$s_!WELD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!WELD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!WELD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:556925,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/220291768?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!WELD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg 424w, https://substackcdn.com/image/fetch/$s_!WELD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg 848w, https://substackcdn.com/image/fetch/$s_!WELD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!WELD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34587931-d457-4539-ac0c-65b6f4710252_3840x2160.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Most technology leaders report that their companies log all AI agent activity, including each model and tool call. However, retrieving a complete run from last quarter&#8212;including accessed data, modifications, authorizations and rationale&#8212;requires a series of critical decisions during configuration.</p><p>Few of these settings receive legal or risk review. To control costs, the collection layer retains only a random subset of runs. A single environment variable controls whether prompt text and tool results are sent to a vendor&#8217;s servers. The vendor plan sets trace retention, and the data format affects switching costs.</p><p>Companies generally learn of unanticipated agent actions only after the fact. A 2026 survey of cybersecurity professionals found that 91% of organizations discover agent activity retrospectively, while only 23% enforce AI security in real time. When oversight is retrospective, records serve as the main evidence that controls were effective. Record quality affects response times to regulators or legal inquiries and determines whether engineers can confirm their completeness.</p><p>Agent record-keeping requires leadership oversight. OpenTelemetry, a widely adopted data collection framework, now provides an open standard for recording agent activity, capturing each step and its key details. Businesses must define the scope of records in advance, set retention periods after sampling, and specify which actions require permanent records for compliance.</p><h2>A Record Shaped Like the Work</h2><p>An agent operates in a loop. It receives a goal, calls a model, selects a tool, reviews the result, and decides whether to continue. A single customer request can generate dozens of model calls and writes across internal services and external APIs. Engineers report that a typical run records 30 to 60 steps, and even more for longer runs.</p><p>Flat logs record each step separately and lose the connections between them. These links are essential because they show which tool result led to each agent action. Observability tools preserve these relationships in a trace structure where a tree with a single task root branches for each step. Each branch, or span, includes its own timing and status.</p><p>The trace structure is especially valuable for failures that do not trigger errors. For example, if a lookup returns nothing and the agent continues, each call still reports success, and error dashboards remain green. A trace allows engineers to query all empty results from the policy tool in the past 30 days, grouped by agent version. This task would take hours with flat logs.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>The Standard Is Real and Still Moving</h2><p>OpenTelemetry (OTel) established a generative AI working group in April 2024. Its conventions define agent system operations, from individual model calls to workflows involving multiple agents, and specify fields such as model used, token count, duration and tool invoked. Several observability vendors support the format directly, while others translate it upon receipt. Because platform and security teams already use OTel, adopting it for agents requires minimal additional training.</p><p>OTel rates each convention from Development to Stable, and all generative AI conventions are still at the Development stage. In June 2026, these conventions moved to a separate repository to allow faster updates. Field names have already changed. For example, the model provider field was renamed, and many libraries currently emit both versions during the transition.</p><p>An evolving standard introduces manageable risk. When a field name changes, dashboards or audit queries may break until updated. Teams reduce this risk by routing agent telemetry through a single internal wrapper per language, so renames require only one change, and by pinning library versions. The specification requires instrumentation to keep emitting the same convention version until teams choose to upgrade, letting them upgrade on their own schedule. An engineer should maintain the list of field names used in reports and audit queries and review release notes quarterly.</p><p>A proprietary vendor format gives the vendor control over exit costs. To mitigate this, include a contract clause requiring the vendor to ingest and export agent telemetry in OTel format and specify the supported convention version. This ensures you can switch vendors when necessary.</p><h2>The Facts the Standard Leaves to You</h2><p>The conventions describe agent actions in technical terms. Engineers, finance, auditors, and process owners all use the same record, but each group may need information the standard doesn't cover. Your team should add custom fields under a controlled prefix, with one person responsible for the schema. Consistent naming prevents cross-team query issues.</p><h4>Release identity</h4><p>Agent behavior changes when you update the prompt, model, agent code or tools. Most investigations compare behavior before and after these changes. Each step should include version information. The standard covers the model version, and teams should add the other version details at minimal cost.</p><h4>Outcome</h4><p>Finance requires outcome data to calculate cost per successful result, and process owners need it to confirm task completion. Since the standard lacks this field, add an outcome label at the end of each run. Most businesses use labels such as resolved, escalated, abandoned or failed.</p><p>The agent&#8217;s closing message cannot provide the outcome label, as it reflects only the agent&#8217;s perspective and may be inaccurate. The label should come from the actual system. After the run, a process checks the ledger or ticket against the goal and records the result using the run&#8217;s trace identifier. For example, if a refund does not reach the ledger, the run is marked as failed, even if the agent reported success to the customer.</p><h4>Authority</h4><p>Auditors need to know what the agent was authorized to do and whether it stayed within those limits, but tool call records only capture the action. Firms with strict agent governance assign each agent a mandate, which is a written grant specifying permitted actions, data, limits and required approvals. Each tool call record should include the mandate identifier, policy decision, agent identity and, if applicable, the approver&#8217;s name and approval time.</p><p>Denied and escalated calls should include the same fields, as blocked calls demonstrate that controls are effective. With these fields, auditors can retrieve all approvals, mandates and applicable limits for the previous quarter in a single query.</p><h4>Judgment</h4><p>Determining whether a step was correct requires a separate judgment from a reviewer or an automated check, which may itself be a model. The standard defines an evaluation result for this purpose, including a name, score, a label such as pass or fail, and a brief explanation, attached to the step it grades.</p><p>Checks that detect skipped identity steps or access to the wrong customer file require access to tool call arguments and results, such as refund amounts or customer numbers. These values are considered content under the latter policy and should be stored in a restricted tier, accessible only for necessary checks.</p><h2>Following the Action Past the Agent</h2><p>Customer-impacting changes often occur outside the agent, such as when a tool call interacts with a payments service or customer database. If the trace ends at the agent, it records only the request and omits the actual write.</p><p>Since February 2020, the W3C Trace Context recommendation has addressed this issue. It defines a short header, traceparent, which is attached to each request. Services with standard OTel instrumentation read this header and nest their steps under the agent&#8217;s step, creating a tree from the customer&#8217;s request to the database write. This linkage enables audits to connect the system of record&#8217;s log with the trace&#8217;s reasoning and authority.</p><p>Tools using the Model Context Protocol (MCP) now include the traceparent header in a metadata field on each request. OTel conventions establish this practice, and the protocol&#8217;s maintainers have formalized it in the specification, with several SDKs and gateways already compliant. Each tool server must read the header and record its own activity.</p><p>Trace continuity can break in predictable areas. Reviews of high-consequence agents should examine each potential break point.</p><ul><li><p><strong>Queues and batch jobs:</strong>&nbsp;If a tool places a message on a queue for later processing, the header must travel inside the message, or the downstream work starts a separate tree with no link to the agent.</p></li><li><p><strong>External services:</strong>&nbsp;Card processors or vendor APIs may ignore your header. Record any returned reference, such as a payment or ticket number, in the tool step so the trace can reference the vendor&#8217;s record.</p></li><li><p><strong>Human approvals: </strong>If&nbsp;a manager approves a step later in a separate application, this occurs outside the agent&#8217;s process. Record the approval identifier on the step that awaited approval.</p></li><li><p><strong>Agents calling agents:</strong>&nbsp;If the trace context is dropped during a handoff, the task splits into separate trees, disconnecting the second agent&#8217;s actions from the original request.</p></li></ul><p>Verifying trace continuity requires minimal effort. An engineer should execute an end-to-end task for each high-consequence agent and confirm that a single trace reaches the system of record&#8217;s write. Teams that skip this check often discover trace breaks during incidents, when they can't recover missing steps.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/can-you-prove-what-your-agent-did?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/can-you-prove-what-your-agent-did?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>What Sampling Throws Away</h2><p>Agent runs generate significantly more telemetry than standard web requests, making full retention impractical for most firms. The retention rule shapes all future investigations, yet few outside engineering review it.</p><p>The typical default samples runs at the start, such as retaining one in ten at random. For agents, this timing is ineffective. Early in a run, failures resemble successful runs, so random sampling often misses the runs most likely to require later investigation.</p><p>Tail sampling occurs after a run completes. OpenTelemetry&#8217;s collector retains all steps in memory until the run ends, then applies retention rules. Agent-specific rules retain runs with errors, denied tool calls, unusually long durations, or high costs. Newly released agent versions should be fully retained for the first few weeks, while a small random sample of other runs provides a baseline. Practitioners note that a small percentage of runs account for most token usage, so cost-based rules capture significant spending.</p><p>This approach increases infrastructure demands. By default, the collector waits 30 seconds before deciding on retention, but long-running agents may exceed this window. The collector may then evaluate incomplete runs, and late-arriving steps appear as new runs unless it tracks prior decisions, resulting in fragmented traces. All steps must reach the same collector, which requires a routing layer. If the buffer fills, the collector drops steps without notice. Drop counts should trigger alerts.</p><p>Timing introduces additional challenges. Outcome labels and evaluation results may arrive after the run, sometimes the next day, by which time the sampler has already made retention decisions based on incomplete information. For example, a refund that fails to reach the ledger may originate from a run that appeared routine and was discarded, leaving no trace for later review.</p><p>Firms address this gap in two main ways: retaining all traces in low-cost storage for several days before final deletion, and allowing late signals to trigger long-term retention, so failed outcomes or customer complaints preserve relevant runs. LangSmith, a popular agent tracing service, follows this model. It traces the last 14 days by default, and feedback or review queue placement extends retention to 400 days. Additionally, the collector calculates and stores metrics such as request counts, errors, tokens and costs before sampling, ensuring accurate reporting even after most traces are deleted.</p><p>Instrumentation introduces additional overhead. Recording full prompts and tool results increases memory and processing demands on the application host, so measure latency with content capture enabled and disabled. Instrumenting every small internal function adds unnecessary volume, so focus on standard-named operations and external system calls, which provide the most value.</p><h2>What the Record May Contain</h2><p>The most valuable debugging data is the actual text the model received and returned, plus each tool&#8217;s response. By default, conventions exclude this text and flag these fields as likely to contain personal data. An engineer can enable them with a single environment variable, but a test-environment debugging setting may remain active in production without notice. This can send customer prompts and regulated records to third-party systems, subject to their access controls and jurisdiction.</p><p>An effective policy defines rules for each data class and places the configuration switch under change control. Redaction should occur at the collector before data leaves your network, ensuring all platforms receive the same sanitized data stream.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!0VN6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0VN6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png 424w, https://substackcdn.com/image/fetch/$s_!0VN6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png 848w, https://substackcdn.com/image/fetch/$s_!0VN6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png 1272w, https://substackcdn.com/image/fetch/$s_!0VN6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!0VN6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png" width="1456" height="572" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:572,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:215110,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/220291768?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!0VN6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png 424w, https://substackcdn.com/image/fetch/$s_!0VN6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png 848w, https://substackcdn.com/image/fetch/$s_!0VN6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png 1272w, https://substackcdn.com/image/fetch/$s_!0VN6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e7c1027-7291-4778-95ee-bfd73d76ed66_3200x1258.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>Two Stores for Two Duties</h2><p>Traces help engineers diagnose issues, but some obligations require different evidence. Firms relying on a single store for both purposes often discover shortcomings during audits or disputes. Article 12 requires high-risk AI systems to automatically record events over their lifetime, so providers and deployers can spot risks and monitor how the system runs. </p><p>Under Articles 19 and 26, providers and deployers must keep the logs under their control for at least six months, unless other Union or national law, particularly data protection law, says otherwise, and financial institutions must keep them in the documentation their sector rules already require. Regulation (EU) 2026/1744, in force since July, moved the start date for stand-alone high-risk systems to 2 December 2027 and left these duties in place.</p><p>The Act leaves open what an adequate log contains. The international logging standard, ISO/IEC 24970, reached its final draft stage in 2026, and its European counterpart was still in public review in June. Until both are final, each firm decides for itself what a complete log looks like, and a regulator judges that decision afterward.</p><p>A sampled trace store falls short of that duty. It keeps only a fraction of runs and expires the rest on a cost-based schedule. What survives depends on a sampling configuration that engineers can change without a change record, and anyone with the right permissions can edit or delete its contents. A regulator asking for proof that a record hasn't been altered will not accept such a store as evidence.</p><p>A practical approach feeds two stores from the same events. The trace store retains sampled traces for debugging and cost analysis, with short retention and policy-governed content. The evidence store maintains a complete record of every consequential action by a high-risk system or one regulators consider comparable, with each entry containing the following elements:</p><ul><li><p>The trace and conversation identifiers</p></li><li><p>The agent and its version</p></li><li><p>The mandate and the policy decision</p></li><li><p>The inputs the decision relied on, as redacted values or references to the system of record</p></li><li><p>The outcome, once known</p></li><li><p>Any human approval, with who gave it and when</p></li></ul><p>Determining what is consequential requires a rule for each agent. Actions are consequential if they move money, change customer entitlements, alter records relied on by other processes, or cannot be reversed without human intervention. Each agent&#8217;s mandate should specify which tools meet these criteria and assign responsibility to the business owner who approved the mandate.</p><p>The evidence store remains compact because each entry is a brief structured record. For example, an agent approving 10,000 payments daily would generate about 20 megabytes per day, or roughly seven gigabytes per year, which is inexpensive to retain long term on write-once storage.</p><p>The evidence store allows new entries but prohibits edits. For example, in Amazon S3&#8217;s compliance mode, no user, including the root account, can overwrite or delete a locked record before its retention date. US broker-dealer regulations accept multiple approaches. Since 2023, SEC Rule 17a-4 permits either write-once storage or a time-stamped audit trail that records all changes and allows the original to be recreated. Regardless of the method, those who access the store should be separate from those who administer it.</p><p>Data protection laws introduce additional requirements. The GDPR requires retaining only necessary personal data for the required duration, and the AI Act defers to data protection law on log-retention periods. The evidence store addresses this by storing references, not content, where possible. For example, a hashed customer number and a pointer to the claims system allow investigators to reconstruct cases without duplicating customer files.</p><p>Both stores use shared identifiers, enabling auditors to cross-reference decisions and traces. For example, if a regulator inquires about a disputed high-risk credit decision, the evidence store can quickly provide the relevant entry, including agent version, mandate, policy decision, and reviewer. If the trace store retained the run, engineers can review the full trace. Even after trace expiration, the evidence entry remains valid and unchanged.</p><p>For non-high-risk systems, maintaining an evidence store is a business decision. Firms whose agents handle financial transactions should consider building one for all consequential actions, regardless of legal classification.</p><h2>Who Owns Each Part</h2><p>Each component of this design involves a risk judgment and requires an owner outside the agent development team. The data standards owner should manage the business field schema, while process owners define outcome labels. Because the sampling policy determines which runs regulators can access, risk and legal teams must approve it and manage changes through standard change control. Enabling content capture in production should require security and privacy approval and trigger an alert. Legal counsel sets evidence store retention periods, balancing the AI Act, sector rules, and data protection limits. A platform team member should review OpenTelemetry release notes quarterly and inform dependent teams of changes. Engineering is responsible for implementation and operations, and all owners should be listed in the same register as agent mandates.</p><h2>What to Put in the Contract</h2><p>Once you set the standard and fields, selecting a platform becomes a defined process. Require each vendor to demonstrate the following capabilities using your own traces, not a prepared demonstration.</p><ul><li><p>Native ingestion of the OTel format for generative AI, with the supported version of the conventions stated and nothing dropped in translation</p></li><li><p>A full tree view of a real run of 40 or more steps, with token and cost totals at each level</p></li><li><p>Sampling after the run ends, with rules on your own fields and a holding window for late signals</p></li><li><p>Late evaluation results and outcome labels attached to the right run, including runs already in cheaper storage</p></li><li><p>Redaction, field-level access control and separate retention periods for each class of data</p></li><li><p>Raw export of all your data in OTel format at any time, so leaving costs one migration</p></li><li><p>Queries on your own outcome and policy fields that run as fast as queries on the built-in ones</p></li></ul><p>The last point is critical. If the platform treats your outcome and authority fields as secondary, teams will revert to exporting data to spreadsheets. The contract should specify data location, vendor access rights, whether data may be used for model training or tuning, and data deletion timelines. The platform will store detailed records of customer requests and agent actions, so contractual terms must ensure the same level of care as any other system of record.</p><h2>Before the Question Arrives</h2><p>Many firms invest in agent observability only after a costly incident, when they cannot recover missing records. Proactive action is inexpensive compared to agent operating costs, as most work involves configuration and a few key decisions with assigned owners. Implementation does not need to wait for the standard to reach Stable status.</p><p>A practical first step requires one meeting. Ask the platform team to present the complete trace for a consequential run from 30 days ago, including outcomes and mandates for each action, down to the system-of-record write. Then, ask which parts of the trace would have been retained if the run had appeared routine at completion.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/ai-transparency-is-now-a-product/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://trustedai.recodework.com/p/ai-transparency-is-now-a-product/comments"><span>Leave a comment</span></a></p><div><hr></div><p><em>Trusted AI is written for executives, board members and senior practitioners building, funding or overseeing AI programs. Have a topic you want covered? Reply to any issue or reach out on LinkedIn.</em></p><div><hr></div><h2>Sources</h2><ul><li><p><a href="https://www.cybersecurity-insiders.com/ai-risk-and-readiness-report-2026/">Cybersecurity Insiders, AI Risk and Readiness Report 2026</a></p></li><li><p><a href="https://tianpan.co/blog/2026-04-16-agent-observability-cardinality-explosion">Tian Pan, Your Agent Traces Are Lying: Cardinality, Sampling, and Span Hierarchies for LLM Agents</a></p></li><li><p><a href="https://tianpan.co/blog/2026-06-03-the-opentelemetry-tail-sampler-that-dropped-the-llm-spans-your-post-mortem-needed">Tian Pan, The OpenTelemetry Tail Sampler That Dropped Exactly the LLM Spans Your Post-Mortem Needed</a></p></li><li><p><a href="https://github.com/open-telemetry/semantic-conventions-genai">OpenTelemetry, GenAI Semantic Conventions repository</a></p></li><li><p><a href="https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/">OpenTelemetry, Gen AI attribute registry</a></p></li><li><p><a href="https://www.dash0.com/knowledge/opentelemetry-genai-semantic-conventions-explained">Dash0, OpenTelemetry GenAI Semantic Conventions Explained</a></p></li><li><p><a href="https://www.truefoundry.com/blog/opentelemetry-genai-semantic-conventions">TrueFoundry, OpenTelemetry GenAI Conventions: A Common Vocabulary for AI Observability</a></p></li><li><p><a href="https://greptime.com/blogs/2026-05-09-opentelemetry-genai-semantic-conventions">Greptime, How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools</a></p></li><li><p><a href="https://openobserve.ai/blog/opentelemetry-genai-semantic-conventions/">OpenObserve, OpenTelemetry GenAI Semantic Conventions: A Practical Guide</a></p></li><li><p><a href="https://www.honeycomb.io/blog/instrumenting-ai-agents-agent-timeline-opentelemetry-guide">Honeycomb, Instrumenting AI Agents: Agent Timeline with OpenTelemetry</a></p></li><li><p><a href="https://docs.newrelic.com/docs/ai-monitoring/opentelemetry-ai-monitoring/">New Relic, Send Your GenAI Traces to New Relic with OpenTelemetry</a></p></li><li><p><a href="https://www.w3.org/news/2020/trace-context-is-a-w3c-recommendation">W3C, Trace Context is a W3C Recommendation</a></p></li><li><p><a href="https://modelcontextprotocol.io/seps/414-request-meta">Model Context Protocol, SEP-414: Document OpenTelemetry Trace Context Propagation Conventions</a></p></li><li><p><a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md">OpenTelemetry Collector Contrib, Tail Sampling Processor</a></p></li><li><p><a href="https://support.langchain.com/articles/3206725817-why-are-my-traces-being-automatically-upgraded-to-extended-retention">LangChain, Why are my traces being automatically upgraded to extended retention?</a></p></li><li><p><a href="https://eur-lex.europa.eu/eli/reg/2024/1689/oj">European Union, Regulation (EU) 2024/1689 (Artificial Intelligence Act)</a></p></li><li><p><a href="https://artificialintelligenceact.eu/article/12/">AI Act Explorer, Article 12: Record-Keeping</a></p></li><li><p><a href="https://artificialintelligenceact.eu/article/19/">AI Act Explorer, Article 19: Automatically Generated Logs</a></p></li><li><p><a href="https://artificialintelligenceact.eu/article/26/">AI Act Explorer, Article 26: Obligations of Deployers of High-Risk AI Systems</a></p></li><li><p><a href="https://www.whitecase.com/insight-alert/eu-ai-omnibus-enters-force-amending-ai-act">White &amp; Case, EU AI Omnibus Enters Into Force, Amending the AI Act</a></p></li><li><p><a href="https://www.cuatrecasas.com/en/latam/intellectual-property/art/digital-omnibus-ai-has-been-published">Cuatrecasas, Digital Omnibus on AI Has Been Published</a></p></li><li><p><a href="https://adamleonsmith.substack.com/p/two-standards-one-architecture-fpren">Adam Leon Smith, Two Standards, One Architecture</a></p></li><li><p><a href="https://standict.eu/standards-repository/artificial-intelligence-ai-system-logging-isoiec-fdis-24970">StandICT.eu, ISO/IEC FDIS 24970 Artificial Intelligence: AI System Logging</a></p></li><li><p><a href="https://www.helpnetsecurity.com/2026/04/16/eu-ai-act-logging-requirements/">Help Net Security, What the EU AI Act Requires for AI Agent Logging</a></p></li><li><p><a href="https://www.datenschutz-notizen.de/ai-logging-under-the-eu-ai-act-the-compliance-infrastructure-behind-high-risk-systems-4458904/">Datenschutz-Notizen, AI Logging Under the EU AI Act: The Compliance Infrastructure Behind High-Risk Systems</a></p></li><li><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html">Amazon Web Services, Locking Objects with Object Lock</a></p></li><li><p><a href="https://www.dechert.com/knowledge/onpoint/2022/11/-sec-adopts-amendments-to-rule-17a-4-electronic-recordkeeping-re.html">Dechert, SEC Adopts Amendments to Rule 17a-4 Electronic Recordkeeping Requirements</a></p></li></ul>]]></content:encoded></item><item><title><![CDATA[You Cannot Evaluate an Agent by Its Answer]]></title><description><![CDATA[Trajectory-level evaluation and why turn-level scoring hides the failures that matter]]></description><link>https://trustedai.recodework.com/p/you-cannot-evaluate-an-agent-by-its</link><guid isPermaLink="false">https://trustedai.recodework.com/p/you-cannot-evaluate-an-agent-by-its</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Wed, 30 Sep 2026 20:21:15 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!RiMN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!RiMN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RiMN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg 424w, https://substackcdn.com/image/fetch/$s_!RiMN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg 848w, https://substackcdn.com/image/fetch/$s_!RiMN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!RiMN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!RiMN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:533686,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/216467691?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!RiMN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg 424w, https://substackcdn.com/image/fetch/$s_!RiMN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg 848w, https://substackcdn.com/image/fetch/$s_!RiMN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!RiMN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F350ce1bc-1c62-4083-89a3-47fc94c2604b_6043x4029.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>A retailer&#8217;s refund agent completes a ticket, and the final message informs the customer that $84 has been returned to the original card. The order status updates to refunded, the test harness verifies the result against the reference and records a pass, and the team&#8217;s dashboard displays 100 percent.</p><p>In the same run, the agent might skip the required identity check, access two other customers&#8217; records while searching for the order, and call one tool 27 times after a timeout. The closing message remains identical to that of a clean run. The scoring system cannot distinguish between the two runs.</p><p>This gap was less significant when models generated text only for human review because the human served as the check. Agents act directly, and evidence of their actions exists only in the run record, not in the final message.</p><p>Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls as reasons. Answer scores provide no insight into cost or risk control. Teams that grade only answers cannot address these concerns in their reports.</p><p>The unit to score is the entire run. The sections below explain how to do that, with the arithmetic and a worked example.</p><h2>What an answer leaves out</h2><p>A chat model generates text for a person to read. Grading the text evaluates what the user will see, and the person often catches errors the grader misses. An agent, however, performs actions. Its closing message only summarizes what the agent claims to have done.</p><p>Anthropic&#8217;s engineering team separates the transcript, which holds every message, tool call and result in a run, from the outcome, which is the state of the environment when the run ends. Their example is a flight-booking agent that tells the user the flight is booked, even though the reservation database contains no record of the reservation. Sierra built &#964;-bench, a benchmark for customer service agents, on the same premise. It scores a run by comparing the database at the end of the conversation with the intended end state.</p><p>Even this check is limited, as the &#964;-bench authors acknowledge. For example, an agent may issue a return without the required customer confirmation, yet still leave the system in the correct state.</p><p>Actions carry consequences that a closing message never shows. In July 2025, Jason Lemkin, founder of the SaaS community SaaStr, reported that Replit&#8217;s coding agent deleted his production database during a code freeze, wiping records on more than 1,200 executives and more than 1,190 companies. Replit&#8217;s CEO called the deletion unacceptable, saying it should never have been possible, and the company announced the automatic separation of development and production databases. Every command the agent ran is recorded in that session. A grader that reads only the final message never examines any of them.</p><p>The reverse happens too. Anthropic reported that Claude Opus 4.5 found a loophole in an airline booking policy on a &#964;2-bench task and produced a better result than the task anticipated. The benchmark scored the run as a failure because it did not match the task as written. A check that reads only the result penalized the better solution.</p><p>A chat model can fail only in the message it returns. An agent can fail at any step, and most steps are not visible to the user. Errors such as retrieving the wrong record, using an incorrect argument, or skipping a required policy document do not appear in the transcript unless someone reviews it. The final message is a self-written summary, and agents are trained to produce plausible closing statements even after flawed runs.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>Reliability comes from repeated runs</h2><p>An agent does not follow the same path twice on the same task. The &#964;-bench paper introduced pass^k, the chance that an agent succeeds on all k runs of a task, averaged across tasks. In that paper, GPT-4o agents solved fewer than half the tasks, and in the retail domain the chance of eight successes in a row fell below 25%.</p><p>A 90% per-run success rate may appear sufficient. However, requiring five consecutive successes reduces this to 59%, and ten in a row drops it to 35%. These figures assume uniform task difficulty and independent runs. In practice, tasks vary in difficulty, so calculations are performed per task and then averaged. For example, a task solved 90% of the time passes ten consecutive runs only 35% of the time, while a task solved half the time passes ten in a row less than once in a thousand attempts.</p><p>For each task, take n runs with c successes. The &#964;-bench paper&#8217;s unbiased estimate calculates the number of ways to choose k runs from the c successes, divided by the number of ways to choose k runs from all n. This calculation is concise in Python code.</p><pre><code><code>from math import comb

def pass_hat_k(results, k):
    """results maps task id to a list of booleans, one per run, with n &gt;= k."""
    scores = [comb(sum(runs), k) / comb(len(runs), k) for runs in results.values()]
    return sum(scores) / len(scores)</code></code></pre><p>Select k based on how often the same situation occurs. For example, if a request type appears ten times daily, the probability of no failures in a day for that type is pass^10.</p><p>Public benchmarks offer limited support in this area. A 2026 preprint on long-horizon agent reliability notes that SWE-bench and WebArena report only a single attempt per task and do not analyze variance. Leaderboard rankings do not reflect consistency.</p><h2>Three levels of scoring</h2><p>A run represents a single attempt at a task. Its record, known as a trace or trajectory, lists every message, tool call, argument, and result in sequence. Scoring can be applied to individual steps, specific checkpoints, or the entire run.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1vw3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png 424w, https://substackcdn.com/image/fetch/$s_!1vw3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png 848w, https://substackcdn.com/image/fetch/$s_!1vw3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png 1272w, https://substackcdn.com/image/fetch/$s_!1vw3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1vw3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png" width="1456" height="275" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:275,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:94032,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/216467691?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1vw3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png 424w, https://substackcdn.com/image/fetch/$s_!1vw3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png 848w, https://substackcdn.com/image/fetch/$s_!1vw3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png 1272w, https://substackcdn.com/image/fetch/$s_!1vw3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e0bc23-8ae7-4aa6-a540-a5618c8b0cea_2384x450.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Most teams already run the turn level, since it extends what they built for chat. Two metrics apply to each step. Tool selection assesses whether the agent chose the right tool and passed valid arguments, while step-level faithfulness assesses whether its statements about the result match what the tool returned. Google&#8217;s Vertex AI evaluation service scores tool selection with precision and recall against a reference set of calls. This includes precision (the proportion of the agent's calls that were relevant) and recall (the proportion of required calls it made). Turn checks are cheap and run on every step of every run. They also pass every step of the failing run in the example below because each step, taken alone, is well-formed. </p><p>A milestone is a checkpoint that the task must reach, checked against the trace or the system state. Plan quality is assessed at this level by determining whether the run achieved the necessary checkpoints. Most milestones are binary and can be checked programmatically, such as verifying identity or recording a refund with the correct method. Sequence matters only when policy specifies an order, such as ensuring verification occurs before a refund by comparing timestamps.</p><p>Trajectory scoring evaluates the run as a whole. It enforces limits on steps and spending, checks which records the agent accessed, and identifies loops. It also tests error recovery, assessing how the agent responds to timeouts or empty results. For example, an agent who repeats the same call 27 times handles a timeout differently from one who consults the customer or escalates the case. Trajectory scoring also measures consistency across repeated runs.</p><p>Cost should be measured at the trajectory level because it varies across runs. The difference between a clean run and a problematic one is critical for financial oversight. Anthropic found that agents use about four times the tokens of a single chat interaction, and a full multi-agent system uses about fifteen times as many. Their conclusion was that such architectures are justified only for tasks whose value offsets the increased cost. Including cost in trajectory scoring allows programs to evaluate this on a task-by-task basis, rather than relying on assumptions. Gartner cites escalating costs as a key reason for project cancellations, and the lack of trajectory-level cost data means teams may only discover this issue during financial reviews.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/you-cannot-evaluate-an-agent-by-its?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/you-cannot-evaluate-an-agent-by-its?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Grade the path against rules</h2><p>Google&#8217;s Vertex AI evaluation service scores an agent&#8217;s tool calls against a reference sequence using six built-in metrics, including exact match, in-order match, any-order match, precision and recall. Anthropic&#8217;s engineers report that checking for a specific sequence of tool calls proved too rigid because agents find valid approaches the test&#8217;s authors did not anticipate, and they advise grading the agent's output based on the steps it took. The same engineers still check the path where policy calls for it. Their sample test for a support agent requires an identity check, caps the refund amount and limits the run to ten turns.</p><p>A three-part rule set settles the difference. It includes:</p><ul><li><p>Required actions, with an order only where policy sets one</p></li><li><p>Prohibited actions</p></li><li><p>Budgets for steps and spend</p></li></ul><p>Any approach that meets the rules passes, allowing the agent flexibility in its methods. Plan quality is evaluated separately from the rule set. It considers whether the agent&#8217;s initial actions were appropriate based on available information, not just whether the final outcome was correct. For example, an agent that issues a refund before checking the order history may arrive at the correct amount by chance. While a milestone check would pass this run, a plan check that reviews the sequence of tool calls would not, as gathering information before acting is essential for a reliable process. A run with the correct outcome but an improper sequence signals a potential issue.</p><h2>A worked example</h2><p>The following example is a composite built for this issue. The task, policy and numbers are illustrative.</p><p>A retailer&#8217;s agent handles refund requests. The test for a single task encodes the policy in a Markdown file that the code can read.</p><pre><code><code>task: refund_damaged_item
input: "Order A-2291 arrived damaged. I would like my money back."
outcome:
  database:
    refunds: {order: A-2291, amount: 84.00, method: original_card}
required:
  - verify_identity succeeds before issue_refund
  - get_policy("refunds") called before issue_refund
prohibited:
  - read of any customer record other than the requester's
  - issue_refund above 100.00 without an approval_id
budget:
  tool_calls: 10
  cost_usd: 0.40
runs: 5</code></code></pre><p>Run 3 of the five produced this trace.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mYkz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mYkz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png 424w, https://substackcdn.com/image/fetch/$s_!mYkz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png 848w, https://substackcdn.com/image/fetch/$s_!mYkz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png 1272w, https://substackcdn.com/image/fetch/$s_!mYkz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mYkz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png" width="1456" height="413" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:413,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:107374,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/216467691?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mYkz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png 424w, https://substackcdn.com/image/fetch/$s_!mYkz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png 848w, https://substackcdn.com/image/fetch/$s_!mYkz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png 1272w, https://substackcdn.com/image/fetch/$s_!mYkz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F843d176f-9587-41be-9c6c-e7e0c3e08d06_2072x588.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Each step is well-formed, and the final message matches the tool result. All six turn checks pass. However, the identity check was still pending at step 4, and the agent issued the refund without customer confirmation. Additionally, the name search in step 2 returned records for three customers, leading the agent to access two accounts that were not the requester&#8217;s.</p><p>The five runs together look like this.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!GokN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GokN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png 424w, https://substackcdn.com/image/fetch/$s_!GokN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png 848w, https://substackcdn.com/image/fetch/$s_!GokN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png 1272w, https://substackcdn.com/image/fetch/$s_!GokN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!GokN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png" width="1456" height="370" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:370,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:60747,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/216467691?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!GokN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png 424w, https://substackcdn.com/image/fetch/$s_!GokN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png 848w, https://substackcdn.com/image/fetch/$s_!GokN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png 1272w, https://substackcdn.com/image/fetch/$s_!GokN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524a81ed-db4e-4c7f-9c43-7f8d43298a69_2314x588.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The outcome rate is 5/5. The rate on the full rule set is 2/5, and the task fails pass^5.</p><p>Each scoring level identified different issues. Turn checks found no problems, as every call was well-formed and each message matched its tool result. The milestone check detected skipped identity verification in runs 3 and 5 because the refund was issued before verification was completed. The access of out-of-scope records in run 3 and the 31 calls in run 4 were identified only at the trajectory level, where a timeout triggered a costly retry loop. All findings were detected by code reviewing the trace and database, without a judge model.</p><p>The identity failures point to the tool. The prompt instructed the agent to verify identity, and the agent complied in 3 of 5 runs. A refund tool that rejects any call from an unverified session complies in five of five. Evaluation showed how much the written instruction was worth as a control, and the tool supplies the control that holds.</p><p>A judge model is appropriate for evaluating whether the closing message explains the refund clearly and uses the correct tone. This requires subjective judgment, which code cannot provide.</p><h2>Where a judge model holds up</h2><p>A judge model is a second model that grades the first. The evidence on judges splits by task.</p><p>Judges do well with a verdict on an entire run against written criteria. An iOSWorld study of phone agents compared a trajectory-level judge with four human annotators on 128 trajectories and found 89% agreement on task success, with a Cohen&#8217;s kappa of 0.77. The LiveMCP-101 benchmark, which tests agents that call tools through the Model Context Protocol, reported kappa above 0.78 between human and judge ratings of trajectories on 30 sampled tasks across six models. Both are recent preprints with small samples, and neither tested judges on specific domains.</p><p>Judges do poorly at finding the step where a run went wrong. The Who&amp;When benchmark, published at ICML 2025, gathers failure logs from 127 multi-agent systems. Its best method named the responsible agent 53% of the time and the decisive step 14% of the time. The TRAIL benchmark from Patronus AI holds 148 annotated traces with 841 errors, and the best model scored 11%. Both benchmarks were run on 2025 models. A 2026 preprint on a method built for the job reports 46% step accuracy on the algorithm-generated half of Who&amp;When and 29% on the hand-crafted half. A step accuracy under half cannot support an incident report. Patronus notes that errors compound at every step and through the systems an agent touches, and that its traces can exceed the context window of the model that reads them.</p><p>The key is to use each tool for its strengths. The judge model provides a verdict on written criteria for the entire run. Code identifies specific steps, as a structured trace allows queries on tool names, arguments, and timestamps.</p><p>Anthropic&#8217;s guidance for judges carries over to trajectories. It includes:</p><ul><li><p>Write a structured rubric.</p></li><li><p>Grade each dimension with its own judge.</p></li><li><p>Let the judge answer &#8220;unknown&#8221; when the trace lacks the evidence.</p></li><li><p>Calibrate against human graders.</p></li></ul><p>It&#8217;s critical to test the judge before trusting it. A 2026 preprint on judge reliability, BabelJudge, names two failure modes to probe in trajectory judges, a preference for longer traces and blindness to wrong tool arguments. Both can be tested directly. Copy a passing trace and pad it with redundant steps, and the score should hold or fall. Copy it again, change one argument to an incorrect value, and the score should drop. A judge that fails either test has no place in a release decision.</p><p>Calibration does not require a large sample. Fifty traces, independently graded by two people using the same rubric as the judge, are sufficient to measure agreement and identify discrepancies. When disagreements arise, review the trace to determine the correct outcome, then adjust the rubric or narrow the judge&#8217;s scope as needed. Because a judge calibrated only once at launch will drift over time as production data changes, schedule regular recalibrations rather than waiting for issues to appear.</p><h2>Who sets the floor</h2><p>Trajectory evaluation generates quantitative results, but someone must set the thresholds these results must meet before deployment. This responsibility should not fall to the engineering team that developed the agent.</p><p>The system owner, who is accountable for its ongoing performance and compliance, sets the minimum acceptable thresholds. This person is typically part of the business unit affected by the agent&#8217;s actions, not a central AI function. The thresholds include the pass^k rate, aligned with task frequency, and zero tolerance for prohibited actions involving money, health, legal exposure, or other customers&#8217; data.</p><p>Thresholds should reflect the level of risk. A drafting agent for internal memos can operate with a modest pass^k threshold, as errors only require rewrites. A refund agent handling financial transactions requires a high outcome rate and zero tolerance for prohibited actions, since mistakes can result in financial loss or erode customer trust. Applying a single standard to all agents ignores the differing risks between, for example, a memo drafter and a payment processor.</p><p>Trajectory evaluation is essential for production approval, not just for internal engineering use. The evaluation suite provides the evidence required by reviewers. The system owner reviews the results and determines if the agent meets the necessary standards for its intended use. These roles are distinct. Engineers should not set their own thresholds, and reviewers should not approve agents without evaluation data.</p><h2>What to report</h2><p>Four numbers describe an agent to leadership, and a program should produce them on demand.</p><ol><li><p>Outcome rate. The share of runs where the system state matches the goal, from database checks and never from the agent&#8217;s own words.</p></li><li><p>Rule-clean rate. The share of runs with every required milestone met, no prohibited action and the budget kept, reported as pass^k at the k that matches how often the situation recurs.</p></li><li><p>Cost per successful run at the median and 95th percentile.</p></li><li><p>Recovery rate. The share of runs with a deliberately injected fault, such as a timeout, an empty result or two records that disagree, that end correctly or hand the case to a person.</p></li></ol><p>The release rule is based on these metrics. Set zero tolerance for prohibited actions, and run a sufficient number of tests to ensure statistical significance. Zero violations in n runs supports a maximum true rate of approximately 3 divided by n at 95% confidence. For example, 200 clean runs set a ceiling of 1.5%. However, an agent processing 10,000 requests daily at this rate could still produce 150 violations per day, so the test suite size must reflect expected volume. For other checks, establish a pass^k threshold for each risk tier and a cost ceiling at the 95th percentile.</p><p>Gartner&#8217;s forecast of cancellations gives cost and risk control as two of its causes. These four numbers are the evidence leadership would ask for on both, and the outcome rate covers the third cause, business value, by tying results to system state.</p><h2>A four-phase rollout</h2><p><strong><span>Weeks 1 to 3: Document the rules</span></strong><span>&nbsp;and enable full trace capture for every agent action, including arguments and results, before starting any scoring. A rule is only effective if there is trace data available for evaluation. For each agent, compile a list of prohibited actions based on policy, legal, and security requirements, and assign ownership to someone other than the agent&#8217;s developer. Gather 20 to 50 tasks from real failures and support tickets. According to Anthropic&#8217;s engineers, this sample size is sufficient for initial evaluation, as early changes have significant impact.</span></p><p><strong><span>Weeks 4 to 8: Use code to verify system state and milestones.</span></strong><span>&nbsp;Implement database checks for outcomes and code checks for milestones and prohibited actions using trace data. Run each task five times. Report both the outcome rate and the rule-clean rate together, as the difference between them is often a key finding and may be the first time it is quantified for stakeholders.</span></p><p><strong><span>Weeks 9 to 12: Introduce judge models </span></strong><span>where code-based checks are insufficient, such&nbsp;as for tone, explanation quality, and escalation message content. Assign a separate judge to each criterion and allow &#8220;unknown&#8221; as a possible response. Have two people independently grade 50 traces to compare with the judge&#8217;s assessments. Conduct padding and argument tests before including judge scores in reports.</span></p><p><strong><span>Week 13+: Begin production scoring.</span></strong><span>&nbsp;Sample live traces daily and apply the same code-based checks. Immediately alert on any prohibited action, as issues like payments or data exposures require prompt attention. Convert each failure into a new task. Move tasks that consistently pass into a regression suite, which should be run after any changes to prompts, tools, or models. This approach ensures new models are fairly evaluated before replacing existing ones, which is increasingly important as model updates become more frequent.</span></p><p>Each phase yields a metric that leaders can understand without technical detail. In Phase 2, the gap between outcome rate and rule-clean rate is often the key factor influencing funding decisions.</p><h2>The risk that remains</h2><p>The primary risk lies in the approval process. Leadership may expand an agent&#8217;s authority based on consistently high scores, even though these scores are derived from metrics that cannot reveal violations. The issue is not misinterpretation, but reliance on incomplete evidence.</p><p></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/ai-transparency-is-now-a-product/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://trustedai.recodework.com/p/ai-transparency-is-now-a-product/comments"><span>Leave a comment</span></a></p><div><hr></div><p><em>Trusted AI is written for executives, board members and senior practitioners building, funding or overseeing AI programs. Have a topic you want covered? Reply to any issue or reach out on LinkedIn.</em></p><div><hr></div><h2>Sources</h2><ul><li><p><a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents">Anthropic, Demystifying evals for AI agents</a></p></li><li><p><a href="https://www.anthropic.com/engineering/multi-agent-research-system">Anthropic, How we built our multi-agent research system</a></p></li><li><p><a href="https://arxiv.org/abs/2406.12045">Yao et al., &#964;-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains</a></p></li><li><p><a href="https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027">Gartner, Over 40% of Agentic AI Projects Will Be Canceled by End of 2027</a></p></li><li><p><a href="https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure">Fortune, AI coding tool Replit wiped database, called it a catastrophic failure</a></p></li><li><p><a href="https://www.theregister.com/2025/07/22/replit_saastr_response/">The Register, Replit makes vibe-y promise to stop its AI agents making vibe coding disasters</a></p></li><li><p><a href="https://arxiv.org/pdf/2603.29231">Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents</a></p></li><li><p><a href="https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-agents">Google Cloud, Evaluate Gen AI agents</a></p></li><li><p><a href="https://arxiv.org/pdf/2606.09764">iOSWorld: A Benchmark for Personally Intelligent Phone Agents</a></p></li><li><p><a href="https://arxiv.org/pdf/2508.15760">LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries</a></p></li><li><p><a href="https://arxiv.org/abs/2505.00212">Zhang et al., Which Agent Causes Task Failures and When? (Who&amp;When)</a></p></li><li><p><a href="https://arxiv.org/abs/2505.08638">Deshpande et al., TRAIL: Trace Reasoning and Agentic Issue Localization</a></p></li><li><p><a href="https://www.patronus.ai/blog/introducing-trail-a-benchmark-for-agentic-evaluation">Patronus AI, Introducing TRAIL</a></p></li><li><p><a href="https://arxiv.org/pdf/2606.00765">FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search</a></p></li><li><p><a href="https://arxiv.org/pdf/2606.22329">BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories</a></p></li></ul>]]></content:encoded></item><item><title><![CDATA[AI Transparency Is Now a Product Decision]]></title><description><![CDATA[Disclosure, labeling and content provenance after the rules went live in the US and Europe]]></description><link>https://trustedai.recodework.com/p/ai-transparency-is-now-a-product</link><guid isPermaLink="false">https://trustedai.recodework.com/p/ai-transparency-is-now-a-product</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Fri, 04 Sep 2026 19:13:49 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!mmVJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mmVJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mmVJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png 424w, https://substackcdn.com/image/fetch/$s_!mmVJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png 848w, https://substackcdn.com/image/fetch/$s_!mmVJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png 1272w, https://substackcdn.com/image/fetch/$s_!mmVJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!mmVJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png" width="1456" height="1102" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1102,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1188891,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/213637070?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mmVJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png 424w, https://substackcdn.com/image/fetch/$s_!mmVJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png 848w, https://substackcdn.com/image/fetch/$s_!mmVJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png 1272w, https://substackcdn.com/image/fetch/$s_!mmVJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39f99902-93aa-4d27-8c50-a2d36a397e5a_1920x1453.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>On August 2, 2026, two of the world's largest markets in the US and Europe began demanding a new level of AI transparency. Both want the person on the receiving end of an AI system to be aware that a machine was involved, and the disclosure should be verified by software rather than taken on faith.</p><p>The European instrument is Article 50 of the AI Act. The American counterpart is the California AI Transparency Act, enacted as SB 942 and amended last year by AB 853, which moved the operative date from January 1, 2026 to August to align with the European schedule. State legislatures have started setting their clocks to Central European Time.</p><p>California&#8217;s law impacts any provider of a publicly accessible generative system with more than a million monthly users in the state. Those providers (1) run a free public AI detection tool, (2) offer users a visible disclosure option on a generated image, video and audio, (3) embed a latent provenance disclosure in that content, and (4) push transparency terms down to licensees. Penalties run $5,000 per violation, with each day of a continuing violation counted separately. AB 853 phases in duties for large online platforms and generative AI hosting platforms on January 1, 2027, and for capture device manufacturers a year after that.</p><p>Utah and Washington followed suit in March 2026. HB 276 and HB 1170 build provenance regimes modeled closely on California, using the same threshold of a million monthly users, with Utah&#8217;s provider and platform duties commencing on January 1, 2027. Three aligned state statutes start to feel like a national standard.</p><p>A second and much broader patchwork of state laws covers chatbot disclosure, starting with Utah in May 2024. New York&#8217;s AI companion statute took effect in November 2025 and California&#8217;s SB 243 in January 2026. Oregon and Washington arrive in January 2027. Beyond the states, the FTC's authority over deceptive practices extends to every AI chatbot in the country, and the FCC ruled in February 2024 that an AI-generated voice qualifies as an artificial voice under the TCPA, bringing AI voice calls under the existing robocall consent rules.</p><p>Your location does not exempt you from these requirements. The EU AI Act applies to anyone whose AI outputs are used within the Union, regardless of legal presence. Even companies without European operations must comply with evolving US laws that increasingly align with European technical standards.</p><p>The European version arrived in July.</p><p>On July 24, 2026, Regulation (EU) 2026/1744 appeared in the Official Journal and entered into force three days later. It was the change that European AI programs had been waiting for 18 months. Obligations for standalone high-risk systems under Annex III were moved from August 2026 to December 2027, and obligations for AI embedded in regulated products were moved to August 2028. Teams that had been building toward an August deadline slowed down.</p><p>In contrast, Article 50 was left almost entirely intact by the amendment and went into effect in August. It binds systems that were already in the market on that date, and breaching it carries administrative fines of up to &#8364;15 million or 3% of worldwide annual turnover, whichever is higher. The original shared go-live date for the high-risk regime was set accidentally rather than intentionally. When the anchor moved, a large number of organizations moved their entire program, only to find that one piece had been left behind.</p><p>Many companies approach disclosure as a one-time legal notice drafted by counsel and inserted into the interface. This method is ineffective: users gain no meaningful information, and the business incurs a measurable trust penalty, as recent research demonstrates.</p><p>Disclosure is a product decision that includes determining its placement, level of detail, responsible parties, and persistence throughout the content lifecycle. Product owners should collaborate with risk owners to make these decisions.</p><h2>What applies now</h2><p>Because Article 50 ignores the risk tier entirely, it is the widest-reaching part of the AI Act. A system is covered if it does one of these four things.</p><h4>Interact directly with a person</h4><p>Providers of AI systems designed to interact with people must ensure those people know they are interacting with a machine. This includes chatbots, voice assistants, complaint-handling bots, AI hotlines, coding agents and social media bots that generate rather than template their replies. The exception is narrow. Disclosure can be skipped where the AI nature of the interaction is obvious to a reasonably well-informed, observant and circumspect person as judged against the audience the system will reach.</p><h4>Generate or manipulate synthetic content</h4><p>Providers of systems that produce synthetic audio, image, video or text must mark the outputs in a machine-readable format and make them detectable as artificially generated. Capability alone triggers the requirement. </p><h4>Recognize emotions or categorize people biometrically</h4><p>Deployers must tell the people exposed to those systems that they are running.</p><h4>Publish AI-generated materials on matters of public interest.</h4><p>Deployers must disclose that the content was artificially generated or manipulated. The public-interest text duty falls away where a human has reviewed the content and a named person or entity carries editorial responsibility for it.</p><p>The Commission adopted formal guidelines on all four components in July 2026, and while the guidelines are not legally binding, national authorities will treat them as the reference text.</p><p>Two structural features can cause confusion.</p><p>The first is the split between providers and deployers. The interaction and marking duties sit with providers, meaning whoever develops a system and puts it on the market under their own name or trademark. The emotion-recognition and deepfake duties sit with deployers, meaning whoever uses a system under their own authority. </p><p>Almost every large organization is both. A bank running a third-party chatbot on its service portal is a deployer, while the same bank fine-tuning its own content model and shipping it under its own brand is a provider. The Commission has confirmed that both sets of duties can fall on a single company at the same time.</p><p>The second is agency work. Where a marketing or creative agency operates a system under your responsibility and control, you remain the deployer. Where the agency decides for itself whether and how to use AI in the assignment, it becomes the deployer and assumes the disclosure duty. Most creative services contracts written before 2025 are silent on which arrangement applies.</p><p>Providers of generative systems already on the market have until<strong> </strong>December 2026 to bring machine-readable marking into conformity, and that transition covers marking alone. Everything else applies from day one with no runway. Content both generated and published before August needs no retroactive labeling, while content generated before that date and published after it does.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>The evidence nobody wants</h2><p>Telling people you used AI costs you trust, and this is now one of the better-replicated findings in the applied behavioral literature. Schilke and Reimann ran thirteen experiments across professional settings, from a professor grading work to an investment fund reporting to clients, and published the findings in <em>Organizational Behavior and Human Decision Processes</em>. Holding actual AI use constant, people who disclosed using AI were trusted less than people who stayed quiet. The mechanism they identify is legitimacy, since disclosure signals a departure from what the evaluator expected as proper conduct, and trust follows legitimacy down. Regardless of whether the disclosure was voluntary or mandatory, the finding held.</p><p>Altay and Gilardi found something sharper in the news context. Labeling a headline as AI-generated reduced how accurately readers judged it and their willingness to share it, regardless of whether the headline was true or whether a machine had written it. Probed for a reason, readers said they took &#8220;AI-generated&#8221; to mean fully automated, with no human oversight anywhere in the chain.</p><p>The commercial version is even starker. Researchers at Washington State and Temple ran six experiments with more than a thousand US adults, varying only whether a product description carried the words &#8220;artificial intelligence.&#8221; Purchase intention fell each time, mediated by emotional trust, and fell most sharply for higher-risk products such as cars and diagnostic tools.</p><p>Executives may conclude that disclosing only the minimum required information in the least visible manner is safest. However, this approach is flawed for two reasons.</p><p>First, the cost of concealment is higher. Schilke and Reimann also tested third-party exposure, where the AI use came to light without the actor volunteering it. The damage to trust from being found out was greater than the damage from disclosing. Given that machine-readable marking is now mandatory for providers and that platforms are building detection into their pipelines, the odds of an organization keeping quiet successfully are falling every quarter. The choice is between a smaller cost you control and a larger one that you do not.</p><p>Second, and this is a deeper problem, the trust penalty attaches to a particular reading of the label rather than to the label itself. Readers penalize what they take to be an absent human. If there is a visible human, the shape of the penalty changes.</p><p>Trattner and colleagues at Bergen tested this theory with more than 6,000 participants from the audiences of six major news outlets in the US, UK and Norway. Each saw article previews with images, either bare or carrying a C2PA provenance label at one of three levels of detail. Showing provenance metadata increased how transparent and credible participants judged the image and increased their trust in the outlet publishing it. What separates that finding from the AI-label studies is that a provenance record gives the reader something to inspect. A bare category label leaves them to interpret a word on their own, and the interpretation runs pessimistic.</p><p>In another survey, Toff and Simon found that readers rated AI-labeled articles as less trustworthy even while rating them accurate and fair, and that disclosing the sources used to produce the article reduced the effect</p><p>Product design should ensure that disclosures include information about human responsibility. A label stating only that a machine was involved may lead to negative assumptions. Including who reviewed the content, their responsibilities, and a contact for complaints provides greater transparency and reassurance.</p><h2>Four variables you control</h2><p>Article 50 requires that disclosure information must reach the person &#8220;in a clear and distinguishable manner at the latest at the time of the first interaction or exposure,&#8221; and that it must meet applicable accessibility requirements. Beyond that, the law is quiet on the format, leaving four decisions to the product team.</p><h4>Placement and timing</h4><p>The Commission&#8217;s guidelines eliminate the easy options. Burying disclosure in terms and conditions fails to meet the requirement, as does placing it behind a URL, in technical documentation, or in machine-readable markings. The guidelines recommend combining text, visual and spoken cues (if possible), reasoning that a single channel is easy to miss.</p><p>The practical standard is whether a user can recognize the disclosure within four seconds. In chat interfaces, this means providing an initial statement and a persistent indicator, as opening messages may be missed in longer sessions. Voice channels should present the disclosure within the first few seconds, before substantive content. For published materials, the icon or label must be visible upon first exposure and not obscured by overlays.</p><h4>Granularity</h4><p>This is where the research really helps and where most legal drafts go wrong.</p><p>Prajod and colleagues at CWI Amsterdam ran a controlled study comparing three conditions in a news reading task, running from no disclosure, to a one-line disclosure, to a detailed one that (1) named which production steps used AI, (2) confirmed human oversight and (3) gave a contact for reporting errors. </p><p>Trust scores and subscription rates under the one-line condition matched the no-disclosure condition. The detailed condition surpassed both scores. While the sample was small, it directionally matches the broader literature.</p><p>Approximately two-thirds of participants preferred the detailed disclosure, citing its transparency and the inclusion of an error-reporting contact. Most who favored the shorter version requested a concise disclosure with an option to access more information if desired. A small number of participants did not notice the disclosure.</p><p>The results suggest a two-layer structure. A short, plain first layer conveys the notice at no cost. An interactive second layer holds the detail for the minority who go looking. The EU icon guidance points to the same architecture, noting that when a second layer is used, the icon should indicate that more information is available, and the second layer should be navigable by assistive technology.</p><h4>Who is named</h4><p>The transparency dilemma runs through legitimacy, and legitimacy runs through accountability. A disclosure that (1) names the responsible team or function, (2) states what a human reviewed, and (3) gives a route to report an error does more work per word than a longer one that says only that a model was used.</p><p>In the EU Act, the public-interest text obligation does not apply where content has undergone human review and a natural or legal person holds editorial responsibility for it. The legislator&#8217;s escape hatch is a named accountable human. Companies can build the same structure into disclosures that do not qualify for the exemption to achieve much of the same trust benefit.</p><h4>Survival</h4><p>A disclosure that disappears when content is reshared is ineffective. Disclosures are most important when content reaches audiences outside the original channel. While owned platforms allow for controlled rendering, syndication, partner placements, and social media require testing to ensure disclosures persist throughout distribution.</p><h2>The provenance layer, and its honest limits</h2><p>Machine-readable marking under Article 50(2) has two components.</p><h4>Marking</h4><p>Outputs must carry marks structured so that software can find and extract them without a human in the way. The Act prescribes no technique. Watermarks, signed metadata, cryptographic provenance, and logging and fingerprinting all qualify, alone or in combination, as long as the solution meets four quality requirements. They are (1) effectiveness, (2) reliability, (3) robustness and (4) interoperability. As an example, a watermark that a screenshot erases or a social upload strips fails the robustness hurdle.</p><h4>Detection</h4><p>Providers must also make available a mechanism that allows people, authorities, researchers, fact-checkers, and journalists to verify whether a piece of content originated from the system. The Code expects the mechanism to be free and publicly accessible in most cases. While many organizations have scoped the marking work, most have skipped the detection dimension entirely.</p><p>The dominant technical answer to the marking requirement is C2PA Content Credentials (version 2.4, April 2026), with more than 6,000 members and affiliates in the coalition and a full conformance program running through the year. The latest version has been moving toward international standardization, and JPEG Trust has taken the earlier version into ISO/IEC 21617-1. The California, Utah and Washington statutes rely on a similar approach and allow a single technical decision to satisfy multiple regimes simultaneously.</p><p>Because a program built on optimism often fails the first real test, it is helpful to address the limits.</p><p>C2PA manifests are stored in the file container with EXIF and XMP metadata. Most social platforms re-encode images and videos on upload, often removing metadata unless specifically preserved. As of mid-2026, major platforms typically strip or do not retain C2PA manifests, except in limited cases where the platform uses the credential for labeling before discarding it. Organizations should test their own distribution channels rather than relying solely on vendor documentation.</p><p>The AI Act creates a marking duty for providers but does not prohibit third parties from removing, altering, or overwriting those marks before content circulates. This is a striking gap compared to copyright law, where knowingly stripping rights management information is an offense with its own remedy. While robustness requirements cover part of the exposure, a determined adversary gets through them. California and Utah are closing that gap and require platforms, starting in January 2027, to prevent the knowing stripping of compliant provenance data or digital signatures from content they distribute.</p><p>The evidentiary logic of provenance is asymmetric and matters to anyone who thinks a detection tool solves the problem. While valid credentials on a file tell you something, absent credentials tell you almost nothing because most authentic content in circulation carries no credentials at all and never did. A verification workflow that treats absence as suspicion will drown in false positives.</p><p>The Code of Practice on Transparency of AI-Generated Content was published in June 2026, and the Commission and the AI Board have confirmed it as an adequate voluntary tool. About 190 companies and organizations signed the code in the first 60 days, which changes the enforcement posture. Supervisors assess signatories against the commitments rather than running a free-standing technical review of their solution. Non-signatories demonstrate adequacy by other means, individually, to whichever market surveillance authority comes asking, and forgo whatever mitigating weight adherence carries when a fine is set. The Dutch data protection authority has publicly advised providers to sign.</p><p>The Code organizes its commitments in four parts.</p><ul><li><p>Implement a marking solution, which for most content types means signed metadata combined with an imperceptible watermark.</p></li><li><p>Make a detection solution available, generally free and publicly accessible.</p></li><li><p>Meet the four quality requirements, with an interoperability solution due in February 2027.</p></li><li><p>Document a compliance process, with regular testing, trained staff and cooperation with market surveillance authorities.</p></li></ul><p>With regard to outsourcing, a provider can build on a watermark embedded by the underlying foundation model or buy the marking from a specialist vendor. The legal duty does not travel with the work. The provider must vet the solution and stand behind it, in the same way a data controller stands behind a processor it appointed.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/ai-transparency-is-now-a-product?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/ai-transparency-is-now-a-product?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>The exceptions, and how narrow they are</h2><p>Three statutory exceptions apply to the marking duty. First, systems performing an assistive function for standard editing are out of scope. This includes routine preparation of existing content, such as grammar, spelling, house style, minor cropping or compression. Second, machine translation is treated as an editorial operation. Third, systems that do not substantially alter the input data or its meaning are out, covering transcription, color correction and light stabilization. Systems authorized by law for criminal investigation are also exempt.</p><p>The list of what remains in scope is longer and includes AI-generated summaries, substantive rewrites that reshape a text, face replacement, voice synthesis, and composite images of events that never happened.</p><p>The guidelines add proportionality carve-outs. Systems embedded in closed physical products from which output cannot leave are exempt. In film, animation, gaming and advertising, only the finished consumer-facing deliverable requires marking, so intermediate outputs within a closed production pipeline can remain unmarked. </p><p>Internal business applications can qualify, but only if three conditions are met. The system runs (1) inside a controlled organizational perimeter, (2) output is not destined to circulate beyond it, and (3) safeguards against foreseeable misuse are in place, such as environment isolation and role-based access. Anything consumer-facing loses the safeguard, and few enterprise tools can promise the materials never leave the building.</p><h2>Four disclosure patterns</h2><p>To help illustrate the challenge, consider a composite mid-market European retailer with roughly &#8364;2 billion in revenue, selling across seven member states. Under Article 50, four of its systems land in four different areas, and the design has to change for each case.</p><h4>Pattern one: the conversational front door</h4><p>The retailer runs a customer service assistant on its site and app, built on a third-party platform and branded as its own.</p><p>The initial consideration is whether the retailer is classified as a provider or deployer under the Act. If the platform vendor markets the system under its own name, the vendor is the provider and assumes the interaction duty. If the retailer brands the system as its own, it becomes the provider. This designation should be clearly defined in the vendor contract, not left to informal communication.</p><p>The disclosure should (1) begin the session with clear language, (2) remain visible throughout the session, and (3) specify how to reach a human representative. Providing this route addresses the requirement and offers accountability, which helps mitigate trust concerns. Companies should avoid including the assistant's name or photograph, as the Commission has indicated that realistic designs are less likely to qualify for the obviousness exception. Additionally, combining a human-like persona with a disclosure may cause confusion.</p><p>It is important to consider the handoff mechanics. If a bot handles the opening and a person takes over, the disclosure duty still applies. Where a person reviews the model&#8217;s suggestion and makes their own call, Article 50(1) does not apply. A human who pastes the model&#8217;s draft without reading it is not, in any sense, in the loop under the guidelines.</p><h4>Pattern two: the content pipeline</h4><p>The brand team generates product imagery, lifestyle photography and campaign copy with generative AI tools.</p><p>This output is not a deepfake under AI Act guidelines, and the deployer labeling duty under Article 50(4) does not apply. While this is the correct legal interpretation, two factors still apply. If the retailer built or branded the generation tool, it is a provider and owes marking and detection on the outputs. If a synthetic model, a synthesized voice, or an AI-altered image of a real person, place, or event is used in a campaign, the deepfake duty attaches, and the EU icon set is available with the placement rules specified in the Code.</p><p>Operationally, maintaining an inventory is essential. The retailer must track which assets in its digital asset management system are machine-generated, which are machine-altered and to what extent, and which tools produced them. Capturing this metadata at the time of generation is cost-effective and avoids the expense and inaccuracy of retrofitting later.</p><p>It is important to review agency contracts. If the creative agency decides whether to use generative tools, the agency is considered the deployer and is responsible for disclosure. While this may be legally sufficient, it poses reputational risks since the retailer&#8217;s logo appears on the final product.</p><h4>Pattern three: the agent acting outward</h4><p>The retailer runs a scheduling agent that books in-home appliance installations by calling and emailing customers to arrange times. </p><p>This pattern creates the widest gap between what teams assume and what the guidelines say. Article 50(1) covers systems intended to interact directly with people. For agents, the Commission&#8217;s guidelines extend to systems capable of interacting with a person while carrying out their assigned work, including scheduling, handling correspondence, negotiating, entering into agreements, and completing purchases. The trigger is capability. The analysis cannot stop at the primary use case; it must cover every scenario in which contact with a person is reasonably foreseeable.</p><p>The guidelines go further. An agent capable of interacting with other agents must account for downstream agents reaching a human, thereby extending the duty to indirect contact. Where the provider cannot predict at design time whether contact will occur, the agent must be built at the architecture and instruction levels to identify itself in every foreseeable case, including when the person on the other side is acting on behalf of a company.</p><p>There is a disclosure the other way as well. The guidelines say the person directing the agent should also be told at (1) authorization requests, (2) status updates, (3) validation checkpoints and (4) the start of each new exchange. An agent that quietly accumulates authority across a long-running session is a governance problem long before it is a compliance problem.</p><p>For the retailer, the identification line needs to be included in the agent&#8217;s system instructions, so it can be tested as a behavior and logged in the same trace that records the agent's actions.</p><h4>Pattern four: the internal system whose output escapes</h4><p>Analysts at the retailer use an internal assistant to draft summaries of supplier performance and market conditions.</p><p>At first glance, this scenario qualifies for the internal carve-out, as the system operates within organizational boundaries and outputs are intended for colleagues with controlled access. However, the third condition is frequently unmet, as content can easily be shared externally, and safeguards are often insufficient in practice.</p><p>The control belongs on the edge rather than on the tool. Anything moving from an internal system to a customer, supplier, regulator, or public channel passes through a defined step in which its origin is recorded, marked, and labeled according to the destination. That step doubles as your evidence file when someone asks about the origin of a specific paragraph.</p><h2>Owning it</h2><p>A significant challenge is that Article 50 does not align neatly with any existing function. Legal teams interpret the requirements but cannot draft disclosures suitable for chat interfaces. Product teams manage the interface, while Marketing commissions content without insight into the underlying processes.</p><p>Four artifacts help correct the situation, and a program can address them in 3-6 months.</p><h4>A system register with the role recorded per system </h4><p>Provider, deployer, or both. The status is reviewed whenever a vendor relationship changes, branding changes, or a system gains generative capabilities.</p><h4>A content register keyed to the systems</h4><p>What was generated, by which tool, with what marking, and who reviewed it.</p><h4>A disclosure pattern library</h4><p>Develop approved first-layer text and second-layer content for each interaction type. Product copywriters should draft these materials, with legal review rather than authorship, and user testing is essential. Since most users will not access the second layer, it should be tested independently. Those who do are often the most discerning and require particular attention.</p><h4>An evidence file</h4><p>Screenshots of live disclosures, the marking standard you selected and why, robustness test results, the reasoning behind any exception you claimed, and the vendor contracts that allocate the roles. This package is much harder to reconstruct after an inquiry lands.</p><p>Three metrics are worth reporting upward. </p><ol><li><p>Coverage is the share of in-scope systems carrying a verified compliant disclosure. </p></li><li><p>Survival is the share of published assets whose provenance data is still intact at the far end of your real distribution path. </p></li><li><p>Latency is the lag between a system gaining a new capability and the disclosure being updated.</p></li></ol><h2>The part that reaches the customer</h2><p>European enforcement will be uneven for a while. Member states missed the original August 2025 deadline to designate national competent authorities by a wide margin, and designation remains patchy. Identical conduct will trigger an inspection in one member state but not in another during the early cycles. Companies should plan against the strictest national interpretation because a single well-resourced regulator will set the working standard, as we saw under GDPR.</p><p>The monetary fine is the wrong thing to organize around. Article 50 is the first part of the AI Act that reaches customers directly with no regulator standing in between. Every time someone opens a chat window on your site or scrolls past an image your brand team made, they run a small private self-assessment of whether you are being straight with them. The high-risk regime arrives in December 2027 and will test whether your documentation withstands professional scrutiny. Article 50 is running a harder test today with an audience that never reads your documentation.</p><p>A disclosure written as legal text will satisfy nobody. You absorb the full trust penalty and collect no credit for being honest. Designing the disclosure (1) costs less, (2) keeps open the option of being believed later, and (3) produces the content inventory and provenance plumbing that the next three years of regulation will require regardless.</p><p>The calendar is crowded. Machine-readable marking for generative systems already in the European market becomes enforceable on December 2, 2026. In January 2027, the California and Utah platform duties take effect, and a large online platform will owe users a duty to detect, disclose, and inspect provenance data, as well as a duty not to knowingly strip it. By February 2027, Code of Practice signatories are expected to have an interoperability solution in place.</p><p>Customers will find out that your systems are AI. Marking is mandatory now, and detection is being built into the distribution layer. The answer reaches them whether you supply it or not. What remains open is whether they hear it from you first, in language you chose, from someone who will answer for it.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/ai-transparency-is-now-a-product/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/ai-transparency-is-now-a-product/comments"><span>Leave a comment</span></a></p><div><hr></div><p><em>Trusted AI is written for executives, board members and senior practitioners building, funding or overseeing AI programs. Have a topic you want covered? Reply to any issue or reach out on LinkedIn.</em></p><div><hr></div><h2>Sources</h2><p><strong>Regulation and guidance</strong></p><ul><li><p>Regulation (EU) 2026/1744 (Digital Omnibus on AI), published in the Official Journal 24 July 2026, in force 27 July 2026</p></li><li><p>EU AI Act (Regulation (EU) 2024/1689), Article 50 and Article 113</p></li><li><p>European Commission, <em>Guidelines on transparency obligations for providers and deployers of AI systems</em>, adopted 20 July 2026</p></li><li><p>European Commission, <em>Code of Practice on Transparency of AI-generated Content</em>, final version 10 June 2026, and the associated EU icon set</p></li><li><p>California AI Transparency Act (SB 942 as amended by AB 853), operative 2 August 2026</p></li><li><p>Utah HB 276, Digital Content Provenance Standards Act, signed 24 March 2026, effective 1 January 2027</p></li><li><p>Washington HB 1170, AI-modified content, signed March 2026</p></li><li><p>Utah AI Policy Act (SB 149, amended by SB 226); California SB 243; New York General Business Law Article 47</p></li><li><p>FCC Declaratory Ruling 24-17 (8 February 2024), AI-generated voices under the TCPA</p></li></ul><p><strong>Research cited</strong></p><ul><li><p>Schilke, O. and Reimann, M. (2025). The transparency dilemma: how AI disclosure erodes trust. <em>Organizational Behavior and Human Decision Processes</em>, 188, 104405</p></li><li><p>Altay, S. and Gilardi, F. (2024). People are skeptical of headlines labeled as AI-generated, even if true or human-made, because they assume full AI automation. <em>PNAS Nexus</em>, 3(10)</p></li><li><p>Trattner, C., Forstner, S. L., Starke, A. D. and Knudsen, E. (2026). C2PA provenance labels increase trust in digital news platforms across Western countries. <em>ICWSM</em>, 20(1), 2267 to 2279</p></li><li><p>Prajod, P. et al. (2026). Full disclosure, less trust? How the level of detail about AI use in news writing affects readers&#8217; trust. arXiv:2601.09620</p></li><li><p>Toff, B. and Simon, F. M. (2025). The dilemma of AI disclosure for audience trust in news. <em>International Journal of Press/Politics</em>, 30(4)</p></li><li><p>Cicek, M., Gursoy, D. and Lu, L. (2024). Adverse impacts of revealing the presence of AI technology in product and service descriptions on purchase intentions. <em>Journal of Hospitality Marketing &amp; Management</em>, 34(1)</p></li></ul>]]></content:encoded></item><item><title><![CDATA[Agent Identity, Not Model Security, Is the Missing Control Plane]]></title><description><![CDATA[What your agents can access, who authorized it, and why no one can fully explain it]]></description><link>https://trustedai.recodework.com/p/agent-identity-not-model-security</link><guid isPermaLink="false">https://trustedai.recodework.com/p/agent-identity-not-model-security</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Sat, 15 Aug 2026 16:39:01 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!9Fxx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9Fxx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9Fxx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg 424w, https://substackcdn.com/image/fetch/$s_!9Fxx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg 848w, https://substackcdn.com/image/fetch/$s_!9Fxx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!9Fxx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!9Fxx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/efb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2521869,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/209666941?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!9Fxx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg 424w, https://substackcdn.com/image/fetch/$s_!9Fxx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg 848w, https://substackcdn.com/image/fetch/$s_!9Fxx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!9Fxx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefb52971-3be3-48a7-86d7-87ea8d446376_5173x3449.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Most leadership teams can readily state how many AI agents are running. However, when asked about a specific agent in production that interacts with real business data, their confidence dips. To manage risk, organizations must be able to answer four key questions about every agent.</p><ol><li><p>What is it called?</p></li><li><p>Who owns it? </p></li><li><p>What can it reach? </p></li><li><p>When is it renewed?</p></li></ol><p>Most organizations require at least a week to answer these questions fully. Many cannot even identify an agent by name, as it is often an API key shared across multiple services. This gap is fundamental to AI governance discussions but receives little attention.</p><p>This issue deserves board-level attention and should not be treated as a routine security backlog. Investments in AI fairness, explainability, and policy compliance all rely on knowing exactly what each agent can access. This assumption is foundational to effective governance. </p><p>The scale of this gap is significant.</p><p>A survey of large US enterprises found that 95% run agents that autonomously perform IT or security work, yet only 22% claim full visibility into the non-human identities behind them. Research published by the Cloud Security Alliance reported 78% of organizations had no documented policy for creating or removing AI identities, with 51% having no clear ownership over those identities, and only 20% maintained a formal process to revoke access once an agent was retired. </p><p>IT leaders surveyed by VentureBeat reported declining confidence in their programs over time. The percentage describing their AI deployment as mature dropped from 40% in 2025 to 23% in 2026. Non-human identity governance was identified as the primary gap influencing this change.</p><p>This pattern is common in technology, where control models designed for one type of actor are often applied to new contexts without adaptation. This issue of TrustedAI examines the resulting mismatch and strategies to address it.</p><h2>Model Safety Governs What It Says. Identity Governs What It Can Do.</h2><p>Most AI safety investments over the past two years have focused on the model itself. Alignment efforts, output filtering, red teaming, and evaluation harnesses are valuable for preventing harmful outputs. However, these measures do not address the consequences if an agent is compromised or manipulated, nor what actions it could take in such scenarios.</p><p>In most enterprises, agents can perform any action their credentials allow, making this fundamentally an identity issue rather than a model issue. This risk remains regardless of model performance.</p><p>Prompt injection clearly illustrates this challenge. Since instructions and data share the same channel in a language model, an attacker does not need to breach the system directly. A simple message left for the agent to read can suffice. For example, a malicious note posted to a GitHub server running Model Context Protocol (MCP) contained hidden instructions that hijacked a connected agent and extracted data from private repositories, without the attacker ever directly accessing the target system.</p><p>Research attributes 61% of agent security incidents to credentials with excessive access, while only 34% are linked to prompt injection. This highlights the primary area of exposure. For example, if an injected instruction hijacks an agent with only calendar access, the impact is minor. However, if the agent can write to a production database, the consequences are far more severe and may require legal intervention.</p><p>This is an example of the confused deputy problem, which has existed for decades. A privileged program can be manipulated by a less privileged actor to misuse its authority. With agentic AI, the scope of inherited privileges expands. When an agent holds broad credentials, every input channel&#8212;such as email, support tickets, documents, or outputs from other tools&#8212;becomes a potential entry point. In February 2026, the Cline coding compromise occurred when a developer authorized Cline to act on their behalf. Once compromised, Cline extended that authority to another agent without the developer&#8217;s knowledge or approval. This was a classic access control failure.</p><p>While model-level defenses are important, practical risk mitigation depends on controlling the credential: its scope, duration, and the ability to audit its actions after the fact.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>Why a Service Account Is the Wrong Template</h2><p>Security teams have long managed machine credentials using service accounts, API keys, and automation tokens. It may seem reasonable to apply the same approach to agents, but this is a critical mistake for reasons that warrant attention.</p><p>A service account executes predefined instructions, performing the same tasks daily and connecting to the same systems unless the code changes. In contrast, an agent interprets goals, determines its own sequence of actions, and may request new system access as needed. Managing credentials for deterministic programs differs fundamentally from managing those for agents that reason and act autonomously.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!pdLe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pdLe!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png 424w, https://substackcdn.com/image/fetch/$s_!pdLe!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png 848w, https://substackcdn.com/image/fetch/$s_!pdLe!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png 1272w, https://substackcdn.com/image/fetch/$s_!pdLe!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!pdLe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png" width="981" height="598" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:598,&quot;width&quot;:981,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:139559,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/209666941?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!pdLe!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png 424w, https://substackcdn.com/image/fetch/$s_!pdLe!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png 848w, https://substackcdn.com/image/fetch/$s_!pdLe!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png 1272w, https://substackcdn.com/image/fetch/$s_!pdLe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0083c0bb-8e86-45fa-8db8-84c762707ca3_981x598.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Four key properties render the legacy approach ineffective for agents.</p><h4>First, agents lack natural lifecycle triggers</h4><p>Traditional identity management is based on the human worker lifecycle, with events such as hiring, transfers, and departures prompting specific actions. Agents have no hire date, manager, or exit interview. In large enterprises, the ratio of non-human to human identities ranges from 50 to 140 to one. Traditional access reviews were not designed for this scale or speed.</p><h4>Second, borrowed credentials undermine attribution</h4><p>When agents use shared service accounts or human logins, the audit trail becomes unreliable. It is often impossible to determine whether an action originated from a person or from an automated process. According to a 2026 Gravitee survey, only 24% of organizations have full visibility into agent-to-agent communications, and over half of agents operate without security oversight or logging. Attribution failures are common rather than exceptional.</p><h4>Third, agents can create other agents</h4><p>A root agent may delegate tasks to sub-agents, which can, in turn, delegate further. Authority typically propagates down this chain, and most architectures do not verify permissions at each level, allowing unchecked access.</p><h4>Fourth, the scale is unprecedented</h4><p>Non-human identities outnumber human ones by an average of 45 to 1, rising to 144 to 1 in cloud-native environments (Cloud Security Alliance, May). Additionally, 71% of these identities exceed recommended rotation periods, increasing risk. Each unrotated credential extends the window for potential compromise.</p><p>Major identity vendors have recognized this need. Microsoft&#8217;s Entra Agent ID, Okta&#8217;s Universal Directory, and Google Cloud&#8217;s Agent Identity for Vertex AI all treat agents as distinct identity principals, separate from human users and service accounts. Each platform is based on the premise that agents require unique identities. However, most enterprise governance programs have not yet adopted this approach.</p><h2>What Happened at Salesloft</h2><p>A notable example involves not an autonomous agent, but a simple chatbot with an integration. All the controls discussed here would have prevented the incident.</p><p>In August 2025, a group used OAuth tokens stolen from Salesloft&#8217;s Drift chatbot integration to reach the Salesforce environments of more than 700 organizations, including Cloudflare, Google, Palo Alto Networks, Proofpoint and Zscaler. The attackers deployed no malware. They used the stolen tokens and standard APIs to query and export records. In several environments, they went further and pulled AWS access keys, Snowflake credentials and stored passwords out of Salesforce records where they had been left in plain text. Exports ran through Salesforce&#8217;s bulk data API and completed in under three minutes per target. The attackers deleted their own query logs to slow detection.</p><p>Two design choices turned a chatbot compromise into a breach spanning hundreds of companies. First, Salesloft had granted the Drift integration access well beyond what a support chatbot requires, and the attackers who stole the tokens inherited that full privilege intact. Second, Drift issued a separate OAuth token for every individual user rather than a single token for the integration as a whole. As a result, wherever a Salesforce administrator had connected the tool, the stolen token provided administrator-level access. A design intended to give each user finer-grained control ended up multiplying the number of ways into the system.</p><p>To stop the damage from getting worse, once Salesloft detected the anomalous activity, they revoked every Drift OAuth token in a single action and pulled the integration from general availability. The response operated entirely at the credential level, and the fix was revocation, applied to an identity.</p><p>Most organizations would struggle to revoke all credentials associated with a single agent or integration in one action, and to do so with confidence. The issue is not inadequate tooling, but rather the absence of a suitable registry.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/agent-identity-not-model-security?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/agent-identity-not-model-security?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Six Disciplines, Rebuilt for Agents</h2><p>Identity and access management covers six functions for every human employee: registration, credentialing, authorization, delegation, monitoring, and offboarding. These functions remain essential for agents, but each must be redesigned to address assumptions that do not apply to autonomous agents.</p><h4>Registration</h4><p>No agent should be deployed to production without an entry in a central registry and an assigned human owner. The registry must capture the model and version, agent purpose, data classes accessed, tools invoked, spending limits where applicable, and an expiry date. Platform administrators require a form-based submission process, while engineering teams need an API-based workflow integrated into deployment pipelines. Both should feed into a unified, governed inventory, regardless of the underlying platform.</p><h4>Credentialing</h4><p>Eliminating static, long-lived secrets is the most impactful change for most organizations. Google Cloud&#8217;s Agent Identity assigns each agent a cryptographic identity based on the SPIFFE standard, using certificates valid for 24 hours and binding access tokens to specific certificates to prevent token replay. Deployments using SPIRE rotate credentials hourly, eliminating static keys in configuration files. Okta&#8217;s Cross App Access and the ID-JAG protocol extend OAuth token exchange under an open, industry-backed specification. Microsoft&#8217;s Entra Agent ID uses its own OAuth flows with federated credentials and delegated access. While a dominant standard has yet to emerge and deep integration carries some risk, it is clear that static secrets in configuration files represent a failure mode these standards aim to eliminate.</p><h4>Authorization</h4><p>An agent&#8217;s effective permissions should reflect the intersection of its own allowed actions and those of the invoking user. Neither limit alone is sufficient. Industry threat modeling identifies agent impersonation as a distinct risk, recommending controls such as trusted registries, cryptographic identities, and narrowly scoped, short-lived tokens. Identity verification alone is not enough; authorization decisions must be enforced separately at runtime, based on context, rather than assumed from identity alone.</p><h4>Delegation</h4><p>When one agent delegates work to another, the authority granted must be explicit, more limited than the parent agent&#8217;s, and logged at the time of delegation. Assigning distinct identities to each agent in the chain enables full auditability. A proper trace should identify the root agent, each sub-agent, the resources accessed, and the authority issuing each identity. Without this structure, multi-agent workflows lack accountability at each step.</p><h4>Monitoring and attribution</h4><p>Regulators will test this discipline directly, and retention requirements already differ by framework. Organizations subject to SOX generally need to retain operational logs for at least 366 days and audit work papers for seven years. In contrast, HIPAA calls for six years. PCI DSS version 4.0 generally requires twelve months of logs with three months immediately accessible, and it now presses the same least-privilege and periodic access-review obligations onto non-human accounts that have always applied to human ones. European financial institutions carry a parallel obligation under DORA, which folds AI systems into ICT risk management covering access controls, audit logs, and third-party provider assessment. The EU AI Act sets a six-month floor specifically for high-risk systems under Article 12. Many enterprises operating across more than one of these regimes have standardized on seven years to avoid managing separate schedules.</p><h4>Revocation and retirement</h4><p>Agents lack the natural triggers associated with human departures. Credential inactivity, when a key has not been used within a set period, is one useful signal. Changes in credential scope are another. These events should trigger governance reviews routed to the owning team for reauthorization. Automating these signals within the existing secrets platform addresses most gaps without introducing manual steps that may be overlooked.</p><h2>Whose Job This Is</h2><p>Many enterprises struggle to determine ownership of agent identities. Identity and access management teams focus on human access reviews, while the teams developing and operating agents are responsible for their behavior but lack authority to issue or revoke credentials.</p><p>The split that works in practice looks like this.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!apkJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!apkJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png 424w, https://substackcdn.com/image/fetch/$s_!apkJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png 848w, https://substackcdn.com/image/fetch/$s_!apkJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png 1272w, https://substackcdn.com/image/fetch/$s_!apkJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!apkJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png" width="916" height="583" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:583,&quot;width&quot;:916,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:67458,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/209666941?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!apkJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png 424w, https://substackcdn.com/image/fetch/$s_!apkJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png 848w, https://substackcdn.com/image/fetch/$s_!apkJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png 1272w, https://substackcdn.com/image/fetch/$s_!apkJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa871b7b2-a9f7-4ab4-834a-5a5ec2ad70a8_916x583.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The authority to approve or revoke an agent&#8217;s scope should reside with identity and security governance. Platform and engineering teams manage daily operations within this framework. Ownership must extend across the entire identity lifecycle, from registration to retirement.</p><p>Agent credentials should be issued by the same authority responsible for human credentials, not through separate processes established for convenience. Board-level oversight of AI governance is increasing rapidly, making this requirement essential.</p><h2>Putting This Into Practice</h2><p>Effective programs begin with credential inventories, not organizational charts. Asking business units to declare agents often results in incomplete lists. The true inventory is found in OAuth grants, cloud roles with model API access, and unrotated secrets in vaults. This approach may still miss browser extensions and low-code tools. A 2026 survey found the average enterprise has 37 deployed agents, with over half lacking security oversight or logging.</p><p>Inventories quickly become outdated, often within a month. Treating discovery as a one-time audit leads to stagnation, as agents are created faster than audits can track. Enforcing registration at the point of creation ensures the inventory remains current.</p><p>Organizations should prevent new agents from reaching production without registration, even before cataloging existing agents. Implementing a registration gate in the deployment pipeline limits unregistered agents. Prohibiting new long-lived credentials ensures every agent follows a documented issuance process. Currently, only 14% of organizations deploy agents with full security or IT approval, while 82% of executives express confidence in their policies. The gap between confidence and actual control is what a deployment gate addresses.</p><p>A rule-based approach to ownership is more effective than manually assigning individuals to orphaned agents. Defaulting ownership to the deploying team, with automatic escalation to a manager when necessary, can resolve much of the backlog. Succession planning is essential, as agents often retain access after their creators have left, leading to most documented lifecycle failures.</p><p>Organizations should validate two capabilities in live systems: (1) revoke a production agent and measure how quickly its access is removed across all systems, and (2) retrieve a week of activity for another agent and reconstruct who authorized each action. These exercises often reveal gaps, which can be addressed by automating credential inactivity and scope-change detection within the existing secrets platform.</p><p>Assigning clear responsibility for governance is essential. Extending the human access review cycle to include agents is cost-effective and integrates the work into established metrics and budgets.</p><p>Finally, five measures need to be tracked and presented to the board at every meeting. </p><ol><li><p>The share of agents with a named owner. </p></li><li><p>The share holding a unique attributable identity rather than a shared credential. </p></li><li><p>The median lifetime of an agent credential. </p></li><li><p>The elapsed time to revoke one on demand. </p></li><li><p>The count of standing access grants older than ninety days. </p></li></ol><p>No new technology is needed to create this inventory. It simply requires organizational commitment.</p><h2>The Real Risk</h2><p>A significant failure often involves a standard agent acting within its credentialed permissions, operating faster than human review processes can track, and leaving the organization unable to reconstruct its actions afterward.</p><p>The audit programs most enterprises already run do not help. SOC 2 Type II examinations scope access review around human users by default. ISO 27001&#8217;s access management controls, written into Annex A.9, assume a human principal and rarely get reinterpreted for credentials that were never provisioned through the formal directory. PCI DSS 4.0 has tightened requirements on system accounts in general, but the requirements for an agent built on a large language model remain unclear in practice.</p><p>An AI agent credential can pass through all three of those audits without anyone examining it. It was likely never provisioned through formal identity management and never shows up in a standard access review. A Cloud Security Alliance survey found that 84% of organizations could not currently pass a compliance audit focused specifically on agent behavior and access controls, and only 23% had anything resembling a formal strategy for agent identity.</p><p>The agent identity gap is evident in your current organizational audits. It is important that someone outside your team does not discover it first.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/agent-identity-not-model-security/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/agent-identity-not-model-security/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Your Agents Have Authority Nobody Granted Them]]></title><description><![CDATA[Delegation, scope and the governance question that models never forced you to answer]]></description><link>https://trustedai.recodework.com/p/your-agents-have-authority-nobody</link><guid isPermaLink="false">https://trustedai.recodework.com/p/your-agents-have-authority-nobody</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Fri, 31 Jul 2026 14:55:13 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!1NmH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1NmH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1NmH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg 424w, https://substackcdn.com/image/fetch/$s_!1NmH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg 848w, https://substackcdn.com/image/fetch/$s_!1NmH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!1NmH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!1NmH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg" width="1000" height="600" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:600,&quot;width&quot;:1000,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:523174,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/208384506?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1NmH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg 424w, https://substackcdn.com/image/fetch/$s_!1NmH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg 848w, https://substackcdn.com/image/fetch/$s_!1NmH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!1NmH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe47e3064-064a-4d70-b996-d06956e706b9_1000x600.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><p>In April 2026, an AI coding agent at the small software firm PocketOS deleted a production database and all the backups stored with it. The incident lasted less than 10 seconds.</p><p>The Cursor agent, using a frontier model, encountered a credential issue during a routine task in a staging environment. It independently deleted a storage volume after finding an API token with sufficient access. There was no confirmation prompt or check to determine if the volume contained production data, and no separation existed between test and production environments. The cloud provider stored backups in the same volume as the protected data, which the founder did not know. Consequently, the latest recoverable backup was three months old, resulting in the loss of all reservations from the past three months.</p><p>When asked to explain its actions, the agent &#8220;confessed&#8221; that deleting the database volume was the most destructive and irreversible option available, and that no one had instructed it to delete anything. It acted independently to address a problem it did not fully understand.</p><p>No model failed a benchmark, and outputs were neither biased nor erroneous. There was no prompt injection, adversary, or data leak. The model operated as designed: it formed a plan, selected a tool, and executed. The failure resulted from the agent having authority to delete production data, even though no human granted that authority. When questioned about excessive access, the founder clarified that the agent was not given access; it discovered it.</p><p>This incident highlights a significant AI governance issue that current standard toolkits largely fail to address.</p><h2>Existing frameworks were designed for a different context</h2><p>Looking at the AI governance program most large organizations have assembled over the past three years, it incorporates (1) a model inventory, (2) a risk assessment that runs at project intake, (3) policies covering acceptable use, data handling and disclosure, (4) an evaluation regime that scores accuracy, bias and safety before release, (5) a monitoring stack that watches for drift, and (6) a human oversight requirement that provides a review step before a decision takes effect.</p><p>Each control assumes that the AI system produces an output and that a person decides how to act on it. For example, the model scores a loan application, and a human approves the loan; the model drafts a letter, and a human sends it. Governance focuses on the output, with the control point being when a person acts on that output.</p><p>Agents change this dynamic. An agent acts on a goal rather than a question. It determines steps, selects tools, executes actions, reviews results, and decides next steps. Without proper guardrails, there is no output awaiting human review because the action itself is the output. By the time anyone reviews the process, the invoice is paid, the record is updated, the message is sent, or the database is deleted.</p><p>This shift changes the focus of governance. When a model generates an answer, governance centers on answer quality. When an agent acts, governance must address the permissions that enable those actions. These are distinct control points, often managed by different teams. Traditionally, developers oversee answer quality, while system administrators manage permissions, often without realizing this includes agent oversight.</p><p>The data confirms this challenge. In April 2026, the Cloud Security Alliance reported that 53% of organizations said their AI agents had exceeded intended permissions, and only 8% said their agents never do. Additionally, 47% experienced a security incident involving an agent in the previous 12 months. Only 15% reported that most agents had a defined owner. Half had at least partially documented governance policies for agents, but only 31% had formally adopted them.</p><p>Another April 2026 report by Rubrik found that 86% expected agents to outrun their own organization&#8217;s controls within a year, and Gartner has predicted that more than 40% of agentic projects will be canceled by the end of 2027, citing inadequate risk controls, cost, and unclear value as primary causes. Model capability notably does not appear among the reasons.</p><p><span>For senior leaders, the key question is not whether agents are safe, but&nbsp;</span><strong><span>who determined what agents are permitted to do and where this decision is documented.</span></strong></p><p>In most organizations, the answers are often &#8216;nobody&#8217; and &#8216;nowhere.&#8217;</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>Authority is the primary object of governance</h2><p>Organizations have long-standing processes for granting authority to entities acting on their behalf.</p><p>Large organizations run a delegation of authority policy. It says who can sign a contract and up to what value, who can approve an invoice, who can commit capital, who can settle a claim, or who can waive a fee. The limits are graduated based on role. The policy names roles, states what each may do, sets a ceiling, and describes what must be escalated. Internal audit tests it every year, and nobody finds it controversial since it&#8217;s one of the oldest controls available.</p><p>The key is to apply a similar perspective to the software deployed in your workflows. For example, an agent that reconciles invoices exercises authority over accounts payable, or an agent that adjusts inventory levels exercises authority over working capital. When changes to a configuration in a production system for a person require a named human and a change ticket, a similar level of authority is needed for an agent.</p><p>In reality, these agents are typically not included in formal delegation of authority policies. Most receive permissions through technical decisions, often by engineers, enabling integrations with available credentials.</p><p><span>This perspective is a core component of the trusted AI thesis.&nbsp;For agents, the unit of governance is not the model itself but the grant of authority through a scoped, revocable, time-bound mandate that specifies what the agent may access, what it may spend, what must be escalated, and who is accountable.</span></p><p>This approach shifts the control point to before the action or decision, which is essential for irreversible actions. It provides a mandate that business owners can approve without technical expertise since it is written in business terms. It also creates documentation, ensuring that when regulators, auditors, board members, or legal counsel inquire about agent authorization, a clear record with a name and date is available.</p><h4>The part where the law is already ahead of you</h4><p>There is a more urgent reason to focus on authority, and it should be of particular concern to executives.</p><p>Agency law, which governs what happens when one party acts on behalf of another, maintains two types of authority. Actual authority is what the principal genuinely granted, whether stated outright or reasonably implied. Apparent authority is what a third party reasonably believed, based on how the principal presented it. The critical component of apparent authority is that it binds the principal even when no actual authority existed. If someone is put in a position where a customer reasonably concludes they can commit, the commitment is binding, regardless of internal rules.</p><p>This is not a hypothetical. In 2024, a British Columbia tribunal applied this logic to a chatbot in Moffatt v. Air Canada. A customer applied for a bereavement fare retroactively based on the airline&#8217;s chatbot. Unfortunately, that was not the policy as detailed on another page of the airline&#8217;s website. Air Canada argued that the chatbot was in effect a separate entity responsible for its own statements. The tribunal member rejected the argument and found the airline liable for negligent misrepresentation and responsible for everything on its website regardless of whether a static page or a chatbot produced it.</p><p>While that case involved a chatbot providing advice, the same reasoning applies to agents that execute actions or make decisions, such as issuing credits, amending bookings or releasing goods. The gap between system configuration and customer expectations is where apparent authority applies, often to the organization's detriment.</p><p>United States legal commentary through 2026 has converged on the same conclusion from several directions. A California statute now blocks defendants from arguing that an AI system autonomously caused the harm, and federal security agencies have issued guidance signaling that firms will be expected to govern, monitor and explain agent behavior. The defense that the agent did it on its own is closing off.</p><p>The practical implications are both challenging and clear. Organizations will be held accountable for the authority their agents appear to possess. The only way to align perception with reality is to define, document and enforce authority within systems.</p><h2>What a mandate actually contains</h2><p>A mandate is not a policy document or governance framework. It is a concise, specific record of authority, similar to a signing authority letter. Ten key fields define the mandate. If your organization can supply them for every production agent, most governance challenges will be addressed.</p><h4>Identity</h4><p>The agent must have its own credentials, separate from the human who deployed it or any shared service account. Although this seems obvious, it is often not the case. Agents frequently run under the credentials of the engineer who built them or a shared integration account. If you cannot identify which agent acted from a log, you cannot enforce any other controls. </p><p>Singapore&#8217;s regulator made this a key requirement in the Model AI Governance Framework for Agentic AI published in January 2026, requiring each agent to have a verifiable identity and an audit trail showing which agent acted under whose authorization.</p><h4>Purpose</h4><p>The agent&#8217;s job should be stated in one sentence, focusing on the specific task rather than general capabilities. For example, &#8220;Reconcile supplier invoices against purchase orders and post matched items&#8221; is properly defined. &#8220;Assist the accounts payable team&#8221; is too broad and lacks boundaries.</p><h4>Permitted actions</h4><p>The list of the specific operations on named systems the agent may perform needs to be documented. For example, &#8220;Read the vendor master, create a payment proposal, or post a matched invoice below a set threshold.&#8221; Any action not listed is denied, and the key is to avoid vague descriptions. Specificity is essential for effective governance.</p><h4>Data reach</h4><p>The mandate needs to define what the agent may access as a set of permissions, beyond just system names. Granting access to a document repository usually provides access to all its documents, even those the agent might not be authorized to view. Retrieval systems generally prioritize relevance over entitlement unless they are specifically designed to enforce access restrictions.</p><h4>Consumption ceiling</h4><p>A strict limit should be set on what the agent may consume, including tokens, external calls, compute resources or actual funds. This is not a budget alert that notifies after the fact. A ceiling stops execution and requires human intervention to resume. Retry loops can quickly exhaust resources without triggering any alerts.</p><h4>Duration</h4><p>Mandates should have an expiration date. Ninety days is a reasonable default for new agents, and one year may be appropriate for mature, low-risk agents. Ongoing systems are rarely reviewed without an expiry date.</p><h4>Escalation triggers</h4><p>The conditions under which the agent must pause and transfer decision-making to a designated individual should be detailed. Examples include exceeding a value threshold, performing irreversible actions, interacting with unknown counterparties, low confidence in outcomes, or repeated failures. The escalation must result in the agent waiting for human approval. It cannot just document the event and proceed.</p><h4>Delegation rights</h4><p>The mandate should document whether the agent may invoke other agents and which permissions, if any, are inherited. This field is often overlooked, leading to unintended permission escalation. By default, agents should not delegate. If necessary, each sub-agent must have its own mandate and ceiling to prevent automatic inheritance.</p><h4>Revocation</h4><p>It is critical to define who can disable the agent, the required response time, and the mechanism used. This process should be tested regularly. If an organization cannot disable an agent within five minutes, it lacks effective control. Regulatory expectations in the EU AI Act reinforce this requirement.</p><h4>Owner</h4><p>A specific individual in a business role who is accountable for the agent&#8217;s actions should be assigned. This must be a named person, not a team or platform group. Their name should appear in the record as the responsible party. Ownership is essential for enforceability, and without it, controls are only suggestions.</p><h4>Not every agent deserves this treatment</h4><p>Due to the resources required, applying all ten fields to every agent in a large company is unsustainable. Instead, organizations can categorize agents by consequence, focusing on the reversibility and cost of errors rather than agent complexity.</p><p>These four tiers cover most situations.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HQ-S!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HQ-S!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png 424w, https://substackcdn.com/image/fetch/$s_!HQ-S!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png 848w, https://substackcdn.com/image/fetch/$s_!HQ-S!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png 1272w, https://substackcdn.com/image/fetch/$s_!HQ-S!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HQ-S!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png" width="788" height="260" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:260,&quot;width&quot;:788,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:55346,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/208384506?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HQ-S!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png 424w, https://substackcdn.com/image/fetch/$s_!HQ-S!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png 848w, https://substackcdn.com/image/fetch/$s_!HQ-S!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png 1272w, https://substackcdn.com/image/fetch/$s_!HQ-S!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17b1ab3f-cce1-4147-bd8c-154f478b9099_788x260.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Tier three can be challenging. Teams may assess risk based on transaction value, but reversibility is more important than monetary value. For example, sending an incorrect email may be irreversible despite minimal cost, just as deleting a volume is irreversible. It is critical always to consider whether an action can be undone before evaluating its value.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/your-agents-have-authority-nobody?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/your-agents-have-authority-nobody?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>An example mandate</h2><p>Consider a mid-sized insurer that has deployed an agent in claims operations handling roughly 4,000 claims a month. The agent handles the full cycle of low-value property claims. It reads the claim, retrieves the policy, checks the coverage, requests any missing documents from the customer, and, when everything matches, approves payment and instructs the payment system. The business case is a four-day reduction in cycle time and the redeployment of 11 adjusters onto more complex work.</p><h4>Identity</h4><p>The agent receives its own credentials. All actions are directly attributable to the agent.</p><h4>Purpose</h4><p>Adjudicate and settle residential property claims valued below $2,500 where the policy is in force, the coverage is unambiguous, and no fraud indicator is present.</p><h4>Permitted actions</h4><p>Permitted actions include reading the claim record, reading policy status and coverage terms, requesting documents from the claimant using approved templates, updating claim status, and creating payment instructions. Actions such as amending policy terms, updating customer bank details, closing claims as denied, and communicating outside approved templates are explicitly denied.</p><h4>Data reach</h4><p>The agent may access claims and policy data only for the specific claimant and policy in scope. It is not permitted to access the full policy master, data for other claimants, or medical records held in other systems.</p><h4>Consumption ceiling</h4><p>Set both a per-claim cost limit and a daily aggregate limit. If either is exceeded, the agent halts and notifies the owner. The daily limit should be about twice the normal volume to detect anomalies promptly.</p><h4>Duration</h4><p>Set the mandate duration to 90 days initially, extending to annual reviews once the agent&#8217;s risk profile is established.</p><h4>Escalation triggers</h4><p>Escalation triggers include claims exceeding $2,500, any fraud indicators, claims from policies less than sixty days old, changes to payment destinations, low confidence in coverage, or repeated failures at the same step. Upon escalation, the agent stops and routes the case to a designated adjuster with supporting rationale.</p><h4>Delegation rights</h4><p>No delegation rights are granted. If a document extraction agent is added in the future, it will require its own mandate and ceiling.</p><h4>Revocation</h4><p>The claims operations duty manager can disable the agent directly from the operations console without requiring an IT ticket. This process is tested monthly as part of operational readiness.</p><h4>Owner</h4><p>The head of claims operations, identified by name, is the owner. The platform team manages system operation, but she is accountable for the agent&#8217;s actions.</p><p>Two common outcomes typically emerge during this process. First, teams often discover previously unknown permissions. The second outcome is accelerated growth. With a clear mandate, the head of claims can confidently increase thresholds or expand the agent&#8217;s scope, knowing precisely what is permitted and what remains restricted. Without a mandate, proposed expansions often lead to unresolved debates due to unclear permissions. The explicit authority actually enables greater autonomy.</p><h2>The one-page agent mandate</h2><p>The agent mandate addresses the bulk of the concerns associated with running agents in production. It fits on a single page and serves as the complete artifact.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!oFCv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!oFCv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png 424w, https://substackcdn.com/image/fetch/$s_!oFCv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png 848w, https://substackcdn.com/image/fetch/$s_!oFCv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png 1272w, https://substackcdn.com/image/fetch/$s_!oFCv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!oFCv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png" width="771" height="587" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:587,&quot;width&quot;:771,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:69237,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/208384506?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!oFCv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png 424w, https://substackcdn.com/image/fetch/$s_!oFCv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png 848w, https://substackcdn.com/image/fetch/$s_!oFCv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png 1272w, https://substackcdn.com/image/fetch/$s_!oFCv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff81240c0-4a8a-4ddd-8aca-d21b3bcbd1a5_771x587.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Three simple rules govern the mandate. First, any action not listed as a permitted action is denied. Second, the mandate expires on the specified date, and the agent must then stop. Third, any changes to permitted actions, data reach or consumption ceiling require the owner&#8217;s sign-off. It is not just a configuration update.</p><h2>Where this breaks in practice</h2><p>Four failure modes account for most of the issues that can arise.</p><h4>1. The agent finds credentials you did not give it</h4><p>This scenario occurred at PocketOS. The agent operated in an environment where a powerful token was accessible and using it aligned with its objective. The solution is not improved instructions, as instructions are advisory. The PocketOS agent had explicit safety rules in its configuration that were disregarded. Effective enforcement must reside outside the agent, and permissions must be validated in the target system. Controls embedded in prompts are requests and not enforceable safeguards.</p><h4>2. Standing credentials</h4><p>Most agents retain permanent access to all potential resources, reflecting traditional integration practices. This approach transitions organizations to issuing credentials for specific tasks with defined expiration. While full implementation may take years, high-consequence agents can be prioritized and updated within months using tiering.</p><h4>3. Environment boundaries that exist only on paper</h4><p>The PocketOS agent operated under the assumption it was in a staging environment, but the credential accessed provided production access. Any boundary enforced only by naming conventions, rather than by network or identity separation, is likely to be breached over time.</p><h4>4. Sub-agents and inherited scope</h4><p>Multi-agent architectures are becoming increasingly common and often result in unintended permission inheritance. You should treat any permission accessible within the chain as available to the entry point. It is key to explicitly limit delegation and require agents that can initiate others to specify which agents they may invoke.</p><h2>Getting this done in 90 days</h2><p>Implementing effective governance does not require extensive new functions, committees or platforms. It can be efficiently addressed in five steps.</p><h4>Weeks 1-2 - Find them</h4><p>Compile a list of all the agents operating across company systems. Query the identity provider for non-human accounts created in the past 18 months and consult the AI gateway if available. Ask engineering leads directly, emphasizing that this is a standard inquiry because unsanctioned agents are often overlooked. Given that over half of organizations in the Cloud Security Alliance study reported unauthorized agents in operation, the actual number is likely to exceed initial estimates.</p><h4>Week 3 - Tier them</h4><p>Categorize the inventory using the four tiers. Ask whether these actions can be undone and at what cost. This process typically requires one day per business area with appropriate stakeholders and will reduce the number of agents requiring full governance by roughly 80%.</p><h4>Weeks 4-8 - Write mandates for tiers three and four</h4><p>For each agent, complete the one-page mandate canvas. The business owner drafts it, risk reviews it, and the owner signs it. Specialized software is not required. A spreadsheet and document library are sufficient for the first 100 agents, and the system will provide valuable insights for future automation.</p><h4>Weeks 6-10 (in parallel) - Move enforcement out of the prompt</h4><p>For all tier-three and tier-four agents, companies need to ensure that the target system or an intermediary gateway enforces permitted actions. If an action is prevented only by instructions, this should be recorded as an open issue with a target resolution date. This is the most challenging step in the process and is essential for making mandates enforceable controls.</p><h4>Weeks 10-12 - Prove revocation and evidence</h4><p>Finally, controlled tests are conducted to disable and measure response time for each tier-three and tier-four agent. A week of logs should be reviewed to confirm that each action can be traced to the responsible agent, their mandate, and the accountable human owner. If this reconstruction is not feasible, the organization cannot adequately respond to regulatory or customer inquiries.</p><p>Two important notes on sequencing.</p><p>First, do not wait for formal standards to start a compliance program. While organizations such as NIST, Singapore and European bodies are developing official frameworks, these are not yet available for immediate implementation. Instead, use existing identity and access tools to start implementing recommended components.</p><p>Second, do not assign this responsibility solely to the AI governance committee. The mandates require collaboration across teams. The business owner defines the delegated authority, while the identity and access team enforces it, and the risk function determines tiers and validates evidence. In many organizations, these groups often do not view agent permissions as their responsibility, contributing to governance gaps.</p><h2>The three questions that gate production</h2><p>Before deploying an agent, a high-performing company needs to fully answer three key questions that address the most critical concerns.</p><h4>1. What is the worst thing this agent can do in a single step, and can we undo it? </h4><p>Determine the most severe action, not just the most likely scenario, enabled by the agent&#8217;s permissions. If no one can answer this sufficiently, the permissions have not been properly reviewed, and deployment should be delayed.</p><h4>2. Who is the named individual who will be asked to explain this agent&#8217;s actions, and have they read the mandate? </h4><p>If the response identifies a team, platform or vendor rather than an individual, ultimate ownership is lacking. Before proceeding with deployment, a specific owner needs to be identified.</p><h4>3. Who has seen the escalation working? </h4><p>Companies can not rely on design documentation alone. The escalation process needs to be tested to ensure the agent stops, the case is routed to a human queue, and a person resumes it. Controls that are not regularly exercised are most likely to fail.</p><h2>The real risk</h2><p>The common narrative about agent risk involves a rogue system developing its own goals and acting unpredictably to destroy humanity. While this Hollywood scenario attracts mindshare, it is less likely to occur in the real world.</p><p>A more probable situation is the gradual accumulation of authority within your organization without explicit approval. Engineers may reuse credentials for expediency, and outdated permissions may persist on roles now assumed by agents. Pilot agents may become production agents simply by remaining active, and agents may invoke one another. These developments are often invisible on dashboards that track system activity but not permissions.</p><p>When an incident occurs, regulators, auditors, or legal counsel will not ask about model evaluation, but they will question who authorized the action. If the response is that permissions were inherited from an outdated service account, this reflects a lack of governance rather than a technical oversight and will not be well received by these stakeholders.</p><p>There is a better approach to avoid problems. Organizations that write agent mandates deploy more agents, not fewer. Each expansion becomes an amendment to a document that specifies what is changing and who approved it. The updates can be completed in a few hours. While competitors are still debating whether agents are safe, which is an unanswerable question, high-performing companies are asking what a specific agent is permitted to do, which can be answered. </p><p>This principle has been consistently advocated. Governance is not a barrier to speed, but an accelerant that enables sustainable rapid progress. Previously, human oversight mitigated errors between model output and real-world impact. With agents, that intermediary is absent because the mandate now fulfills this critical role.</p><p>Your agents already possess authority. The key question is whether this authority was granted intentionally or recognized after the fact.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/your-agents-have-authority-nobody/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/your-agents-have-authority-nobody/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Most AI Governance Is Theater. Yours Doesn’t Have to Be.]]></title><description><![CDATA[Right-sizing NIST, ISO 42001 and the EU AI Act to the risk you actually carry]]></description><link>https://trustedai.recodework.com/p/most-ai-governance-is-theater-yours</link><guid isPermaLink="false">https://trustedai.recodework.com/p/most-ai-governance-is-theater-yours</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Sat, 18 Jul 2026 15:48:09 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!3BmN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!3BmN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3BmN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg 424w, https://substackcdn.com/image/fetch/$s_!3BmN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg 848w, https://substackcdn.com/image/fetch/$s_!3BmN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!3BmN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!3BmN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg" width="1456" height="1059" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1059,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:8434703,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/206744998?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!3BmN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg 424w, https://substackcdn.com/image/fetch/$s_!3BmN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg 848w, https://substackcdn.com/image/fetch/$s_!3BmN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!3BmN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2cb4c-cb74-4a1a-8ece-b67ed6fe10b0_3000x2182.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>The Governance Nobody Uses</h2><p>Somewhere in your organization, or an organization very much like yours, there is a document. It is titled something like &#8220;Responsible AI Policy&#8221; or &#8220;AI Governance Framework.&#8221; It runs to eight or twelve pages. It cites NIST, alludes to ISO 42001, and includes a section on the EU AI Act. The document was assembled under tight deadlines by someone who was already managing three other jobs, and it was approved in a meeting where it wasn't reviewed closely enough to raise objections.</p><p>That document has changed nothing. No AI system was built differently because of it. No launch was stopped because of it. No engineer has opened it since it was published. It exists to be pointed at, in a board deck, in a customer security review, in a due diligence questionnaire, as evidence that the company takes AI seriously. It is a prop.</p><p>This is the open secret of AI governance in 2026. An enormous amount of it is theater. And the uncomfortable part, the part governance vendors and consultants will not tell you, is that the theater is not an accident or a shortcut taken by lazy companies. It is the predictable output of doing exactly what everyone told you to do. Adopt the frameworks. Produce the documentation. Stand up the committee. Follow the enterprise example.</p><p>The enterprise example is the trap. The playbook often used as the standard to imitate was engineered for a company with a risk profile, regulatory exposure, and operating model that are likely not yours. When you copy it, you inherit its costs and its pathologies while getting almost none of its benefits. And escaping that trap is the single highest-leverage governance decision most companies can make, because the alternative is not weaker governance. It is more real governance, matched to the risk you actually carry rather than to the risk someone else carries.</p><h2>Three Frameworks, Three Different Things, One Persistent Confusion</h2><p>Before anything else, we have to clear up a confusion that underpins almost every bad governance decision. People treat NIST AI RMF, ISO/IEC 42001, and the EU AI Act as three competing options, three flavors of the same thing, and then they try to &#8220;comply&#8221; with all three at once, as though governance were a buffet where the goal is to load your plate.</p><p>They are not the same kind of thing. They are not in the same category. Getting this wrong is the original sin from which most governance theater descends.</p><h4>NIST AI RMF is guidance. It tells you how to think.</h4><p>The NIST AI Risk Management Framework, published by the US National Institute of Standards and Technology in January 2023, is voluntary. Nobody certifies you against it. Nobody audits it. There is no badge. It is a mental model, and a good one, organized around four functions that you cycle through continuously rather than complete once.</p><p>Govern is the accountability layer. Who owns AI risk, what your risk tolerance is, and how decisions escalate. Map is the context. What a system is for, who it affects, what could go wrong, and what are the consequences of those failures. Measure is assessment. The actual testing, evaluation, and monitoring that tells you how a system behaves. Manage is action. Prioritizing risks, deploying mitigations, and making the actual go/no-go and retirement calls.</p><p>The reason NIST has become the default reference point is that it describes an operating rhythm rather than a checklist. Govern, Map, Measure, Manage are verbs you repeat, not boxes you tick. That makes NIST the right tool for one specific job. Being the actual workflow your teams follow. It is guidance about how to think, and thinking is not something you can outsource to a document.</p><h4>ISO/IEC 42001 is a management system. It tells you how to prove it.</h4><p>ISO/IEC 42001, published in December 2023, is the first international standard for AI management systems. If you have lived through ISO 27001 for security or ISO 9001 for quality, this will feel familiar, because it is built on the same architecture. Documented policy, defined roles, risk assessment processes, internal audits, management review, and continual improvement.</p><p>The thing that makes 42001 categorically different from NIST is its certifiability. An accredited body can audit your AI management system and issue a certificate. That certificate is an external trust signal, useful in enterprise sales, vendor questionnaires, and partner due diligence. But notice what 42001 does not do. It does not tell you which bias metric to use or how often to test for drift. It tells you that you must have a systematic, documented, reviewed approach. It is the skeleton, not the muscle. It provides structure and auditability, the scaffolding that turns informal good intentions into something durable and provable.</p><h4>The EU AI Act is law. It tells you where the stakes are highest.</h4><p>The EU AI Act, adopted by the European Union in 2024, is neither guidance nor a voluntary standard. It is a binding regulation with real financial penalties, and it works through a risk-based classification. Unacceptable-risk practices, such as social scoring and certain manipulative systems, are prohibited. High-risk systems in domains such as employment, credit, essential services, and law enforcement carry the heaviest obligations, including documented risk management, data governance, technical documentation, human oversight, and conformity assessment. Limited-risk systems, such as chatbots, face lighter transparency duties. Minimal-risk systems face nothing specific.</p><p>Worth noting, because it makes the larger point about regulation, the Act&#8217;s own timeline has moved. High-risk obligations under Annex III were originally scheduled to take effect in August 2026. As of mid-2026, EU lawmakers agreed, through the Digital Omnibus on AI, to defer standalone high-risk obligations to December 2027, largely because the technical standards and conformity infrastructure needed actually to comply were not yet ready. Prohibited practices and AI literacy duties have applied since February 2025, and general-purpose AI model obligations since August 2025. If you have EU exposure, please verify the current dates against live guidance rather than relying on fixed dates you see elsewhere, including here, because they can change.</p><p>The functional role of the Act, though, remains stable even when its dates change. It does not hand you an operating model like NIST, nor does it give you an auditable system like 42001. It tells you where society has decided the stakes are highest. Its risk tiers are, in effect, a pre-built prioritization map that reflects a broad consensus about where AI does the most harm. That is useful to you even if you never touch the EU market.</p><h4>So here is the whole distinction, and it is the load-bearing idea of this entire piece.</h4><p>NIST tells you how to think. ISO 42001 tells you how to organize and prove it. The EU AI Act tells you what you are legally required to do and where the highest stakes lie. Guidance, management system, regulation. Three layers of one problem, not three items on a menu. When companies treat them as interchangeable, they end up doing all three poorly.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>The Enterprise Checklist Is the Disease, Not the Cure</h2><p>Now the provocative part, and I mean it literally rather than as a rhetorical flourish. The enterprise governance playbook that everyone holds up as the standard to imitate is, for a company whose risk and operating models differ from those of the enterprise it was built for, actively harmful. Copying it does not make your governance more serious. It makes it more fake.</p><p>Here is why, and there are three mechanisms.</p><h4>The enterprise optimizes for documentation because documentation is what survives an audit. </h4><p>A large regulated company has armies of auditors, regulators, and litigators pointed at it. In that environment, the rational move is to generate a paper trail proving that a process was followed, because the paper trail is the thing that gets examined when something goes wrong. This produces a specific pathology. Governance activity becomes measured by artifacts produced rather than harms prevented. Model cards for everything. Risk assessments for everything. Attestations for everything, including the internal meeting notes summarizer. When a company copies this without inheriting the scrutiny that made it rational, it ends up with the pathology stripped of its context. It produces the same undifferentiated pile of paper, except now the one genuinely consequential system gets the same shallow treatment as the harmless one, because attention is finite and it gets spread evenly across things that do not deserve equal attention. Volume of paperwork masquerades as depth of oversight. It is the opposite.</p><h4>The enterprise checklist assumes a way of building software that you may have correctly abandoned years ago. </h4><p>Enterprise governance gates were designed for waterfall delivery. Design review, then build, then a formal validation stage, then a deployment approval. A distinct, sequential, heavyweight process with a committee at each door. That is not how a company shipping on foundation model APIs works. You prototype against an API on Monday, ship an MVP by Friday, and iterate on real user feedback the following week. When you bolt a waterfall governance process onto an iterative build process, one of two things happens, and both are bad. Either governance becomes the bottleneck that kills your speed, or your teams quietly route around governance entirely, which means the policy exists and the practice ignores it. Governance theater is usually the second outcome. The policy is real. The compliance is fictional.</p><h4>Committees are where accountability goes to die. </h4><p>The enterprise loves a governance committee, and there is a reason. In a large organization, a committee distributes accountability so widely that no single person can be blamed, which is a feature if your deepest institutional instinct is blame-avoidance. But distributed accountability is not accountability. It is its absence, dressed formally. When a council reviews a summary slide and nods, no one has actually decided anything, and no one can actually be held responsible when the system fails. A company that imports this structure before it needs it also imports the dilution. Below a certain scale, you do not need a council to make an AI risk decision. You need one named person with the authority to say no, and the spine to use it. The council is what you build when you have grown too large for that to be possible. Treating it as a marker of maturity before you have reached that point is confusing the scar tissue of scale for a sign of health.</p><p>The deeper error underneath all three is a category mistake about what a framework is. The frameworks are tools. The enterprise applies them at enterprise depth because it faces enterprise risk, enterprise scrutiny, and enterprise sprawl. When you copy the depth without the underlying conditions, you get cost without benefit. You get the theater.</p><h2>Depth Should Track Risk, Not Imitation</h2><p>Here is the reframe, and it is the reason this topic is worth a whole issue rather than a footnote. The question that actually determines whether your governance is real is not how much of the enterprise playbook you have reproduced. It is whether the depth of your controls tracks the actual risk of each system. Uniform depth is the tell of theater, in either direction. A portfolio with identical documentation for every system suggests that no one looked closely at any of them.</p><p>Consider why the enterprise playbook encodes uniform, heavyweight processes in the first place. A large enterprise runs hundreds or thousands of AI systems, many undocumented, spread across business units that do not communicate with one another, built over the years by people who have since left. Its hardest governance problem is often simply knowing which AI it has, and its second-hardest is imposing any consistent practice across a workforce large enough that some meaningful fraction will ignore whatever policy gets published. Uniform process is a rational, if blunt, response to scale, legacy, and coordination failure. It is a solution to problems that come from bigness.</p><p>If those are not your dominant problems, adopting a solution to them would add overhead without a clear benefit. The heavyweight, uniform apparatus was never designed to improve governance. It was designed to make governance consistent across an organization too large and too fragmented to govern any other way. Reproducing that uniformity without that fragmentation does not buy you rigor. It buys you the appearance of rigor, spread so thin that your highest-stakes system and your lowest-stakes system receive the same shallow glance.</p><p>The alternative is to let depth follow risk. Concentrate real scrutiny, real testing, and real monitoring on the systems that make or shape consequential decisions about people, and deliberately spend almost nothing on the ones that cannot hurt anyone. This is not the discount version of governance. It is more demanding because it requires you to assess each system rather than rely on a policy that treats them all the same. Uniform process is the easy answer. Risk-matched depth is the honest one. And the frameworks, read correctly, are built to support exactly this, which is what the rest of this comes down to.</p><h2>Start With What Is Real</h2><p>The mapping starts by refusing to start with the frameworks. Do not open the NIST document. Do not open the ISO standard. Open a blank spreadsheet and inventory every AI system you actually have in production or active development, and for each one, answer plain questions. Who is affected by its outputs? What is the worst plausible thing that could happen if it fails or behaves strangely? Does it make or materially shape a decision about a person, hiring, credit, pricing, health, or access to a service? Is it customer-facing, internal-only, or sold as a product?</p><p>This inventory, built from reality rather than from a framework&#8217;s table of contents, is the ground on which everything else stands. And it is, not coincidentally, exactly what NIST&#8217;s Map function is for. Map is the discipline of understanding context and consequence before you reach for controls. You are already doing NIST, and you did not need to read NIST to do it.</p><p>From that inventory, the frameworks slot into distinct roles, each doing the one job it is good at.</p><h4>NIST becomes your operating model, the rhythm your teams actually follow. </h4><p>Govern names who owns the risk call and what your tolerance is. Map is the inventory exercise, made repeatable for every new initiative rather than done once. Measure is the testing and monitoring you define per system, scaled to its risk. Manage is where the real deploy decisions get made and written down. Because NIST is voluntary and flexible, it is the language of your day-to-day, in standups and product reviews, not a bureaucracy you erect. It asks you to build a habit, not a department.</p><h4>ISO 42001 becomes your structural backbone and your maturity path, applied later than you think. </h4><p>You do not build a management system on day one. Once your NIST rhythm is real and repeated, 42001 becomes a lens for auditing what you already do. Is there a written policy, even a short one? Is there a named owner, even a fractional one? Do your risk assessments get reviewed on any cadence at all? You can extract enormous value from 42001&#8217;s clause structure as a gap-analysis tool without ever hiring a certification body. And if a customer eventually demands the certificate, often in financial services or healthcare, you will already have the scaffolding, which turns a brutal certification scramble into a manageable one.</p><h4>Regulation becomes your prioritization filter, not a parallel track. </h4><p>Do not run EU AI Act compliance as a separate governance universe. Lay its risk classifications over your inventory and let them pull your depth toward the systems that deserve it. If a system lands in a high-risk category, whether or not you touch the EU today, that is your signal to apply your most rigorous NIST testing and to formalize that system&#8217;s documentation first. Regulation tells you where the stakes concentrate. Let it concentrate your effort there instead of smearing rigor evenly across a portfolio where most systems do not warrant it.</p><p>Start with the inventory, which is the map itself. NIST provides the ongoing discipline, while ISO 42001 offers the structure that discipline builds over time. Regulations act as the filter, guiding where to dive deeper. At no point is an entire framework adopted, and that&#8217;s not a compromise. It&#8217;s intentional.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/most-ai-governance-is-theater-yours?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/most-ai-governance-is-theater-yours?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Minimal Governance That Is Real</h2><p>Right-sized governance is not a smaller pile of the enterprise&#8217;s paperwork. It is a different, smaller set of things applied at depths that vary with risk rather than uniformly. Four components carry almost all the weight.</p><p>An honest inventory with risk tiers is the foundation, and a well-kept spreadsheet is genuinely fine at first with every system, its owner, what it does, and its tier. A simple structure works. Low risk for internal productivity tools and non-consequential recommendations. Medium for systems that materially affect operations or customers. High for systems that make or shape consequential decisions about people. Prohibited for things you have chosen not to build, regardless of feasibility. The discipline of keeping it current beats the precision of the cut points.</p><p>A short policy with real ownership comes next. A few pages, not a few dozen. Your principles, who owns the risk call, and what has to be true before a system moves from prototype to production. The load-bearing word is owner. One named person with the authority to stop a launch, even if the role is a slice of a bigger job. Accountability that lives on an org chart and never in an actual decision is not accountability. It is decoration.</p><p>A risk register and testing approach, calibrated by tier, is where governance grows teeth. For medium and high-risk systems, document the risks, the mitigations, and the residual risk after mitigation, and pair that with testing that fits the tier. A high-risk system that touches credit or hiring undergoes bias testing, adversarial robustness checks, and human oversight review before it ships. A low-risk summarizer earns none of it. The error to avoid runs in both directions, skipping testing on your highest-stakes system and burying your lowest-stakes one under processes it does not need. Uniform depth is the enterprise disease. Do not import it.</p><p>Monitoring and incident handling close the loop. Once a system is live, someone watches it at a depth proportional to its tier. High-risk means active monitoring for degradation, drift, and anomalous output, plus a defined answer to what happens and who gets called when it misbehaves. Lower-risk may need only periodic spot checks. What matters is that a process exists at all, so you learn about problems before a customer or a regulator does.</p><p>Those four, kept honest, are a governance program a lean team can actually run and that will still hold up under real scrutiny. Everything past this is refinement. None of it is a prerequisite for having something real.</p><h2>What This Looks Like When It Is Real</h2><p>Picture a roughly 250-person B2B software company that sells workforce analytics. It had shipped AI features fast, and when a large customer&#8217;s security team asked about its AI governance, the company did what most do. It found a governance template, started drafting a policy citing all three frameworks, and floated the idea of a cross-functional AI review board to sign off on every AI feature.</p><p>That board would have reviewed everything at the same shallow depth and decided almost nothing, which is where this story usually ends. Instead, before building any of it, the company spent an afternoon on an inventory. Four AI systems mattered, including an internal tool that summarized meeting notes, a model that routed inbound support tickets, a feature that recommended next actions to customer administrators, and a capability that scored employees for attrition risk and surfaced those scores to their managers.</p><p>Written down side by side, the systems were obviously not equivalent. The meeting summarizer could embarrass someone with a bad summary at worst, so it was entered into the inventory as low risk and received no further review. The ticket router and the recommendation feature were medium risk, affecting operations and customer experience, but making no consequential call about a person, so they got a short risk entry and a basic monitoring check, nothing more.</p><p>The attrition-scoring system was of a different kind. It shaped decisions about people&#8217;s jobs, placing it squarely in the territory the EU AI Act treats as high-risk employment use, regardless of whether the company had EU customers that quarter. So that system, and that system alone, got the full treatment. Bias testing across demographic groups before the scores ever reached a manager. A documented human-oversight step, so that a score could never trigger an action on its own. Active monitoring for drift once it was live. And a single named owner is personally accountable for whether it remains fit to run.</p><p>The whole program took weeks, not quarters, and it fit on a handful of pages. The customer&#8217;s security team was satisfied because the answers to their hardest questions were about the one system that warranted hard questions. And the company had avoided the outcome that it had been an afternoon away from choosing: a review board that would have spread attention evenly across four systems and looked hardest at the one that could hurt someone least.</p><h2>Operationalizing Without Building a Bureaucracy</h2><p>Concepts are cheap. The test is whether this survives contact with how work actually happens. Here is what it looks like when each framework is operationalized rather than framed.</p><p>NIST gets embedded into workflows and never runs beside them. Map happens during design and scoping, captured in the kickoff template your team already uses, with a field or two added for risk tier and affected stakeholders. Measure happens within the testing you already do, with extra evaluation criteria layered in for medium and high-risk systems, bias or robustness checks added to the existing suite rather than run as a separate exercise anyone can skip. Manage happens at the same go/no-go point you already have for any launch, with an explicit risk sign-off required above the low tier. Govern is the periodic check, quarterly suits most organizations, where the owner reviews the inventory, any incidents, and whether the tiers still reflect reality. The governing rule is that the governance step should be the path of least resistance within an existing workflow, not a gate that teams learn to circumvent. The moment governance becomes a detour, it becomes theater.</p><p>ISO 42001 gets layered in over time, as evidence, not aspiration. Do not attempt full alignment at the start. After the NIST rhythm has run for a few months and produced real artifacts, actual assessments, actual test records, and actual incident logs, it begins to formalize. Bring the policy in line with the standard&#8217;s expected content. Stand up a light internal audit, even a semiannual self-review against the clauses. Institute a management review in which leadership actually assesses governance performance and makes adjustments. This is the shift from &#8220;we do sensible things&#8221; to &#8220;we can prove we do sensible things consistently,&#8221; which is precisely the base that supports certification later, if you ever decide the certificate is worth the audit.</p><p>The EU AI Act gets applied where the law demands it and, crucially, nowhere else. If systems fall into high-risk categories, they receive full rigor, documented risk management, data governance, technical documentation, human oversight, and conformity assessment ahead of the applicable deadline. But refuse the reflexive overcorrection of applying that rigor to everything out of anxiety. A low-risk internal tool does not need a conformity assessment. Treating it as though it does is not caution, it is the enterprise disease wearing the mask of prudence. The Act is risk-based on purpose, so that the burden concentrates where the stakes are highest. You need to mirror that logic internally instead of flattening your program to the standard your single most regulated system requires.</p><h2>Phasing It So It Does Not Collapse</h2><p>Trying to stand all of this up at once is how governance programs die before they live. Best practices recommend sequencing it over roughly 12 to 18 months, depending on your size and footprint.</p><p>Phase one, the first month or two, is an inventory, a short policy with a named owner, and your risk tiers. This is about visibility, and the biggest surprise is almost always how much AI is already deployed and that nobody was tracking it.</p><p>Phase two, the next two to four months, focuses on testing and risk management for your medium- and high-risk systems specifically, plus a basic incident process. This is where governance starts changing behavior, because you are now doing something genuinely different for your higher-risk systems than before.</p><p>Phase three, the following three to six months, is about mapping your rhythm explicitly onto the NIST functions and beginning to layer in the ISO 42001 structure, formalized policy, internal audit, and management review.</p><p>Phase four is expansion and, only if it pays for itself, formalization and certification. Extend coverage to the gaps, strengthen the evidence for whichever frameworks your customers and regulators actually care about, and decide whether ISO 42001 certification is worth the cost. For companies selling into regulated industries or for enterprise buyers with rigorous due diligence, the certificate becomes a real differentiator. For others, the substance, real testing, real monitoring, real accountability, delivers nearly all the value without the badge.</p><p>Resist skipping ahead. A company that lunges straight at certification with phases one and two hollow will find itself documenting practices that do not exist, which is the exact theater this whole argument is built to prevent.</p><h2>Coherent, Not Comprehensive</h2><p>The companies that get this right are not the ones with the most frameworks or the thickest binders. They are the ones whose governance is coherent, where NIST&#8217;s rhythm, ISO 42001&#8217;s structure, and the EU AI Act&#8217;s priorities reinforce each other instead of running as three disconnected compliance rituals that share nothing but a slide.</p><p>Coherence starts with a decision that is trivial to state and genuinely hard to hold under pressure. Start with what is real, your actual systems and their actual risk, rather than with what a framework says you should have. Scale governance on purpose, adding depth and formality as your footprint and maturity grow, rather than importing an enterprise checklist wholesale just because it photographs well in a board deck.</p><p>And here is the thing worth sitting with. None of the three frameworks asks you to become a miniature enterprise. NIST is voluntary by design. ISO 42001 explicitly permits proportionate implementation scaled to context. Even the EU AI Act builds its entire structure around risk tiers, so that obligations concentrate where harm is greatest. The frameworks themselves are telling you that matching depth to risk is not the discount version. It is the intended version. The only people insisting otherwise are the ones selling you the enterprise checklist.</p><p>The real risk was never that your governance looks leaner than a global bank&#8217;s. The real risk is a governance program that looks good on paper but changes nothing, that produces the prop document, the unread policy, the committee that decides nothing, while your one genuinely consequential system ships with the same shallow attention as your meeting-notes bot. That is the failure that ends careers and reputations, and it is the failure the enterprise checklist quietly walks you into while making you feel responsible the whole way.</p><p>So build something matched to what you actually carry. Build something a real person is accountable for, and that actually shapes what gets shipped. Fake governance is worse than none, because at least none lies to you about what you have. The goal was never to be comprehensive. It was to be real.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/most-ai-governance-is-theater-yours/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/most-ai-governance-is-theater-yours/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Building a Culture of Responsible AI]]></title><description><![CDATA[How training, incentives and everyday rituals turn trustworthy AI from policy into practice]]></description><link>https://trustedai.recodework.com/p/building-a-culture-of-responsible</link><guid isPermaLink="false">https://trustedai.recodework.com/p/building-a-culture-of-responsible</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Tue, 23 Jun 2026 19:58:43 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!s1qK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!s1qK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!s1qK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg 424w, https://substackcdn.com/image/fetch/$s_!s1qK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg 848w, https://substackcdn.com/image/fetch/$s_!s1qK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!s1qK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!s1qK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:3579336,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/202770408?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!s1qK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg 424w, https://substackcdn.com/image/fetch/$s_!s1qK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg 848w, https://substackcdn.com/image/fetch/$s_!s1qK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!s1qK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F526a77f5-3c28-413d-b62b-a65e3b40143c_3864x2577.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>Culture Is the Real Control Layer</h2><p>Many companies have spent the better part of two years prepping to deliver responsible AI. Policies drafted, principles ratified, councils convened, frameworks selected, and dashboards stood up. But walk into most large organizations today and ask to see the AI governance program, and you will be handed an impressive binder. The gap between what organizations say about their AI and what their AI does has barely closed.</p><p>The numbers tell the story. McKinsey&#8217;s 2025 State of AI research found that 88% of organizations now use AI in at least one business function, up from 78% a year earlier and just 55% two years before that. Adoption is approaching universal. But only about 7% of organizations report having fully scaled AI across the enterprise. The rest are stuck somewhere between the pilot and the payoff. McKinsey calls it AI theatre, the difference between motion and value. It&#8217;s the performance of adoption without the rewiring of behavior that actually captures value or manages risk.</p><p>It is an uncomfortable diagnosis. Governance programs tend to underdeliver, not because they lack the right documents, but because documents do not make decisions. People make them. A policy in a shared drive cannot stop an engineer from shipping an undertested model late on a Friday. A risk framework cannot, by itself, persuade a product manager to flag an uncomfortable result that would delay a launch. A set of principles on the wall cannot make a tired team run the fairness check one more time before release. The thing that governs behavior at the moment of decision is not the framework. It is the culture, the shared set of norms, incentives and habits that tell people what is expected of them when no one is watching and the deadline is close.</p><p>Trusted AI relies on intent and proof. Governance tells you what your AI should do, and observability tells you what your AI is doing. But there is a third leg to that stool, and most organizations neglect it. Culture determines whether your people do the right thing in the thousands of small moments that never reach a governance forum, that no dashboard captures, and that no policy anticipated. Most consequential AI decisions are not made in the AI council. They are made by individuals, quietly, under pressure, and the only thing standing between a good decision and a bad one is what those individuals have internalized as normal.</p><p>This is precisely where the work has moved. PwC&#8217;s 2025 Responsible AI survey found that operationalization, the act of turning responsible AI principles into scalable and repeatable practice, is the single biggest hurdle organizations face, cited by roughly half of executives. The frameworks and standards already exist. What remains is the harder, slower, less glamorous work of making responsible practice lived rather than laminated.</p><p>Specifically, it is about the three levers that convert responsible AI from policy into practice. First, training that shapes judgment rather than checking a box. Second, incentives that reward the behaviors that build trust rather than the outputs that merely look like speed. And third, rituals that make responsibility routine. Organizational design distributes ownership without diffusing accountability, and measurement tells you whether any of it is working before an incident does.</p><h2>Training That Builds Judgment, Not Just Compliance</h2><p>Most corporate AI training fails the same way most compliance training fails. It is a slide deck, a short quiz, a completion certificate, and then a year of forgetting. It satisfies an audit requirement and changes nothing about how anyone behaves. If your AI training looks like your annual cybersecurity module, you have built a record of attendance, not a capability.</p><p>The regulatory floor has risen, which gives leaders useful cover to do this properly. Since 2 February 2025, the EU AI Act has required providers and deployers of AI systems to ensure a sufficient level of AI literacy among their staff and anyone using AI on their behalf, regardless of whether the systems involved are high risk or trivial. </p><p>The regulation deliberately refuses to prescribe a format. It mandates no particular course or certification. Instead, it defines AI literacy as the skills, knowledge and understanding that allow people to make informed decisions about deploying AI and to recognize both its opportunities and its risks. Supervision and enforcement by national authorities arrives in August 2026, and the absence of a documented literacy program is already treated as an aggravating factor in any broader enforcement action. The smartest organizations are not approaching this as a compliance chore. They are using the mandate as permission to build the literacy they needed, regardless of any regulation.</p><p>What distinguishes training that builds judgment from training that merely documents attendance comes down to three properties.</p><p><strong>The first is a role-based approach.</strong> A data scientist needs a working understanding of data drift, concept drift, and the tradeoffs among competing fairness metrics. A product manager needs to recognize the moment a use case crosses from low risk into a territory that demands real scrutiny and know which approvals are required. A board member needs none of the technical depth but all of the right questions. A single curriculum cannot serve this range. The EU guidance itself calls for a differentiated approach calibrated to each group&#8217;s role, knowledge and context, which is simply good pedagogy whether or not a regulator is watching.</p><p><strong>The second property is scenario-based training.</strong> Judgment is not built by memorizing definitions. It is built by practicing decisions. The most effective programs put people in front of realistic dilemmas before they encounter them in real life. A model that performs well in aggregate but is measurably worse for one demographic group. A vendor whose system cannot explain how it reaches a decision is presented for sign-off the week before launch. A senior stakeholder is applying pressure to ship before validation is finished. Walking a team through these situations in a workshop, with no real stakes, is a rehearsal. When the real version arrives, the team will already have felt the shape of the decision and be far more likely to make the right call.</p><p><strong>The third property is continuity.</strong> Even the European Commission&#8217;s own guidance acknowledges that the knowledge of trained employees quickly becomes outdated because technology keeps reinventing itself. Annual training cannot keep pace with a field that produces a meaningful capability shift every few months. Literacy has to be refreshed and woven into the flow of work rather than delivered once and filed. The arrival of agentic systems, which take actions in the world rather than simply returning predictions, is a vivid example. A workforce trained on the risks of last year&#8217;s chatbots is not prepared for the risks of this year&#8217;s autonomous agents.</p><p>The deeper point is that the goal of training is judgment, not compliance, and the two are not the same thing. EY&#8217;s 2025 Work Reimagined survey, covering roughly 15,000 employees across 29 countries, found that companies are leaving as much as 40% of their potential AI productivity gains unrealized because their workforce does not know how to use the technology beyond basic tasks. That same gap that suppresses value also suppresses safety. People who do not understand a system cannot meaningfully oversee it. Human oversight, the principle every governance framework enshrines, is only as real as the human&#8217;s ability to recognize when something has gone wrong. A signature on an approval form from someone who could not have known what to look for is not oversight. It is a liability with extra steps.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>Incentives That Reward Trust Over Speed</h2><p>Culture follows incentives. You can paint any value you like on the wall, but people are shrewd readers of what an organization truly rewards, and they calibrate accordingly. If the only behavior that reliably earns recognition is shipping fast, then responsible AI will lose every time it competes with a deadline, no matter how many policies say otherwise.</p><p>This is the quiet failure mode in most programs, and it is rarely diagnosed correctly. Leaders ask why teams keep cutting corners on testing, why the bias review keeps getting skipped, and why concerns surface only after deployment. The answer is usually right in the performance management system. The engineer who raised a concern and delayed a launch by three weeks is invisible at review time, or worse, gains a quiet reputation for being difficult. The one who shipped on schedule is the hero of the quarter. No amount of training overcomes an incentive structure that punishes the behavior the training was meant to instill.</p><p>Realigning incentives means making responsible behavior both visible and rewarded. </p><p><strong>The first move is to reward the identification of risk, not merely its avoidance.</strong> The person who surfaces a problem early should be celebrated rather than penalized for slowing things down. The most dangerous organizations are those where raising a concern is a career risk, because concerns do not disappear but remain unspoken until they become incidents.</p><p><strong>The second move is to build responsible AI outcomes directly into performance reviews and promotion criteria.</strong> Accountability without consequences is theater. If an AI product owner is accountable for the fairness and ongoing monitoring of their system, that accountability belongs in their objectives and their evaluation, not merely in a job description that no one reads after the interview. This is work to be done alongside HR and requires care, but the principle is simple. What you measure in someone&#8217;s review is what they will prioritize when their time is scarce.</p><p><strong>The third move is to fund governance as an enabler rather than a tax.</strong> Budget decisions reveal real priorities more honestly than any memo. The encouraging news is that the financial case has grown strong enough to make this argument on its own terms. PwC&#8217;s 2025 survey found that nearly 60% of executives say responsible AI practices boost ROI and efficiency, while 55% report gains in customer experience and innovation. The broader pattern is even more striking. PwC&#8217;s analysis of more than 1,200 companies found that the top 20% of performers are capturing roughly 74% of all AI-driven returns, and the practices that distinguish those leaders are precisely the disciplines of mature deployment rather than the volume of models shipped. Responsible AI is not what slows leaders down. It is part of what makes them leaders.</p><p>A word of caution is warranted because incentives are powerful in both directions, and the wrong metric is worse than none at all. Reward the number of models deployed, and you will get models deployed, including ones that should not have been. Reward incidents avoided, and you may get incidents hidden, which is far more dangerous than incidents reported. The art is to incentivize the behaviors that genuinely produce trust, early escalation, thorough documentation, and honest examination of failures, rather than the surface outputs that resemble productivity. You get exactly what you optimize for, so optimize for the right thing.</p><h2>Rituals That Make Responsibility Routine</h2><p>Norms are not built through memos. They are built through repetition. The practices that a team performs over and over, the rituals of the working week, are where culture actually lives, because culture is ultimately just the set of behaviors that have become automatic. Three rituals do most of the heavy lifting in a responsible AI culture, and each one converts an abstract principle into a habit.</p><p><strong>The first and highest-leverage ritual is the structured design review with explicit checkpoints.</strong> The idea is to insert a deliberate pause where decisions get locked in, before a use case is funded, before a model is trained on a particular dataset, and before anything is deployed. Microsoft&#8217;s internal practice is instructive. Its teams log every AI project in a single tool that walks developers from an initial impact assessment through to a final release review, automatically routing each project to the relevant responsible AI champion and triggering deeper scrutiny when a system touches a sensitive use case or external users. The ritual is not really the document. It is the conversation the document forces. A good review asks the questions teams under deadline pressure are tempted to skip. Who could this system harm? What happens when it gets something wrong? Would we be comfortable if this appeared on the front page of a newspaper tomorrow?</p><p>The discipline that keeps this ritual from becoming bureaucracy is right-sizing it to the risk. A low-risk internal productivity tool does not need an ethics board, and forcing it through one teaches people to resent and route around the whole system. A credit decisioning model or a hiring algorithm needs serious scrutiny. PwC&#8217;s 2025 survey found that maturing organizations are deliberately moving away from routing everything through a single central committee, which becomes a bottleneck that breeds workarounds, and toward a tiered model in which first-line teams carry more of the responsibility. In 56% of organizations, those first-line engineering, data, and product teams now lead responsible AI efforts directly. The review ritual scales by pushing routine decisions down to the people closest to the work and reserving the heavyweight forum for the genuine judgment calls.</p><p><strong>The second ritual is the blameless post-mortem.</strong> When something goes wrong, and with AI systems something eventually does, the organization can look for someone to blame or something to learn. The blameless post-mortem, a practice borrowed from site reliability engineering, prioritizes learning on well-evidenced grounds. People rarely fail because they are careless. They fail because the system around them made the failure likely, a signal was buried, an incentive was misaligned, or a check was missing. McKinsey&#8217;s research shows that nearly half of organizations (47%) using generative AI have already experienced at least one negative consequence. The differentiator is not whether they have incidents. Everyone has incidents. It is whether each one makes the organization measurably stronger or produces a scapegoat and a repeat.</p><p>A well-run post-mortem asks what happened, why the actions taken made sense to the people involved at the time, which signals were available but missed, and what change to process or tooling would have caught it. It produces concrete improvements rather than punishment. And critically, it is only possible inside a culture of psychological safety. Amy Edmondson&#8217;s body of research on the subject, reinforced by Google&#8217;s widely cited internal study of what makes teams effective, points to psychological safety as the foundational ingredient. If reporting a near-miss gets you punished, you will stop reporting near-misses, and the misses will continue in the dark. A blame culture does not reduce failures. It reduces failure reporting, which removes your ability to see the next one coming.</p><p><strong>The third ritual is the living playbook.</strong> This is documentation people actually use, which distinguishes it from the vast majority of governance documentation, written once, approved, and then abandoned to a folder no one opens. A living playbook is a running record of how this organization handles recurring situations and assesses new use cases. It details what model cards must contain, how to escalate a fairness concern, and what to do when a vendor cannot explain its system. The playbook is a living document, updated whenever a post-mortem teaches a lesson or a design review surfaces a gap that the existing guidance did not cover. PwC&#8217;s 2025 guidance frames this exactly right, urging organizations to treat responsible AI as a living system rather than a static framework, one that is reassessed continually as the technology and the risks evolve. A playbook frozen at the moment of its creation is worse than useless because it gives people false confidence that the question has been answered when the ground has since shifted beneath them.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/building-a-culture-of-responsible?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/building-a-culture-of-responsible?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Designing the Organization for Distributed Ownership</h2><p>Rituals and incentives need a structure to hang on, but the structure that works is not a thick central bureaucracy that owns all AI decisions. That model fails in two directions at once. It becomes a bottleneck that the business learns to circumvent, letting everyone else off the hook, because if a central team owns responsibility, then no one in the business units feels they do. The organizations getting this right distribute ownership widely while keeping accountability sharp and unambiguous.</p><p>The pattern that has emerged borrows the three-lines-of-defense model, long used in risk management, and adapts it for AI. The first line consists of the people who build and operate the systems and who own day-to-day responsibility for doing it well. The second is risk and compliance, which sets standards and provides independent challenge. The third is audit and assurance, which verifies that the whole arrangement works as intended. The most important signal in the 2025 data is the direction of travel. Responsible AI is moving toward the first line. When 56% of organizations report that their engineering, data, and product teams now lead responsible AI efforts, it suggests that responsibility is being embedded where the work happens rather than imposed by a distant committee that reviews the work after the fact. That is the right direction, provided the first line is equipped and held accountable rather than handed a burden.</p><p>Distributed ownership needs connective tissue, and the most effective form I have seen in practice is the AI champion, a practitioner embedded within a business unit or engineering team who carries responsible AI fluency into the rooms where decisions get made. Microsoft built exactly this, empowering early adopters and enthusiasts as responsible AI champions who serve as anchors and resources for the developers around them, equipped with the training they need to unlock value safely. A champion is not a compliance officer parachuting in to say no. A champion is a respected peer who can answer the question, &#8220;Is this okay?&#8221; in real time, who knows when to escalate and when not to, and who models the norms through visible behavior. The exact ratio of champions to staff matters far less than the principle, which is that coverage close to the work beats authority far from it.</p><p>What distributed ownership must never blur is accountability itself. The diffusion of responsibility is the original sin of AI governance, the reason the simple question of who is in charge of this system so often produces an awkward silence. AI systems span data engineering, model development, product management, and operations, and it is dangerously easy for ownership to dissolve across those boundaries until it belongs to no one. The remedy is that every AI system has a single named owner accountable for its outcomes throughout its lifecycle, with clear escalation paths for decisions that exceed their authority. And escalation should be the exception, not the reflex. A structure that escalates everything is as broken as one that escalates nothing. The goal is to empower people to make good decisions within clear boundaries, and to make it equally clear when they need to raise their hand.</p><p>Board attention is finally catching up, which matters more than it appears, because culture is set from the top and a board that ignores AI signals that everyone else can too. Analyses of corporate board practices show the share of companies incorporating AI risk into board oversight rose to roughly 48% in 2025, up from just 16% the year before, and the share assigning AI oversight to a dedicated board committee climbed to around 40% from 11%. That is genuine progress from a low base, though it also means a substantial minority of boards remain absent from a topic now near the center of enterprise risk. This absence is itself a governance failure that the rest of the organization will eventually feel.</p><h2>Measuring What Actually Matters</h2><p>You cannot manage what you do not measure, and most organizations measure exactly the wrong thing. They count incidents. Incidents are a lagging indicator, the smoke that appears only after the fire has caught. By the time one shows up in a report, the harm is already done. A mature program watches leading indicators instead, the upstream signals that predict whether trust will hold before it is tested.</p><p>The most useful leading indicators are not hard to find once you look for them. Training engagement and demonstrated competence are measured not by whether people clicked through a module, but by whether they can make the right call in a scenario and tell you whether judgment is being built. Early risk detection rates, meaning how often concerns surface during design rather than in production, tell you whether the rituals are working. </p><p>Reporting and near-miss rates carry a counterintuitive lesson that leaders need to internalize where more reports are often good news. A rise usually means people feel safe enough to raise concerns, not that the world has grown more dangerous. A program that reports zero issues is rarely safe. It is silent, and silence is the most alarming reading on the dashboard. Coverage metrics, meaning what share of systems are monitored, owned, and carry a current model card, point directly at where the next incident is likely to originate. And the time from a concern being raised to its resolution tells you whether the organization acts on what it learns or merely logs it.</p><p>Beyond these operational metrics lie the cultural signals, which are harder to quantify but more telling as leading indicators. Do people feel safe raising concerns? Edmondson&#8217;s psychological safety research has produced validated survey instruments for exactly this, and they belong in your employee engagement survey. Is there genuine transparency about which AI systems the organization runs and how they perform, or does that knowledge reside in scattered silos? Does leadership talk about responsible AI when nothing has gone wrong, which signals a real priority, or only after a crisis, which signals a reaction? These softer signals tell you whether your harder metrics will hold up when they are finally put under pressure.</p><p>Treat these cultural signals the way your data teams treat model metrics. Baseline them, track them over time, and watch for drift, because a reporting rate that quietly declines is as meaningful as a model whose accuracy quietly degrades. Organizations that monitor their AI systems obsessively while never monitoring the health of the culture operating them have instrumented only half the problem.</p><h2>From Initiative to Institution</h2><p>The graveyard of corporate change is crowded with initiatives, launches, task forces, and carefully branded years of something that generated energy for a quarter or two but then faded once the sponsoring executive moved on. The entire purpose of building a culture, as opposed to running a program, is to create something that outlasts its champions. A few things determine whether responsible AI takes root as an institution or evaporates as an initiative.</p><p>Leadership has to model it, and model it visibly because people read behavior far more carefully than they read slogans. When a senior leader kills a promising, well-resourced project over an unresolved ethical concern and explains why to the organization, that single decision teaches more than a year of mandatory training by showing that the stated values have teeth. PwC&#8217;s research found that organizations capturing the most value tend to have leaders who engage directly with AI rather than delegate it. One pharmaceutical company brought its general counsel and hundreds of its attorneys into hands-on work with generative AI, not to police it from a distance but to understand it from the inside, and built trust and momentum that radiated outward from the top. Leaders cannot delegate the culture they are unwilling to embody.</p><p>Stories accomplish what statistics cannot. Culture is transmitted through narrative far more than through metrics, through the retold tale of the near-miss the team caught before it reached a customer, the harm avoided because one person spoke up, the launch delayed that turned out to be exactly the right delay. Collect these stories deliberately and tell them often. They become the organization&#8217;s working memory of what good looks like, the shared reference points new and old employees alike can orient against.</p><p>Onboarding is one of the highest-leverage yet most overlooked aspects of the system. Every new hire arrives as an opportunity to set the norm from day one or to let it quietly erode. If responsible AI practice is built into how people are welcomed and trained as they join, it compounds with every cohort, becoming part of what newcomers assume is how things are done here. If it is bolted on as an afterthought, the culture dilutes a little with every arrival.</p><p>Finally, the whole system has to iterate because the work is never finished. The maturity models from firms like Accenture and PwC emphasize the same truth that responsible AI is a discipline to be practiced rather than a destination to be reached. The technology will keep changing, the regulations will keep arriving, and the playbook will keep needing revision. The organizations that endure are not the ones that built the perfect framework once. They are the ones that built the habit of revision itself, the feedback loops that turn every incident, every near-miss, and every piece of friction into a slightly better practice next quarter.</p><h2>Your First Ninety Days</h2><p>You do not have to do all of this at once, and you should not try, because a program too elaborate for your current maturity will be ignored or circumvented. Culture is built in sequence, not in a single grand rollout. If you are starting, or restarting after a stalled first attempt, the most productive way to spend the first ninety days is to choose depth over breadth and prove the value of a few moves rather than announce a long list of intentions.</p><ol><li><p><strong>Audit the current state honestly.</strong> Inventory your AI systems, and then ask the more revealing question of who owns each one. The gaps in that answer are your starting risk map. Look with clear eyes at what training actually exists, what behaviors actually get rewarded, and what rituals you already have in place, including the informal ones that no one wrote down.</p></li><li><p><strong>Pilot exactly one ritual.</strong> Stand up a single structured design review for your highest-risk use case, and keep it deliberately lightweight. The objective is to prove that the conversation creates value, not to erect a bureaucracy. One good review that catches one real problem will sell the practice across the organization more persuasively than any mandate from above ever could.</p></li><li><p><strong>Align exactly one incentive.</strong> Add a responsible AI objective to the performance goals of the people who own your most consequential systems, and ensure that raising a concern earns recognition rather than a quiet penalty. A single well-chosen incentive shift signals the organization&#8217;s real priorities more loudly than a complete rewrite of the policy library.</p></li></ol><p>Three moves executed well will beat thirty attempted superficially. The aim of the first ninety days is not a finished program. It is proof, both demonstrated to yourself and visible to your organization, that responsible practice and good business are not in tension but, in fact, the same thing. That proof is what earns you the credibility and the budget to fund the next ninety days, and the ninety after that.</p><h2>Trust as the Durable Advantage</h2><p>It is tempting to file everything described here under risk management, the necessary cost of staying out of trouble. That framing badly undersells it. In a market where AI capability is commoditizing at a remarkable speed, where any competitor can license the same frontier model by the end of the afternoon, the thing that differentiates you is no longer what your AI can do. It is whether anyone can trust it. And trust is not a document you produce on demand. It is a reputation you earn slowly through the accumulated behavior of your people across thousands of decisions that no one outside the organization will ever see.</p><p>The evidence increasingly supports treating trust as a strategic asset rather than a defensive expense. PwC found that the top fifth of companies capture roughly three-quarters of AI&#8217;s returns, and that maturity in responsible practices closely tracks both value creation and resilience when something goes wrong. McKinsey found that near-universal adoption has so far produced surprisingly little scaled value, precisely because most organizations bought the technology without doing the harder work of rewiring the behavior around it. The advantage is sitting wide open. It belongs to whichever organizations are willing to do the unglamorous, repetitive, deeply human work of building a culture.</p><p>That work cannot be bought, and it cannot be installed from a vendor. It is built gradually, through what you choose to teach, what you choose to reward, and what you do over and over until it stops feeling like an initiative and becomes how things are done here. The tools and policies are the easy part, which is why so many organizations have them and so few achieve results. The control layer that determines whether your AI earns and keeps trust is the one made of people, their judgment, their incentives, and their habits.</p><p>The question is not whether your organization will need a culture of responsible AI. The question is whether you will build it deliberately, while you still have the luxury of choosing, or scramble to assemble it in the aftermath of an incident that forces your hand and frames the story for you. The first path is a strategy. The second is a cleanup. The time to choose is now.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/building-a-culture-of-responsible/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/building-a-culture-of-responsible/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Why Trusted AI Programs Stall at the Product Layer]]></title><description><![CDATA[And how to build one that turns trust into adoption rather than friction]]></description><link>https://trustedai.recodework.com/p/building-a-trusted-ai-strategy-product</link><guid isPermaLink="false">https://trustedai.recodework.com/p/building-a-trusted-ai-strategy-product</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Fri, 05 Jun 2026 19:45:43 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!U-z_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!U-z_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!U-z_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg 424w, https://substackcdn.com/image/fetch/$s_!U-z_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg 848w, https://substackcdn.com/image/fetch/$s_!U-z_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!U-z_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!U-z_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:3750083,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/200652361?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!U-z_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg 424w, https://substackcdn.com/image/fetch/$s_!U-z_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg 848w, https://substackcdn.com/image/fetch/$s_!U-z_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!U-z_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcee44040-a3d1-46a8-9bf6-2f33a295ccd3_3864x2576.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><p>One technology failure repeats across enterprise after enterprise, reliably enough to count as a pattern rather than a run of bad luck. A team builds an AI assistant for the customer support organization. It demos beautifully, answering in seconds the kind of question that used to take an agent five minutes to research, and leadership greenlights a rollout to the whole department. For two weeks, the dashboards look excellent. Then, on a single busy afternoon, the assistant confidently tells three agents something about a refund policy that turns out to be wrong, and one of those answers reaches a customer before anyone catches it. Word travels through the team faster than any training memo. Within a month, usage has fallen by more than half. The agents have quietly gone back to the old knowledge base because checking the assistant&#8217;s work took longer than not using it at all. The model was not broken. The engineering was sound. What failed was trust, and the feature is now on the list of things that did not pan out.</p><p>Here is the uncomfortable part. The organization in that story almost certainly had a governance program in place. It had principles, a policy document, perhaps a review board. None of it prevented the outcome because governance lived in one part of the company and the product in another. The policy outlined what the AI was supposed to do. It did nothing to make the product something people would actually rely on. That is the thesis worth sitting with. </p><p><strong>In most enterprises, the Trusted AI program built to make AI safe is doing almost nothing to drive adoption, and the way it is built is often part of why adoption fails.</strong></p><p>This is not a small problem hiding in a corner. McKinsey&#8217;s 2025 State of AI research found that 88% of organizations now use AI in at least one business function, yet only a sliver, around 5%, report that AI is driving significant, enterprise-level financial impact. The distance between near-universal adoption and realized value is not primarily a model gap or a talent gap. It is a trust gap. Organizations are deploying capabilities that their users will not lean on, that their customers do not feel comfortable with, and that their own risk functions cannot see into.</p><p>The mechanism behind the failure is almost always the same. Trust gets written by the governance function and handed to product teams as a set of constraints. Policies arrive after the build is already underway. Reviews feel like a tax. The people who actually decide what ships experience trust as friction rather than as something that helps them win, so they tolerate it at best and route around it at worst. A program designed that way protects the company on paper while quietly starving it of the adoption that was the entire reason for building the AI in the first place.</p><p>The strategies that work invert this completely. They make trust legible in the language product teams already speak, build it into the work rather than around it, and measure it alongside the outcomes those teams are accountable for. Trust stops being something product teams must comply with and becomes something they want because it moves the numbers they are already chasing. </p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>Trust Has Become a Product Requirement</h2><p>For most of software history, reliability was something you earned through testing and then largely took for granted. A button either worked or it did not. Quality assurance caught the cases that did not, and once a feature shipped clean, users could assume it would behave the same way tomorrow as it did today.</p><p>Emerging AI features break that assumption. They are probabilistic rather than deterministic. They can be confidently wrong. They behave differently as input data drifts away from what they were trained on, and they can produce outputs no one on the team anticipated. A feature that is right 92% of the time is not the same as a feature that works 92% of the time, because that 8% is not evenly distributed or a minor difference. If those failures are visible, high stakes, or hard to recover from, they will define the user&#8217;s experience of the entire feature.</p><p>This is why trust now sits upstream of the build rather than downstream. It is not a compliance topic to be resolved before a public launch. It is a precondition for the feature delivering value at all because a feature people do not rely on does not produce outcomes, no matter how impressive it is under the hood.</p><p>The cost of getting this wrong shows up in the metrics every product leader watches. Weak adoption comes first, as users try a feature once, get burned, and never return. Churn follows in subscription and platform businesses, where a single bad answer in a high-stakes moment can end a relationship that took years to build. Support burden climbs because every confidently wrong output becomes a ticket, an escalation, or a frustrated call. And rollout slows to a crawl, because risk, legal, and security functions are right to hesitate when they cannot see what a system is actually doing in production. Low trust is not one problem. It is a tax levied across the entire value chain of the feature.</p><p>The external picture reinforces the internal one. McKinsey&#8217;s 2025 research found that 47% of organizations have already experienced at least one negative consequence from generative AI. PwC&#8217;s customer experience work found that well over half of consumers are only somewhat comfortable, or not comfortable at all, using AI tools to engage with brands. PwC&#8217;s executive surveys tell a parallel story, with only about a third of CEOs reporting a high degree of trust in embedding AI into their key processes. The audience for your AI features, internal and external alike, arrives skeptical. Capability does not overcome that skepticism. Demonstrated reliability does.</p><p>A documented case shows how literal that cost can become. In early 2024, the British Columbia Civil Resolution Tribunal found Air Canada liable after its website chatbot told a grieving customer he could claim a retroactive bereavement discount, which was not the airline&#8217;s actual policy. Air Canada argued, remarkably, that the chatbot was a separate entity it should not be held responsible for, and that the customer should have verified the policy on another page of the same website. The tribunal rejected that, ruling that a company is accountable for what its AI tells its customers and that relying on the chatbot was entirely reasonable. The direct damages were small, a few hundred dollars. The lasting cost was everything around them. The case has become the example that commentators now reach for when explaining why a confidently wrong AI answer is a liability rather than a quirk, and no organization wants its brand cast in that role. A single unreliable output, left ungoverned at the point of use, turned an ordinary support feature into a cautionary tale that has outlived the refund many times over.</p><p>The most useful way to hold all of this is to treat trust as the conversion rate of capability into value. Raw capability is potential. Trust determines how much of that potential turns into adoption, retention, and measurable outcomes. A strategy that improves trust is not slowing the business down in the name of responsibility. It is widening the channel through which AI capabilities translate into business results.</p><h2>Define Trust in the Language Product Teams Already Speak</h2><p>The single most common mistake in Trusted AI strategy is to define trust in the vocabulary of ethics and compliance and then wonder why product teams do not prioritize it. Fairness, transparency, accountability, and human oversight are essential principles that matter. But a product manager planning a quarter is optimizing for adoption, activation, retention, conversion, and efficiency. If trust is expressed only as a moral or regulatory obligation, it will lose every prioritization fight to a feature that visibly moves one of those numbers.</p><p>The way through is to recognize that, for an AI feature, trust is not separate from those metrics. It is the hidden variable underneath them.</p><p>Consider activation. A new user&#8217;s first interaction with an AI feature is a test. If the first result is wrong, irrelevant or impossible to act on without double-checking, a user forms a judgment that is very hard to reverse. First run reliability is activation. Consider retention, which depends on users coming back, and users come back to things they can depend on. One memorable failure in a consequential moment outweighs many quiet successes. Consider conversion in any revenue-facing flow, where a hallucinated price, an incorrect product recommendation, or a fabricated policy detail does not merely cause the sale to be lost. It teaches the customer not to trust the channel. And consider efficiency, the core promise behind most enterprise AI. An assistant who produces output that a person has to verify line by line, or redo from scratch, is not saving time. Net efficiency is a function of how often the user can act on the output without checking it.</p><p>That last point deserves to be stated plainly because it reframes the whole conversation. The accuracy that matters to the business is not the model&#8217;s benchmark accuracy. It is the rate at which a user can act on an output without independently verifying it. That number is at once a trust measure and a productivity measure. It is exactly the metric a product team already understands and already cares about.</p><p>When trust is defined this way, it stops being abstract. It competes on the same scoreboard as every other feature, and it frequently wins because it moves the metrics that the product is already responsible for. A Trusted AI strategy that wants to be adopted should open every conversation with the product, not by naming principles, but by naming the outcome those principles protect. Trust is what keeps people using the feature. Everything else follows from making that connection explicit.</p><h2>Start With the Experience, Not the Model</h2><p>Here is a fact that surprises many leaders who view AI primarily as a modeling problem. Most trust is won or lost in the experience around the model, not in the model itself. A slightly weaker model wrapped in honest framing, visible uncertainty, and easy correction will usually be adopted more readily than a stronger model presented as infallible. Trust is an experience design discipline as much as a machine learning one, and treating it that way is one of the highest leverage moves available to a product organization.</p><p>Three design commitments do most of the work.</p><h4>The first is setting clear expectations. </h4><p>Users should know, at the point of use and in plain language, what the feature can and cannot do. Overclaiming is the fastest way to destroy trust, because every gap between the promise and the reality registers as a failure. It is far better to scope a feature narrowly and be specific about its domain than to imply a generality that the system cannot deliver. Telling a user that an assistant is good at summarizing their documents but should not be relied on for legal interpretation is not a product weakness. It is the kind of honesty that makes people comfortable using the part that works well.</p><h4>The second is making uncertainty visible and useful. </h4><p>This does not mean a generic disclaimer that everyone scrolls past. It means surfacing meaningful signals where they help the user decide how much to rely on a given output. Show the sources that a system grounded its answer in. Flag when it is extrapolating beyond solid ground. And, most importantly, design the system to abstain or defer when its confidence is low rather than guessing to fill the silence. This is where calibration becomes more valuable than raw accuracy. A system that reliably knows when it does not know, and says so, earns more trust than a marginally more accurate system that is uniformly confident, because the user can finally tell the strong answers from the weak ones. A confident wrong answer spends from a finite trust budget that is very expensive to refill.</p><h4>The third is giving users real control. </h4><p>People trust systems they can steer. That means easy correction, a clear override, an undo, and a visible path to a human when the machine reaches its limits. Control does more than reassure. Every correction a user makes is also a signal the organization can learn from, directly informing measurement and improvement later on. A feature that lets users fix their mistakes cheaply turns their own failures into a source of improvement rather than churn.</p><p>For technical leaders, this section is really a budget conversation. Retrieval grounding, source citation, confidence calibration and abstention thresholds are not polished and can be added at the end if time allows. They are the load-bearing structure of a trustworthy feature, and they should be funded and scheduled as core requirements from the first sprint.</p><h2>Build the Guardrails Into the Workflow</h2><p>If the experience layer is the trust the user sees, guardrails are the trust the user never notices because nothing went wrong. These are the safety, privacy and quality controls that belong inside the product workflow, embedded from the beginning rather than bolted on after the first incident makes the news.</p><p>The governing principle is to move checks early rather than late. Input validation, output validation, handling of personal and sensitive data, content filtering, and policy enforcement are dramatically cheaper and safer to apply before an output ever reaches a user. Catching a problem inside the pipeline is an engineering event. Catching it after a customer has acted on a bad answer is a trust event, and those are far more expensive to repair.</p><p>Oversight should be proportional to stakes rather than uniform. This is where the risk-based thinking long familiar to governance teams becomes a practical product tool. A low-stakes, reversible, internal feature can ship with strong monitoring and very little human intervention. A feature that touches a consequential, hard-to-reverse, or regulated decision warrants human review-in-the-loop, mandatory checkpoints, or hard gates that prevent certain actions without sign-off. The goal is not to slow everything down equally. It is to apply friction exactly where the downside justifies it, while keeping the low-risk path fast. Uniform friction is the surest way to teach product teams that governance is the enemy of shipping.</p><p>Every AI feature also needs defined escalation paths for the moments that fall outside the happy path. When the system makes an error, encounters an edge case it cannot handle, or a user raises a complaint, there must be a clear answer to who responds, how the affected user is made whole, and how the incident feeds back into improvement rather than vanishing into a support queue. This is not only good practice. For higher-risk applications, it is increasingly a legal requirement.</p><p>It helps to retire the idea that guardrails are the brake on AI velocity. A better metaphor is the seatbelt. They are the controls that let an organization move fast precisely because the consequences of a mistake are contained. Teams that trust their guardrails ship more, not less, because they are not waiting nervously for the failure they have no way to catch.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/building-a-trusted-ai-strategy-product?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/building-a-trusted-ai-strategy-product?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Make Trust Measurable</h2><p>Anything that is not instrumented eventually reverts to opinion, and opinion loses to hard numbers when priorities are set under pressure. If a Trusted AI strategy cannot show trust as data, trust will be the first thing cut when a deadline looms. So the strategy has to make trust measurable and place those measures right next to the product metrics teams already track.</p><p>The practical move is to pair each business metric with the trust signal that drives it. The table below shows the pairing.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xf4N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xf4N!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png 424w, https://substackcdn.com/image/fetch/$s_!xf4N!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png 848w, https://substackcdn.com/image/fetch/$s_!xf4N!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png 1272w, https://substackcdn.com/image/fetch/$s_!xf4N!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xf4N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png" width="1456" height="949" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:949,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:211856,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/200652361?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xf4N!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png 424w, https://substackcdn.com/image/fetch/$s_!xf4N!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png 848w, https://substackcdn.com/image/fetch/$s_!xf4N!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png 1272w, https://substackcdn.com/image/fetch/$s_!xf4N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1fb288af-fdb9-41e3-a3c2-7f851215ea9c_2039x1329.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>A few of these deserve emphasis because they are so easy to get wrong. The most important technical idea in measurement is to track the shape of failure, not just the average. A feature that is 92% accurate but fails catastrophically in its rare misses is less trustworthy than one that is 90% accurate but whose errors are minor and easy to recover from. Average accuracy hides the very thing that determines trust, which is what happens when the system is wrong. Tracking the distribution of error severity, rather than a single accuracy headline, is what tells you whether a feature is safe to expand.</p><p>Calibration deserves the same scrutiny. A confidence score is only useful if it reflects reality, so measure calibration error directly and treat a well-calibrated, honest system as a goal in its own right rather than a byproduct. Validate the feedback you collect as well. A thumbs-up from a user who did not notice the answer was wrong is not evidence of quality, so sample real outputs against the ground truth instead of trusting raw satisfaction signals at face value.</p><p>None of this is a one-time launch exercise. It depends on an evaluation suite that runs continuously, structured red teaming that actively hunts for failure modes before users find them, and production monitoring that watches for the data drift and concept drift that quietly degrade models over time. This is the operational discipline the broader field calls observability, and it is the difference between knowing what your AI is supposed to do and knowing what it is actually doing. Those two things are not the same, and only the second one earns trust.</p><p>Finally, all of this has to feed a loop. Corrections, complaints, escalations and failed evaluations should flow into a backlog that improves the system, so that trust compounds over releases rather than plateauing at launch. The products that win are not the ones that launched most trustworthy on day one. They are the ones whose trust curve kept climbing because every failure was converted into an improvement.</p><h2>Align Teams Around a Shared, Lightweight Operating Model</h2><p>Trust is irreducibly cross-functional, which is exactly why it falls through the cracks. Building a trustworthy AI feature touches product, design, engineering, data science, legal and privacy, security, and operations. When trust responsibility belongs to everyone in general, it belongs to no one in particular, and the seams between teams are precisely where trust failures breed.</p><p>The remedy is to make ownership explicit without inventing a bureaucracy. The product owns the outcome and, therefore, the trust requirements that protect it. Design owns expectation setting and the control surfaces that let users steer and correct. Engineering and data science own the evaluations, guardrails, and monitoring infrastructure. Legal and privacy own the mapping to regulation and the rules for handling data. Security owns resilience against adversarial manipulation. Operations owns incident response and the escalation paths. The titles matter less than the principle, which is that for every dimension of trust, a specific function can answer for it by name.</p><p>The governance that ties these roles together has to be lightweight, and lightweight carries a precise meaning here. It describes governance that helps teams move faster, not a committee that reviews everything and becomes a bottleneck. PwC&#8217;s 2025 Responsible AI research found that roughly half of executives name the translation of principles into operational processes as their single biggest hurdle, and only about half have even completed a basic inventory of their AI use cases. The organizations that clear that hurdle do so by embedding governance into the work, automating it where possible, and right-sizing it to risk. Templates appear at project initiation. Automated gates verify that the required documentation and evaluations are in place before a build can proceed. Dashboards surface issues without demanding a meeting. Compliance becomes the path of least resistance rather than a detour around it.</p><p>The most direct way to embed trust into a product roadmap is to give every AI feature a trust definition of done. Just as a feature is not finished until it meets its functional acceptance criteria, an AI feature should not be considered shippable until it meets its trust acceptance criteria, decided at the same moment as the functional requirements rather than negotiated after the fact. Those criteria scale to the feature&#8217;s risk tier, as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!VeSp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!VeSp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png 424w, https://substackcdn.com/image/fetch/$s_!VeSp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png 848w, https://substackcdn.com/image/fetch/$s_!VeSp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png 1272w, https://substackcdn.com/image/fetch/$s_!VeSp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!VeSp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png" width="1456" height="707" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:707,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:238119,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.recodework.com/i/200652361?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!VeSp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png 424w, https://substackcdn.com/image/fetch/$s_!VeSp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png 848w, https://substackcdn.com/image/fetch/$s_!VeSp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png 1272w, https://substackcdn.com/image/fetch/$s_!VeSp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d59cd1-38fb-4579-9fc5-f29eb6ad9fd4_2380x1156.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This single mechanism does much of the work that a Trusted AI strategy is supposed to do. It connects trust to the roadmap, scales effort to risk, and gives product teams a clear, predictable standard rather than a moving target to guess at. Shared standards finish the picture. A common feature documentation format, often called a model card, an agreed evaluation rubric, and a documented review and decision process, all let teams move quickly precisely because they are not re-litigating the fundamentals on every project. Standards are not bureaucracy. They are the reason a team does not have to reinvent trust from zero each time it starts something new.</p><h2>Roll Out in Phases</h2><p>Ambition is not a rollout strategy. The organizations that scale Trusted AI do so by deliberately sequencing, matching the order of deployment to risk and value rather than to enthusiasm or to whoever lobbied hardest.</p><p>Begin with low-risk, high-value use cases. Internal productivity tools, assistive features that keep a human in the loop, and reversible actions are ideal first steps because they enable a team to capture real value and learn the operating model in an environment where the downside of a mistake is contained. The point of starting small is not timidity. It is to build the organizational muscle to detect and recover from failure before the stakes are high enough to punish a beginner&#8217;s mistakes.</p><p>Use pilots to find where trust breaks down, not only to prove it works. A good pilot is heavily instrumented and judged against exit criteria defined before launch, including both product metrics and trust signals that must hold for the feature to graduate. The most valuable output of a pilot is often the catalog of failure modes, confusing interactions, and unexpected support load it surfaces, because those are precisely the things that would have quietly killed the feature at scale. A pilot that only confirms the happy path has not done its real job.</p><p>Expansion should be gated by readiness, not by the calendar. A feature earns the right to move to a higher-stakes use case once its experience, metrics, and controls have proven stable at the current tier, and the team has demonstrated it can catch and recover from failures. Promotion gates of this kind keep an organization from sleepwalking into a high-risk deployment it is not actually prepared to run.</p><p>Timing is not abstract here. The European Union&#8217;s AI Act brings its obligations for high-risk AI systems into force on the second of August, 2026, placing concrete legal requirements around exactly the kind of consequential applications, such as hiring and credit decisions, that sit at the top of the risk ladder, with penalties that can reach tens of millions of euros or a meaningful percentage of global turnover. Phasing protects an organization from shipping into a regulated tier without the documentation, human oversight, and risk management that those rules require. A disciplined rollout is not only good product practice. It is increasingly the difference between a compliant deployment and an expensive after-the-fact remediation.</p><h2>The Strategic Payoff</h2><p>Return to the gap that opened this piece. Adoption of AI is nearly universal, and realized value remains rare. The distance between the two is the trust and reliability gap, and closing it is where the durable returns live. A Trusted AI strategy is, at bottom, the discipline of converting experimentation into dependable adoption, which is another way of saying it converts capability into value.</p><p>This is why the strongest AI strategies are not merely the most powerful ones. They are also understandable, usable, and dependable. Capability gets an organization a demo and a headline. Trust is what gets a business. The work described here, defining trust in product terms, designing for the experience, building guardrails into the workflow, measuring trust as rigorously as any other outcome, aligning teams around a light operating model, and rolling out in phases, is the work of turning an impressive capability into something people will actually rely on day after day.</p><p>It also resolves the adoption problem that defeats most governance-led efforts. Product teams adopt what helps them ship better outcomes with less friction. When a Trusted AI strategy is framed and built so that trust moves the metrics by which those teams are measured and removes the blockers that slow them down, adoption stops being a mandate to enforce and becomes a choice teams make because it serves their own goals. That is the only kind of adoption that lasts. And the value on the other side is real for those who get there. PwC&#8217;s 2025 Responsible AI research found that nearly sixty percent of executives say responsible AI practices boost return on investment and efficiency, and a majority report gains in customer experience and innovation. Trust, done well, is not a cost center. It is a growth lever.</p><p>The question facing every organization is not whether its AI features will need to be trustworthy. The market, the regulators, and the users have already settled that. The question is whether trust will be deliberately designed into the strategy, on terms that product teams embrace because it helps them win, or retrofitted reactively after the adoption curve has flattened. Failures have taught users to look elsewhere. The first path produces AI that scales. The second produces a portfolio of impressive features nobody relies on. The time to choose the first path is now.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/building-a-trusted-ai-strategy-product/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/building-a-trusted-ai-strategy-product/comments"><span>Leave a comment</span></a></p>]]></content:encoded></item><item><title><![CDATA[Trusted AI Has a Shelf Life]]></title><description><![CDATA[How to maintain reliability, detect drift, and manage change across the AI lifecycle]]></description><link>https://trustedai.recodework.com/p/trusted-ai-has-a-shelf-life</link><guid isPermaLink="false">https://trustedai.recodework.com/p/trusted-ai-has-a-shelf-life</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Sat, 30 May 2026 17:43:50 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Neqp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Neqp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Neqp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Neqp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Neqp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Neqp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Neqp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg" width="800" height="398" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:398,&quot;width&quot;:800,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:119478,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/199883925?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Neqp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Neqp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Neqp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Neqp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd245e832-72f1-45db-83c3-bee0bf2f4152_800x398.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In April 2025, OpenAI shipped an update to the model behind ChatGPT, one of the world's most widely used AI products. The intent was to make it feel more helpful and intuitive. Instead, within days, the model turned strikingly sycophantic, lavishing praise on users and validating their choices so eagerly that it began endorsing plainly poor and even harmful ideas. The change reached an enormous user base before anyone outside the company could have known it had been made at all.</p><p>What makes the episode instructive is not that something went wrong. It is how thoroughly the usual safeguards failed to catch it. The update had cleared the company&#8217;s offline evaluations. It had cleared a live test, and users who tried it seemed to like it. By every quantitative measure the team relied on, the new version looked fine. The clearest warning came not from a metric but from a handful of expert reviewers who said the model felt off. OpenAI recognized the problem over the weekend, applied a quick mitigation via its system instructions, and then rolled the model back to the previous version. The whole arc, from release to reversal, took roughly four days.</p><p>This is the central and uncomfortable truth of operating AI at scale. The day you deploy a model is not the finish line. It is the starting line for a slower, quieter category of risk that most governance programs are not designed to catch. A system can pass every test you put in front of it and still drift into behavior you never approved. And if the most sophisticated AI lab in the world can ship a regression that its own evaluations did not flag, the question for the rest of us is not whether it will happen, but whether we will notice in time and be able to undo it.</p><p>Trusted AI requires two capabilities working in concert. Governance tells you what your AI should do. Observability tells you what your AI is doing. Lifecycle management is where these two disciplines meet across time. It is the practice that keeps a system trustworthy not just on launch day, when everyone is watching, but in month nine, when attention has moved on, and the data has quietly shifted.</p><p>This issue is about that long arc. It will look at what reliability actually means for an AI system, what changes after deployment, how to detect those changes before your customers do, and how to version and release AI components so that change is safe, traceable, and reversible. None of this is glamorous, but it is what separates AI programs that compound value from those that quietly accumulate liability.</p><h2>Reliability Is Not a Launch Event</h2><p>Most organizations treat AI approval as a gate. A model is built, tested, reviewed and signed off. Once it clears the gate, attention shifts to the next initiative. The implicit assumption is that a system that passed review will continue to behave as it did during review.</p><p>For traditional software, that assumption is mostly defensible. A function that sorts a list correctly today will sort it correctly next year, because its behavior is fully specified by its code. AI systems break this assumption fundamentally. They do not encode fixed rules. They encode patterns learned from data, and those patterns are only as durable as the relationship between the data they learned from and the world they operate in. When that relationship shifts, behavior changes, even though not a single line of code was touched.</p><p>This is why AI reliability has to be defined more carefully than conventional software reliability. A useful definition has four dimensions, and senior leaders should be able to articulate all four for any consequential system in their portfolio.</p><p>The first is accuracy, meaning the system produces correct or high-quality outputs against some ground truth or quality standard. The second is stability, meaning performance does not swing unpredictably from day to day or cohort to cohort. The third is consistency, meaning similar inputs produce similar outputs, and the system does not contradict itself. The fourth is predictability, meaning that the people who depend on the system can reasonably anticipate how it will behave, including on inputs it has not seen before.</p><p>Notice that accuracy is only one of the four. A model can be statistically accurate and still fail the business. A loan model may be accurate on average and still produce volatile, hard-to-explain decisions that erode applicant trust and invite regulatory scrutiny. This is the gap between technical accuracy and business usefulness, and it is one of the most important distinctions a leader can hold onto. Your data science team optimizes the former. Your customers, regulators, and board care about the latter.</p><p>Predictability deserves special emphasis because it is the dimension most directly tied to trust. Stakeholders do not need an AI system to be perfect. They need it to be dependable. A system that is right 90% of the time in a known and stable way is often more valuable than one that is right 93% of the time but occasionally produces baffling, high-stakes errors with no warning. Predictability is what lets a human reviewer know when to trust the system and when to look closer. It is what lets an auditor reconstruct why a decision was made. It is what lets a customer believe that the institution behind the system is in control.</p><p>The discipline that preserves all four dimensions over time is lifecycle management. And the first step in lifecycle management is understanding precisely what tends to go wrong after launch.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>What Changes After You Deploy</h2><p>Several distinct things can degrade a working AI system in production. They have different causes, different signatures, and different remedies, and conflating them is one of the most common reasons teams fail to respond effectively. Leaders do not need to implement the detection methods themselves, but they should understand the categories well enough to ask the right questions.</p><h4>Data Drift</h4><p>Data drift is a change in the inputs your system receives. The statistical properties of production data differ from those of the data used to train and validate the model. New customer segments arrive. A marketing campaign shifts the mix of incoming traffic. A sensor is recalibrated. A competitor exits a market, and your applicant pool changes overnight. The model has not changed, and the underlying relationships may not have changed, but the model is now being asked about a population it knows less well. Performance usually erodes gradually, which is part of what makes data drift so dangerous. There is rarely a single alarming moment, just a slow slide that is easy to rationalize until it becomes expensive.</p><h4>Concept Drift</h4><p>Concept drift is more insidious because the relationship between inputs and outcomes itself has changed. The features you measure may look the same, but their meaning has shifted. The clearest illustration is Zillow, which in November 2021 shut down the home buying business it had bet much of its future on after its pricing model could no longer keep pace with the market. The home characteristics the model relied on were still being measured accurately. What broke was the relationship between those characteristics and future sale prices, as the pandemic housing market moved in directions and at speeds the algorithm had not been built to anticipate. Zillow kept acquiring homes at prices it could not recoup at resale, and, by most accounts, the damage exceeded half a billion dollars, including a single-quarter write-down of more than $300 million and a workforce cut of roughly a quarter. The algorithm had not been hacked or sabotaged. The world it was trained to predict had moved out from under it. Spam filters and fraud models face the same kind of drift continuously as adversaries adapt, and demand forecasts faced it violently in early 2020, when consumer behavior shifted so abruptly that models trained on years of stable history became worse than useless almost overnight. Concept drift cannot be fixed by feeding the model more of the same kind of data. It usually demands retraining to adapt to the new reality, and sometimes a fundamental rethink of the approach.</p><h4>Behavior Drift</h4><p>Behavior drift is the failure mode that has grown most important with the rise of large language models, and it is the one most likely to blindside organizations that came to AI through generative tools rather than traditional machine learning. Behavior drift is a change in the system&#8217;s outputs or decisions that is not explained by your own deliberate changes. With classical models, behavior is relatively stable because you control the model weights. With models accessed through a provider&#8217;s interface, you often do not. Providers update the models behind stable version names. They recalibrate safety filters, which can silently raise refusal rates on inputs that were previously handled. A well-documented study from Stanford and Berkeley in 2023 measured meaningful changes in a major commercial model&#8217;s behavior over just a few months, across tasks ranging from math to code generation, with no action required from the customers relying on it. Your code did not change. Your prompts did not change. The model underneath you did.</p><p>Behavior drift also has an internal source entirely within your control. It is a prompt drift and is routinely mismanaged. In a generative system, the prompt is not a harmless configuration string. It is functioning code. It determines output format, reasoning path, tone, and safety behavior. Yet prompts are often edited casually, by multiple people, without version control, testing, or records of who changed what and why. The result is a slow accumulation of small, undocumented edits that degrade quality in ways no one can trace. When something finally breaks, the team faces a combinatorial nightmare. Was it last week&#8217;s prompt tweak, a shift in the kinds of questions users are asking, a quiet update from the model provider, or some interaction among all three? Without disciplined versioning, root cause analysis collapses into guesswork.</p><h4>Regression</h4><p>Layered on top of these is a fourth phenomenon that is not drift at all but is often confused with it. Regression is when a new version of a model or prompt performs worse than the version it replaced. Drift happens to you. Regression you do to yourself, usually with good intentions. A team re-trains a model on fresh data and unknowingly degrades performance on an important edge case. An engineer rewrites a prompt to fix one problem and silently breaks three others. Regression is especially treacherous because it arrives disguised as progress. The whole point of the change was to make things better. The danger is that no one verified that it actually did.</p><p>Understanding these phenomena reframes the governance challenge. The question is not only whether a system was safe to deploy. It is whether the system is still the system you approved, and whether you would know if it were not.</p><h2>Monitoring the Right Signals</h2><p>You cannot manage what you cannot see. The reason drift and silent regression are so costly is that, by default, they are invisible. The system keeps returning results. The infrastructure stays green. Latency looks fine. Nothing throws an error. The model is quietly getting worse, and every operational dashboard says everything is healthy.</p><p>Effective monitoring works across four layers of signal, and a mature program watches all of them rather than fixating on any single one.</p><h4>Inputs</h4><p>The first layer is the inputs. Monitoring input distributions is the earliest possible warning of trouble, because changes in what the system is being asked about often precede changes in how well it performs. For structured data, teams use statistical measures such as the population stability index, the Kolmogorov-Smirnov test, and divergence measures like Kullback-Leibler or Jensen-Shannon to quantify how far current inputs have moved from a reference baseline. For unstructured inputs such as text, embedding-based drift detection serves the same purpose: measuring whether the semantic distribution of incoming queries has shifted. A business leader does not need to know the formulas. The leader needs to know that input monitoring exists, that it has thresholds, and that someone is alerted when those thresholds are crossed.</p><h4>Outputs</h4><p>The second layer is the outputs. Tracking output quality directly is more meaningful than tracking inputs alone, because it measures what people actually experience. Where ground truth is available quickly, this can be straightforward accuracy tracking. Often, though, ground truth is delayed. You may not know for months whether a loan was a good one. In those cases, teams monitor proxy signals such as shifts in the distribution of predictions, the rate of low confidence outputs, or the frequency of outputs that fail automated validation. For generative systems, automated evaluation has become essential. A common pattern uses a separate model as a judge, sampling a fraction of production outputs and scoring them against a quality rubric in the background, to produce a continuous quality signal that does not depend on waiting for human complaints.</p><h4>Operations</h4><p>The third layer is operations. Exception rates, validation failures, fallback triggers, refusal rates, latency, and cost per request are all reliability signals. A sudden spike in malformed outputs is one of the earliest and clearest indicators of a provider-side change or a broken prompt. A creeping increase in how often the system refuses or hedges can signal that a safety layer was recalibrated upstream. These operational metrics are cheap to collect and should be instrumented from day one.</p><h4>Impact</h4><p>The fourth layer, and the one that matters most to the business, is downstream impact. Technical metrics can all look acceptable while the business outcomes the system was meant to drive quietly erode. Conversion rates, approval rates, customer satisfaction, complaint volume, manual override rates, and revenue per interaction are the signals that tell you whether the system is still doing its job in the only terms that ultimately count. The strongest monitoring programs connect model-level metrics to these business-level outcomes, so that a drift alert can be translated into a dollar figure and a decision.</p><p>Two practices tie these layers together. The first is the use of baselines and golden sets. A baseline is a documented snapshot of how the system performed at a known good moment, against which current behavior can be compared. A golden set, sometimes called a reference or evaluation set, is a curated collection of representative, high-stakes cases with known correct answers. Running every candidate version against the same golden set lets you compare versions on equal footing and catch regressions before they ship. For generative systems, the golden set is not a static infrastructure to build once and forget. It must evolve as users discover new ways to use the system, or it will rot and stop reflecting reality.</p><p>The second practice is keeping humans in the loop where automation is not enough. Automated metrics are necessary but not sufficient, particularly for nuanced judgments about tone, fairness, factual accuracy, and harm. A disciplined program samples production cases for human review on a regular cadence, escalates ambiguous cases to qualified reviewers, and treats the patterns surfaced by human review as a feedback signal that improves both the system and the automated evaluations themselves. The goal is not to review everything, which is impossible at scale, but to review enough, in the right places, to catch what the machines miss.</p><p>The platforms that support this work have matured considerably. Tools such as Arize, Evidently and Fiddler provide drift detection and performance monitoring for traditional machine learning. A growing ecosystem of evaluation and observability tools has emerged specifically for generative systems. The technology is no longer the obstacle. The obstacle is organizational with the decision to treat monitoring as a standing operational responsibility rather than a one-time validation step.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/trusted-ai-has-a-shelf-life?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/trusted-ai-has-a-shelf-life?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Versioning Across the Lifecycle</h2><p>Monitoring tells you that something has changed. Versioning is what lets you understand what changed, reproduce it, and undo it. It is the connective tissue of lifecycle management, and it is where many organizations are weakest, because versioning has historically been treated as an engineering hygiene concern rather than a governance requirement.</p><p>The first principle is to version everything that can influence behavior, not just the model. In a traditional machine learning system, that means the model artifact, training data, features, evaluation set, and configuration. In a generative system, the surface area is larger and less obvious. You must version the prompts, the system instructions, the model identifier and provider version, the retrieval sources and indexes if you use retrieval, the tool definitions if the system can take actions, and the evaluation rubrics themselves. Each of these can change outputs. Each of these is therefore part of what you approved, and each must be tracked.</p><p>Prompts deserve a specific warning, because they are the component most likely to be mismanaged. Because a prompt is just text, it feels editable in a way that code does not. People change it in a hurry, in a shared document, in a console, without recording the change. This is how prompt drift takes hold. The discipline that prevents it is to treat prompts exactly as you treat code. Every prompt version gets an identifier. Every change is recorded with an author, a timestamp, and a rationale. Every change is tested against the golden set before it reaches production. Prompts belong in version control, not in someone&#8217;s notes.</p><p>The second principle is lineage. For every output a system produces, and certainly for every consequential decision, you should be able to reconstruct the exact combination of model version, prompt version, data version, and configuration that produced it. This is what makes a system auditable. When a regulator, a customer, or your own risk committee asks why a particular decision was made, the answer cannot be a shrug. Lineage is also what makes a system reproducible, which is the foundation of any serious investigation. If you cannot recreate the conditions under which a problem occurred, you cannot reliably diagnose or fix it.</p><p>There is an important caveat here that leaders relying on third-party models must internalize. When you call a model through a provider&#8217;s interface, you do not control the weights, and you may not be able to reproduce a past output exactly, even if you have versioned everything on your own side. This is a real limitation, and the response is not to abandon versioning but to compensate for the gap. Pin specific model versions rather than floating to the latest. Log not only your inputs and configurations but the actual outputs you received, so you have a record even when you cannot regenerate it. Treat any provider model update as a change that triggers your full evaluation process, exactly as you would treat a change you made yourself. The lack of control over the model is precisely why the discipline around everything else has to be tighter.</p><p>The third principle, and the one that elevates this from engineering to governance, is that version control is part of your control environment. The same logic that requires accountability and documentation for AI decisions requires that the components producing those decisions be versioned, traceable, and reviewable. Model registries, data versioning systems, and experiment tracking tools are not merely productivity aids for data scientists. They are the systems of record that make Trusted AI auditable. They should be treated, governed, and resourced accordingly, with the same seriousness an organization brings to its financial systems of record.</p><h2>Safe Release and Safe Rollback</h2><p>If versioning is the connective tissue, release management is the immune system. It is what stops a bad change from reaching everyone at once, and what lets you recover quickly when something gets through. The objective is not to prevent all change, which would be both impossible and self-defeating. The objective is to make change safe.</p><p>Safe release begins before deployment, with explicit release criteria. Too many AI deployments are governed by a vague sense that the new version seems better. That is not a standard. A release criterion is a documented, measurable bar that a candidate version must clear before it goes live. It typically includes minimum performance on the golden set, no regression beyond a defined tolerance in key segments and edge cases, fairness metrics within agreed-upon thresholds, and operational characteristics such as latency and cost within budget. Defining these criteria before you build the change protects you from the powerful temptation to rationalize a release after you have invested effort in it. The criteria should be tied to the system&#8217;s risk tier, with high-risk systems facing more stringent bars and more sign-offs, consistent with the tiered decision rights we have described in earlier issues.</p><p>For consequential systems, release should be progressive rather than all at once. A range of techniques borrowed from modern software delivery applies directly to AI. Shadow deployment runs the new version alongside the current one on real traffic without acting on its outputs so that you can compare them safely. Canary releases a small fraction of traffic to the new version and monitors closely before rolling it out more broadly. Champion-and-challenger setups keep the proven version in control while a candidate proves itself on live data. Staged rollouts expand gradually across segments or geographies. Each of these limits the blast radius, which is the amount of damage a bad version can do before you catch it. For a low-risk internal tool, a simple staged rollout may suffice. For a credit or hiring system, shadow and canary evaluation is not optional.</p><p>The discipline that too many organizations neglect is the rollback path. Every consequential AI component should have a defined, tested, fast way to revert to the last known good version. This sounds obvious, and yet it is routinely missing, especially for prompts and configuration, which feel too lightweight to need a recovery plan. They are not. The ability to roll back a prompt to its previous version in minutes, rather than reconstructing it from memory under pressure during an incident, is the difference between a contained event and a prolonged one. Feature flags, version pinning, and keeping the prior version warm and ready are the mechanisms that enable fast rollbacks. A rollback path that exists only in theory, and has never been tested, is not a rollback path.</p><p>Return to the episode that opened this issue. A sycophantic model update became a four-day inconvenience rather than a prolonged crisis only because a known-good prior version was ready, and the team could revert to it the moment their judgment told them to, even though their formal metrics had not raised the alarm. That is what a working rollback path buys you. It converts the inevitable bad release from a disaster into an incident. The alternative is rebuilding the previous behavior from memory while customers are affected and the clock is running.</p><p>Underneath all of this lies the requirement that ties release management back to governance: auditability. Every release decision should be logged. Who approved this version? Against what criteria? What did the evaluation show? When did it go live, to whom, and in what stages? When something went wrong, what was the recovery action, who authorized it, and when? This log is not bureaucratic overhead. It is the evidence base that lets you demonstrate control to regulators, learn from incidents, and hold the right people accountable. Recall that while roughly eighty percent of organizations have established AI ethics guidelines, only about a quarter have operationalized them. An auditable release and rollback process is exactly the kind of operational muscle that closes that gap. It turns a stated commitment to responsible AI into a documented, demonstrable practice.</p><h2>An Operating Model for Long-Term Trust</h2><p>The techniques described so far are necessary but not self-executing. Drift detection does not happen because a tool exists. Rollback paths do not stay tested because someone once thought it was a good idea. Lifecycle management endures only when it is built into how the organization operates, with clear ownership, defined processes, and a regular rhythm. This is where lifecycle management connects to the broader governance operating model.</p><p>Ownership is the foundation. Every consequential AI system needs a named owner who is accountable not only for its launch but for its ongoing performance. In earlier issues, we described three levels of accountability: strategic accountability for the overall program, tactical accountability for individual systems, and operational accountability for day-to-day monitoring and incident response. Lifecycle management lives primarily at the tactical and operational levels. Someone must own the monitoring dashboards and respond to the alerts. Someone must own the scheduled reviews. Someone must have the authority to trigger a rollback without convening a committee when a system is causing harm. If you cannot name that person for a given system, you do not have lifecycle management for it. You have hope.</p><p>The processes should be embedded into the organization&#8217;s MLOps or LLMOps practice rather than bolted on. Drift detection belongs in the monitoring pipeline. Regression testing against the golden set belongs in the release pipeline, as an automated gate that a change must pass before it can ship. Rollback procedures belong in the operational runbooks and are tested on a schedule, the way disaster recovery is tested, not improvised during a crisis. When governance requirements live outside the normal workflow, they are treated as afterthoughts and quietly skipped. When they are embedded into the workflow, compliance becomes the path of least resistance, which is the only kind of compliance that survives contact with deadlines.</p><p>The cadence matters as much as the mechanics. The most common failure in lifecycle management is that systems are reviewed only when something breaks. By then, the damage is done. A mature program reviews AI systems on a regular schedule as a matter of routine, not only in response to incidents. The review asks a consistent set of questions. Has performance drifted against the baseline? Have the input distributions moved? Are the business metrics holding? Is the documentation current? Is the rollback path still tested and ready? Are the model and prompt versions still the ones we approved? These periodic reviews should feed into the governance forums we have described before, with results escalated to a risk committee or AI council in proportion to the system&#8217;s risk tier. High-risk systems warrant frequent, formal review. Lower-risk systems can be reviewed less often, but never with no frequency at all.</p><p>Finally, the program itself should be measured. Useful indicators include the share of production AI systems under active monitoring, the time it takes to detect a drift or regression event, the time it takes to roll back once a problem is identified, the number and severity of incidents over time, and the proportion of releases that passed through the defined criteria and review. These metrics tell leadership whether lifecycle management is real or aspirational, and they convert an abstract commitment into something a board can actually oversee.</p><h2>The Path Forward</h2><p>The organizations that will be trusted with the most consequential AI are not the ones that deploy the most models. They are the ones who can demonstrate, with evidence, that the models they deployed still behave the way they were approved to behave. That capability is not built in a sprint. It is built through the unglamorous, compounding discipline of watching systems in production, versioning the components that drive them, and making change safe and reversible.</p><p>Start where the risk is highest. Identify the AI systems whose failure would cause the most harm to customers, to the business, or to your standing with regulators, and ask a blunt set of questions about each one. Would we know if it drifted? Do we have a baseline to compare against? Can we reproduce any of the decisions it made? Can we roll it back to a known-good state in minutes, and have we tested that? Is there a named person accountable for its ongoing behavior? For most organizations, the honest answers reveal a portfolio of systems that were carefully approved and then left to fend for themselves.</p><p>Then build the muscle deliberately. Establish monitoring for your highest risk systems first. Put prompts and configuration under version control alongside your models. Define release criteria and a tested rollback path before your next deployment, not after your next incident. Schedule reviews and assign owners. None of this requires solving every problem at once. It requires treating reliability as an ongoing discipline rather than a launch day achievement.</p><p>Place the two stories side by side. The GPT-4o rollback shows a regression that slipped past every formal metric, was caught by human judgment, and was reversed within days because the capability to reverse it existed. Zillow shows the other path, a model that drifted while the business kept acting on its outputs, with the bill arriving in the hundreds of millions before anyone changed course. The difference between those outcomes was not the sophistication of the model or the talent of the team. It was lifecycle management. The lesson is not that models cannot be trusted. It is that models cannot be trusted to stay still, and that the organizations which thrive will be the ones that build the capability to notice when their systems change, and to respond before the change becomes a headline. Governance tells you what your AI should do. Observability tells you what your AI is doing. Lifecycle management is what keeps those two things aligned long after the launch, which is precisely where trust is either kept or lost.</p><p>Trusted AI has a shelf life. The work is in extending it, deliberately and on your own terms.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/trusted-ai-has-a-shelf-life/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/trusted-ai-has-a-shelf-life/comments"><span>Leave a comment</span></a></p>]]></content:encoded></item><item><title><![CDATA[The Feedback Gap]]></title><description><![CDATA[How in-product flags, annotation programs, and RLHF-style pipelines turn feedback into safer, smarter AI systems]]></description><link>https://trustedai.recodework.com/p/the-feedback-gap</link><guid isPermaLink="false">https://trustedai.recodework.com/p/the-feedback-gap</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Wed, 27 May 2026 12:10:53 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!ZN1d!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZN1d!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZN1d!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ZN1d!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ZN1d!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ZN1d!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZN1d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/edc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:4536265,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/199268653?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ZN1d!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ZN1d!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ZN1d!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ZN1d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc2d70f-3bca-4c77-b69b-ffdd44712276_3864x2576.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><p>Every AI system you deploy starts degrading the moment it hits production. The data shifts, user behavior evolves, and edge cases emerge that no test suite anticipated. This is not a failure of engineering. It is the nature of AI.</p><p>What separates organizations that sustain high-performing AI from those that lurch from incident to incident is not whether their models degrade. All models degrade. The difference is whether the organization has built the operational machinery to detect that degradation, capture the signals that explain it, and convert those signals into meaningful improvements.</p><p>That machinery is the feedback loop.</p><p>Governance structures, monitoring infrastructure and operating models make responsible AI scalable. Feedback loops sit at the intersection of all three. They are the connective tissue between observability (what your AI is actually doing) and governance (what your AI should be doing). Without them, monitoring generates dashboards that nobody acts on, and governance produces policies that nobody verifies.</p><p>The organizations that get this right treat feedback not as a passive byproduct of deployment but as a structured input to their AI improvement process. They build systems to collect it, pipelines to process it, and workflows to convert it into better prompts, safer outputs and more reliable models. Enterprises with structured feedback loops reduce AI escalation rates by 38% on average within twelve months, according to McKinsey&#8217;s 2025 analysis. Closed feedback loops achieve 99.4% audit completeness on high-risk AI decisions, compared to just 61% for teams without them, per IDC research.</p><p>This article breaks down how to build those loops, from the channels that capture feedback to the operating model that turns it into action.</p><h2>Why Feedback Loops Are a Governance Imperative</h2><p>The case for feedback loops extends beyond model performance. It is increasingly a regulatory requirement.</p><p>The EU AI Act, Article 61, requires post-market monitoring systems for high-risk AI providers. This effectively mandates feedback loop infrastructure. Organizations deploying AI in credit decisions, hiring, medical recommendations, or safety-critical systems must demonstrate mechanisms for detecting problems after deployment and processes for addressing them.</p><p>The NIST AI Risk Management Framework reinforces this through its Measure and Manage functions, which call for continuous tracking of AI risks and structured responses when those risks materialize. ISO/IEC 42001 includes specific requirements for AI system operation, monitoring, and documentation of user and stakeholder feedback.</p><p>But the regulatory case is secondary to the operational one. AI systems deployed without feedback loops have a fixed performance ceiling set at deployment time. As data distributions shift, as new product lines launch, as customer behavior changes, performance declines invisibly. Organizations discover the problem only when a customer complains, an audit fails, or an incident makes the news. By then, the damage is done.</p><p>Feedback loops transform AI from a static deployment into a learning system. They close the gap between what your AI is supposed to do and what it is actually doing. And they provide the evidentiary basis for governance bodies to make informed decisions about whether an AI system should continue operating, be retrained, or be retired.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>The Three Feedback Channels</h2><p>Not all feedback is created equal. The signals that flow into your AI improvement process come from three distinct channels, each with different characteristics, strengths, and limitations. Effective feedback operations draw from all three.</p><h4>Channel 1. In-Product Feedback</h4><p>This is the most immediate and accessible form of feedback. It captures signals directly from the people using your AI system, at the moment they interact with it.</p><p>The most familiar implementation is the thumbs-up, thumbs-down pattern now standard across AI-powered products. ChatGPT, Microsoft Copilot, and dozens of enterprise tools use binary feedback icons placed adjacent to AI-generated responses. When a user clicks thumbs-down, many systems surface a brief follow-up asking why the response was unsatisfactory. This two-step pattern, binary signal followed by optional detail, balances low friction with diagnostic value.</p><p>But in-product feedback extends well beyond thumbs-up and thumbs-down. It includes explicit error flags where users report that an AI response was incorrect, inappropriate, or harmful. It includes edit signals, where users modify AI-generated text, code, or recommendations before accepting them, and the delta between the original output and the user&#8217;s revision becomes a training signal. It includes abandonment patterns, in which users start interacting with an AI feature and then stop, suggesting the system failed to meet their needs even though they did not explicitly report a problem.</p><p>The strength of in-product feedback lies in its volume and immediacy. You capture signals from every user interaction, in real time, in the actual production environment. The weakness is noise. Binary ratings are crude. Users&#8217; thumbs-down responses for reasons that have nothing to do with model quality. They may dislike the formatting, misunderstand the intent, or simply be having a bad day. Microsoft&#8217;s own research team has noted that binary feedback mechanisms &#8220;do not result in granular feedback that can help truly improve system and model performance.&#8221; A thumbs-down tells you something went wrong. It rarely tells you what.</p><p>The most mature organizations layer additional structure onto binary signals. When a user flags an AI response negatively, they are presented with a short set of categorized options. Was the response inaccurate? Was it incomplete? Was it harmful or offensive? Was it irrelevant? These categories map directly to the taxonomy your AI team uses to classify and prioritize issues, which dramatically reduces the time between signal and action.</p><p>For AI systems embedded in business-critical workflows, consider going further. Allow users to annotate specific problematic portions of AI output, rather than rating the entire response. Build escalation paths so that a flagged response can be routed to a subject matter expert for review. And critically, close the loop by showing users that their feedback matters. When beta testers and early users see changes tied to their input, participation rates increase significantly.</p><h4>Channel 2. Annotation Programs</h4><p>Where in-product feedback captures the voice of end users, annotation programs capture the judgment of trained reviewers. These are structured programs in which human annotators systematically evaluate AI outputs against defined criteria. They represent the quality control layer of your feedback infrastructure.</p><p>Annotation programs serve several purposes that in-product feedback cannot. They provide consistent, calibrated assessments against documented standards. They evaluate dimensions that end users may not notice or report, such as subtle bias, factual accuracy in technical domains, or regulatory compliance. And they generate labeled datasets essential for model fine-tuning and retraining.</p><p>The annotation ecosystem has matured considerably. Platforms like Scale AI, Labelbox, SuperAnnotate and Encord now offer enterprise-grade infrastructure that combines data labeling with quality assurance, workflow management, and governance controls. Nearly 90% of businesses building AI rely on some form of external data labeling support, reflecting the scale and specialization required to maintain annotation quality.</p><p>A well-designed annotation program includes several components. First, clear annotation guidelines that define what &#8220;good&#8221; and &#8220;bad&#8221; look like for each dimension being evaluated. These guidelines should be specific enough to produce consistent results across annotators while flexible enough to accommodate edge cases. Second, a quality assurance process that measures inter-annotator agreement (the degree to which different annotators arrive at the same judgment for the same output) and identifies annotators whose ratings diverge significantly from the consensus. Target inter-annotator agreement scores of 0.7 or higher on Cohen&#8217;s kappa to ensure reliability. Third, a stratified sampling strategy that ensures your annotation program covers the full distribution of AI outputs, not just the easy cases or the obvious failures.</p><p>The distinction between annotation programs and in-product feedback is important for governance. In-product feedback tells you how users experience your AI. Annotation programs tell you how your AI performs against objective standards. You need both perspectives, and they should inform each other. When in-product feedback surfaces a pattern of user dissatisfaction in a particular domain, your annotation program should investigate that domain more deeply. When annotators identify a failure mode, in-product monitoring should track whether it appears in live traffic.</p><h4>Channel 3. RLHF-Style Pipelines</h4><p>Reinforcement Learning from Human Feedback, or RLHF, has become the default alignment strategy for large language models. By 2025, an estimated 70% of enterprise LLM deployments used some variant of RLHF or its successors, including Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO), for post-training alignment. Every frontier model relies on human preference training in some form.</p><p>The core idea is straightforward, even if the implementation is complex. Humans compare pairs of AI outputs and indicate which one is better. These preference judgments are used to train a reward model that approximates human judgment. The reward model then guides the optimization of the AI system, steering it toward outputs that humans prefer.</p><p>For most enterprises, building a full RLHF pipeline from scratch is neither practical nor necessary. But the principles underlying RLHF are directly applicable to enterprise AI operations, even for organizations that fine-tune open-source models rather than train from scratch.</p><p>The relevant applications include preference-based evaluation of AI outputs, where subject matter experts compare multiple candidate responses and rank them. They include reward model development, in which you train a model to predict the outputs your domain experts would prefer, enabling automated quality assessment at scale. And they include systematic prompt optimization, where feedback from RLHF-style evaluations informs how you structure prompts, system instructions, and retrieval-augmented generation (RAG) configurations.</p><p>The EU AI Act&#8217;s transparency requirements, specifically Article 52, now mandate documentation of how human feedback is collected, how annotators are instructed, and what quality controls are applied. Organizations that use any form of human preference training must maintain audit trails for their preference datasets and reward model evaluations. The Act&#8217;s bias provisions, under Article 10, require that preference data be examined for demographic and cultural biases, a known challenge when annotator pools skew toward specific demographics or geographies.</p><p>For enterprises operating RLHF-style pipelines, this means governance cannot be an afterthought. Who selects the annotators? What guidelines govern their judgments? How are disagreements resolved? How is the preference dataset audited for bias? These questions must be answered before feedback enters the pipeline, not after the reward model has been trained on potentially compromised data.</p><h2>How Feedback Becomes Action</h2><p>Collecting feedback is necessary but not sufficient. The more common failure is not a lack of feedback signals but a lack of mechanisms to convert those signals into changes to products and models. Raw feedback that sits in a database, unprocessed and unowned, delivers zero value. The path from raw feedback to meaningful improvement requires four stages.</p><h4>Stage 1. Categorization and Enrichment</h4><p>Raw feedback arrives in many forms. A thumbs-down. A free-text comment. An annotator&#8217;s rating. A preference judgment. Before it can be acted upon, it must be categorized into a taxonomy that your engineering, product, and ML teams can work with.</p><p>A practical taxonomy for enterprise AI feedback might include accuracy issues (the AI output was factually wrong), relevance issues (the AI output was correct but did not address the user&#8217;s intent), safety issues (the AI output was harmful, offensive, or violated policy), completeness issues (the AI output was partially correct but missing critical information), format issues (the AI output was substantively fine but poorly structured or presented), and latency or reliability issues (the AI system was slow, unresponsive, or inconsistent).</p><p>Enrichment adds context to each feedback item. What was the user&#8217;s query? What did the AI produce? What model version was serving at the time? What data sources were consulted? This contextual metadata transforms an isolated complaint into a diagnostic artifact that engineers can actually investigate. Automated enrichment, pulling in session logs, model metadata, and retrieval context at the time of flagging, dramatically reduces the time required to triage each item.</p><h4>Stage 2. Prioritization and Severity Assessment</h4><p>Not all feedback items warrant the same response. A safety issue that surfaces harmful content to customers demands immediate attention. A formatting preference from a single user can wait. Prioritization frameworks should consider the severity of the issue (safety and accuracy issues outweigh style preferences), the frequency of occurrence (a problem affecting 5% of queries is more urgent than one affecting 0.01%), the population affected (issues impacting vulnerable populations or high-value customers may warrant elevated priority), and the regulatory implications (issues that could constitute non-compliance with the EU AI Act or other regulations require expedited review).</p><p>Many organizations use a tiered severity model. Critical issues, those involving safety, bias, or regulatory non-compliance, trigger immediate escalation and may warrant pulling an AI feature from production. High-severity issues, such as systematic accuracy problems in a specific domain, are entered into the current sprint. Medium and low-severity issues flow into the standard backlog and are prioritized alongside other product work.</p><h4>Stage 3. Ownership and Routing</h4><p>This is where most feedback programs break down. Feedback gets collected, even categorized, but then lands in a shared queue that nobody owns. The AI team assumes product management will triage it. Product management assumes the AI team is handling it. The feedback ages, context decays, and the same issues keep recurring.</p><p>Effective feedback operations assign clear ownership at the point of categorization. Safety issues route to a dedicated trust-and-safety function. Model performance issues route to the ML engineering team. Prompt and configuration issues route to the AI product owner. UX and formatting issues route to the product design team. Ambiguous items route to a triage function that includes cross-functional representation.</p><p>Routing rules should be codified, not informal. When a feedback item is categorized as a safety issue, it should automatically appear in the trust and safety team&#8217;s queue with the appropriate severity flag. When it is categorized as a model performance issue, it should be added to the ML team&#8217;s backlog along with the enrichment data needed to investigate it. The goal is zero manual routing for the majority of feedback items.</p><h4>Stage 4. Conversion to Backlog Items</h4><p>The final stage converts triaged feedback into actionable work. This means translating a cluster of related feedback signals into a specific backlog item with a clear definition of done.</p><p>A feedback-driven backlog item might read as follows. &#8220;Users report that the AI assistant provides outdated pricing information for enterprise products. Analysis of 47 flagged interactions over two weeks shows the RAG pipeline is retrieving pricing documents from Q3 2025. Update the retrieval index to include Q1 2026 pricing sheets and add a freshness filter that deprioritizes documents older than 90 days.&#8221;</p><p>Notice the difference between this and a vague &#8220;improve pricing accuracy&#8221; ticket. The feedback has been aggregated, the root cause has been investigated, the fix is specific, and the definition of done is testable. This is what it means to operationalize feedback.</p><p>For model-level issues that require retraining or fine-tuning, the conversion process is more involved. Feedback signals must be aggregated into training datasets, validated for quality and bias, and prioritized against other training data needs. The decision to retrain a model, even partially, should involve both the ML team and the governance function, since retraining introduces new risks that must be assessed.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/the-feedback-gap?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/the-feedback-gap?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>What Good Feedback Operations Look Like</h2><p>Organizations that excel at feedback operations share several characteristics that distinguish them from those that merely collect feedback without acting on it.</p><h4>Governance integration. </h4><p>Feedback operations are connected to the AI governance framework, not floating as a standalone process. The AI Risk Committee (or equivalent) reviews feedback trends, escalation patterns, and resolution metrics on a regular cadence. Feedback data informs the risk assessments conducted for each AI system. And feedback-driven changes, particularly those involving model retraining or policy updates, flow through the same approval processes as other AI changes.</p><h4>Consistency across channels. </h4><p>Feedback from in-product flags, annotation programs, and RLHF-style evaluations flows into a unified system with a common taxonomy. An accuracy issue identified by an end user and by an annotator should be categorized and routed identically. This prevents duplication and ensures that feedback from different channels reinforces rather than contradicts each other.</p><h4>Escalation rules that are documented and enforced. </h4><p>When a safety issue is flagged, the escalation path should be unambiguous. Who gets notified? Within what timeframe? What authority do they have to pull an AI feature from production? These rules should be written down, socialized across teams, and tested periodically through tabletop exercises, similar to how organizations test incident response procedures for cybersecurity.</p><h4>Closed-loop reporting. </h4><p>The teams that generate feedback, whether end users, annotators, or domain experts, should see evidence that their input leads to change. For internal annotators, this might mean a monthly summary of issues identified, actions taken, and improvements measured. For end users, it might mean release notes that reference user feedback as the catalyst for specific improvements. Closed-loop reporting sustains participation. Without it, feedback volumes decline over time as contributors conclude their input is being ignored.</p><h4>Metrics that matter. </h4><p>Good feedback operations track several metrics. These include time from feedback submission to triage, time from triage to resolution, the percentage of feedback items that result in product or model changes, feedback volume trends by category (which can signal emerging issues or improving quality), and the correlation between feedback-driven changes and improvements in downstream performance metrics.</p><h2>Common Pitfalls</h2><p>Even well-intentioned feedback programs can fall short. These are the failure modes I see most frequently across enterprise AI programs.</p><h4>Noisy signals that overwhelm the system. </h4><p>When every thumbs-down and every user comment flows into the same queue without filtering or categorization, the signal-to-noise ratio drops to a level where nothing is actionable. Teams drown in volume and default to ignoring the queue entirely. The solution is not to collect less feedback but to invest in automated categorization and severity assessment that surfaces the most important items first.</p><h4>Biased labels that corrupt the learning process. </h4><p>Annotator pools that lack diversity introduce systematic bias into preference data and evaluation datasets. If your annotators are predominantly from one demographic, geography, or professional background, their judgments will reflect those perspectives. The EU AI Act explicitly requires that preference data be examined for such biases. Organizations should audit annotator demographics, measure systematic rating differences across demographic groups, and proactively diversify their annotator pools.</p><h4>Unclear thresholds that prevent action. </h4><p>When is a model performance issue severe enough to warrant retraining? How many user complaints about a specific failure mode constitute a pattern? Without defined thresholds, teams default to either overreacting to isolated incidents or underreacting to systemic problems. Establish quantitative thresholds for each severity level and review them quarterly as your AI systems and user base evolve.</p><h4>Feedback that never reaches the teams that can act on it. </h4><p>This is perhaps the most common and most damaging failure mode. End users flag issues through customer support channels, but those signals never reach the ML engineering team. Annotators identify bias patterns, but the findings sit in a report that product management never reads. The root cause is almost always organizational: feedback operations span multiple teams, and without explicit routing rules and ownership, signals fall into the gaps between organizational boundaries.</p><h4>Overweighting vocal minorities. </h4><p>A small number of highly active users may generate a disproportionate share of in-product feedback. If your prioritization framework heavily weights volume, these vocal users can skew your improvement priorities away from what matters most to the broader user base. Balance feedback volume with statistical sampling of the broader user population to ensure your priorities reflect actual impact.</p><h4>Treating feedback as a one-time project rather than a continuous capability. </h4><p>Some organizations launch feedback programs with great energy, build the infrastructure, train the annotators, and establish the workflows, only to let the program atrophy once the initial enthusiasm fades. Feedback operations, like the monitoring and governance capabilities they support, require sustained investment. Staff must be dedicated. Processes must be maintained. And leadership must reinforce the importance of feedback as a permanent part of the AI operating model, not a temporary initiative.</p><h2>A Practical Operating Model for Feedback Operations</h2><p>For organizations ready to build or formalize their feedback operations, the following workflow provides a starting point. It is designed to be simple enough to implement immediately and extensible as your AI program matures.</p><h4>Step 1. Instrument your AI systems to capture feedback. </h4><p>For user-facing AI, implement in-product feedback mechanisms (binary ratings plus optional categorized follow-up) adjacent to every AI-generated output. For internal AI systems, establish annotation review cycles where trained reviewers evaluate a statistically representative sample of outputs on a weekly or biweekly cadence. For organizations fine-tuning or training models, establish a preference-evaluation program in which domain experts compare candidate outputs.</p><h4>Step 2. Establish a unified feedback taxonomy. </h4><p>Define the categories into which all feedback, regardless of source, will be classified. Align this taxonomy with your AI risk framework so that feedback categories map directly to the risk dimensions your governance function monitors. Publish the taxonomy, train all teams that interact with feedback on how to use it, and assign an owner responsible for maintaining and evolving it over time.</p><h4>Step 3. Build automated categorization and routing. </h4><p>Use rule-based systems or lightweight classifiers to auto-categorize incoming feedback and route it to the appropriate team. Safety issues go to trust and safety. Model performance issues go to ML engineering. Configuration and prompt issues go to the AI product owner. This automation should handle 80% or more of incoming feedback without manual intervention, freeing your triage function to focus on ambiguous or complex cases.</p><h4>Step 4. Define severity levels and escalation paths. </h4><p>Establish four or five severity levels with clear criteria and associated response timeframes. Critical issues (safety, bias, regulatory non-compliance) should trigger immediate notification and be covered by a defined on-call rotation. Document the escalation path for each severity level, including who has authority to pull an AI feature from production if necessary.</p><h4>Step 5. Convert feedback into backlog items with a regular cadence. </h4><p>Every week, the AI product owner (or a designated feedback operations lead) should review categorized feedback, aggregate related items, investigate root causes for recurring patterns, and create specific, actionable backlog items with clear definitions of done. These items should enter the same prioritization process as other product and engineering work, weighted by severity, frequency, and impact.</p><h4>Step 6. Close the loop. </h4><p>Report back to feedback contributors on what changed as a result of their input. Publish feedback metrics to your AI governance function. And review the effectiveness of the feedback operation itself quarterly, asking whether you are capturing the right signals, routing them effectively, and converting them into improvements at an acceptable pace.</p><h2>Feedback as a Strategic Capability</h2><p>The organizations capturing AI&#8217;s full potential are those that treat feedback not as a burden but as a competitive advantage. Every flagged response, every annotator rating, and every preference judgment is a data point that makes your AI systems more reliable, more aligned with user expectations, and more defensible under regulatory scrutiny.</p><p>The infrastructure required is not exotic. It is the disciplined application of established practices: instrumentation, categorization, routing, prioritization, and closed-loop improvement. What makes it hard is not the technology but the organizational commitment to sustain it. Feedback operations span product, engineering, data science, risk, and compliance. They require cross-functional coordination, clear ownership and executive sponsorship that treats feedback as an investment in AI quality rather than an overhead cost.</p><p>Don&#8217;t forget this. A feedback loop without action is just monitoring. A feedback loop with action is a learning system. The organizations that build learning systems will outperform those that build static ones, and the gap will compound over time.</p><p>Start by auditing what you have today. Can your users flag AI outputs? If so, where does that feedback go? Who owns it? How long does it take to become an improvement? If you cannot answer those questions clearly, that is your starting point.</p><p>The question, as always, is not whether your organization needs feedback operations for its AI systems. The question is whether you will build them deliberately or discover their absence the hard way.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/the-feedback-gap/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/the-feedback-gap/comments"><span>Leave a comment</span></a></p>]]></content:encoded></item><item><title><![CDATA[Evaluating Generative AI: Practical Tests for Safety, Bias and Fairness]]></title><description><![CDATA[How to assess toxicity, hallucinations, representational harms and group fairness]]></description><link>https://trustedai.recodework.com/p/evaluating-generative-ai-practical</link><guid isPermaLink="false">https://trustedai.recodework.com/p/evaluating-generative-ai-practical</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Sat, 16 May 2026 17:19:25 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!hDsY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!hDsY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hDsY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hDsY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hDsY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hDsY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!hDsY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:6251194,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/197896691?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!hDsY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hDsY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hDsY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hDsY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30e73fb7-f32b-4874-9c07-89103d65248f_3864x2576.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>If your AI evaluation strategy still centers on accuracy scores and latency benchmarks, you are measuring the easy while ignoring the consequential.</p><p>Accuracy tells you whether a generative model can produce a correct answer. It tells you nothing about whether that model will fabricate a citation to a nonexistent court case, generate a recruitment summary that systematically favors one demographic over another, or respond to a vulnerable user with language that causes real harm. These are not hypothetical failure modes. They are documented incidents, and their frequency and severity are increasing.</p><p>Stanford HAI&#8217;s 2026 AI Index Report found that documented AI incidents rose to 362 in 2025, up 55% from 233 in 2024. Among organizations that experienced incidents, those reporting three to five incidents jumped from 30% to 50%. At the same time, organizations rating their incident response capability as &#8220;excellent&#8221; dropped from 28% to 18%. The gap between what we are deploying and what we can responsibly manage is widening, not closing.</p><p>The root cause is an evaluation gap. Almost all leading frontier model developers report results on capability benchmarks like MMLU and SWE-bench. But reporting on responsible AI benchmarks, those covering safety, bias, fairness and harmful outputs, remains sparse and inconsistent. We measure what models can do with great precision. We measure what models should not do with almost none.</p><p>Beyond the fundamentals of AI metrics and testing methodology, the socio-technical dimensions determine whether a generative AI system is safe to deploy and equitable in its impact. This article will define the four primary failure modes, walk through practical tests you can run for each, explain how to interpret results without overreacting or underreacting, and lay out a prioritization framework for deciding what to fix and when.</p><p>This is not academic theory. It is an operational playbook for the evaluation work that separates responsible AI programs from aspirational ones.</p><h2>What Can Go Wrong in Generative Models</h2><p>Traditional software fails in predictable ways. It throws an error, returns null, or crashes. Generative AI fails in ways that are far more insidious because the outputs often look plausible, fluent and authoritative even when they are wrong, harmful or biased. Understanding the taxonomy of failure modes is the first step toward evaluating them.</p><h4>Toxic and harmful output</h4><p>Generative models can produce content that is abusive, threatening, sexually explicit or otherwise harmful. This includes direct toxicity, where the model generates overtly offensive language, and indirect toxicity, where the model follows harmful instructions embedded in adversarial prompts or produces content that normalizes violence or self-harm. A customer-facing chatbot that generates a single toxic response over 10,000 interactions may seem statistically insignificant. Still, for the customer on the receiving end, it is the only interaction that matters. And when that interaction surfaces on social media, the statistical argument provides no protection.</p><h4>Hallucinations and confabulation</h4><p>Generative models produce content that is factually incorrect, fabricated or internally inconsistent while presenting it with the same confidence as accurate information. NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework, published in July 2024, identifies confabulation as one of twelve distinct risk categories for generative AI systems. The scale of the problem is significant. On Stanford&#8217;s AA-Omniscient Index, a benchmark designed to assess whether models will admit uncertainty rather than guess, hallucination rates across 26 top models ranged from 22% to 94%. In legal queries, Stanford researchers found hallucination rates between 69% and 88%. In medical case summaries, hallucinations reached 64% without mitigation prompts. These are not edge cases. They are systemic characteristics of how these models operate.</p><h4>Representational harms</h4><p>These occur when generative models produce outputs that reinforce stereotypes, erase or underrepresent certain groups, or associate demographic characteristics with negative attributes. A 2023 analysis of over 5,000 images created with generative AI tools found that they amplified both gender and racial stereotypes relative to real-world distributions. When a model consistently depicts engineers as men, nurses as women, or associates certain ethnicities with poverty, it is not merely reflecting existing data. It is reinforcing and amplifying patterns of exclusion. Representational harms are particularly dangerous because they are often subtle enough to pass casual review but pervasive enough to shape perceptions at scale.</p><h4>Group fairness failures</h4><p>Even when individual outputs appear acceptable, a model may systematically perform differently across demographic groups. This might manifest as higher error rates for certain languages or dialects, less helpful responses for users from particular backgrounds, or recommendation patterns that advantage some groups over others. Recent research has shown that even when models provide equal access (zero differences in refusal rates across demographic groups), they can still exhibit systematic disparities in interaction quality, including differences in sentiment, hedging, and the degree of confidence expressed in responses to users of different ages, genders, or nationalities.</p><p>These four failure modes are not mutually exclusive. A model that hallucinates disproportionately in responses to users from certain backgrounds combines confabulation with group fairness failure. A model that produces stereotyped content in response to adversarial prompts exhibits both representational harm and a safety vulnerability. Effective evaluation must address all four dimensions and their intersections.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>Practical Tests You Can Run</h2><p>Evaluation frameworks for generative AI have matured significantly over the past two years. What follows is not a comprehensive literature review but a practical guide to the tests that deliver the most actionable insight per unit of effort. Each section focuses on a specific failure mode and outlines approaches that range from straightforward (suitable for teams just beginning systematic evaluation) to sophisticated (appropriate for organizations with dedicated responsible AI programs).</p><h4>Toxicity and Safety Testing</h4><p>The foundation of safety evaluation is structured red-teaming, the practice of systematically probing a model with inputs designed to elicit harmful outputs. Stanford&#8217;s HELM Safety benchmark provides a standardized framework that includes HarmBench for jailbreak resistance, BBQ for social discrimination, SimpleSafetyTest for basic safety behaviors, XSTest for the balance between helpfulness and harmlessness, and the AnthropicRedTeam evaluation for resilience against adversarial probing.</p><p>For organizations building their own evaluation programs, start by assembling prompt sets that cover the categories of harm most relevant to your deployment context. A healthcare application faces different toxicity risks than an internal productivity tool. Your prompt library should include direct requests for harmful content to test baseline refusals, adversarial prompts that attempt to circumvent safety controls through role-playing or hypothetical framing, multi-turn conversations that gradually escalate toward harmful territory, and prompts that embed harmful instructions within a seemingly benign context.</p><p>Critically, do not limit testing to English. The Stanford 2026 AI Index found that HELM Arabic, a regionally developed model, outscored leading frontier models, and performance gaps widened significantly at the dialect level. If your users speak multiple languages, your safety testing must cover all of them. Models that perform well on English safety benchmarks may exhibit significantly degraded safety behavior in other languages.</p><p>Automated toxicity classifiers, such as Perspective API or specialized toxicity detection models, can score outputs at scale. But automated classifiers have known limitations, including higher false positive rates for text discussing marginalized groups and difficulty detecting subtle toxicity. Use automated scoring to identify patterns and flag concerning outputs, but always supplement with human review for final assessment.</p><h4>Groundedness and Hallucination Testing</h4><p>Testing for hallucination requires a different approach than testing for toxicity because the failure mode is not about harmful intent but about factual accuracy and calibration.</p><p>The most direct test is a grounded question-answering evaluation. Provide the model with a source document and ask questions whose answers are contained in that document. Measure how often the model introduces information not present in the source, contradicts statements in the source, or fabricates citations, statistics, or quotations. This approach tests factual fidelity in a controlled setting where the ground truth is known.</p><p>For open-ended generation, the evaluation is harder but not impossible. Select a representative sample of outputs and have subject matter experts verify factual claims. Track the rate of verifiable errors, the types of errors (minor factual inaccuracies versus completely fabricated information), and the model&#8217;s confidence calibration (whether appropriate uncertainty is expressed). Pay particular attention to how the model handles questions at the boundary of its knowledge. A model that confidently answers questions it should not be able to answer is more dangerous than one that occasionally gets facts wrong but signals uncertainty.</p><p>The AA-Omniscient benchmark approach is instructive here. It does not just ask whether models get facts right. It asks whether models will admit when they do not know something. That distinction matters enormously in enterprise deployments, where users may not have the expertise to verify model outputs independently.</p><p>Consider building domain-specific hallucination test sets that reflect your actual use cases. If your model generates financial summaries, create a set of ground-truth financial documents and measure confabulation rates against them. If it supports customer service interactions, compile verified product information, and test whether the model invents features, pricing, or policies that do not exist. Generic benchmarks establish a baseline. Domain-specific tests tell you whether the model is safe for your particular deployment.</p><h4>Bias and Representational Harm Testing</h4><p>Counterfactual testing is the workhorse methodology for detecting bias in generative models. The concept is straightforward. Take a prompt, change a single demographic attribute (a name, a pronoun, a nationality, a reference to religion or ethnicity), and compare the outputs. If the model produces meaningfully different responses when the only variation is a demographic characteristic, you have identified a potential bias.</p><p>For example, create pairs of prompts that describe identical professional qualifications but use names commonly associated with different ethnic backgrounds. Ask the model to evaluate the candidates, write recommendation letters, or suggest career paths. Measure whether the quality, enthusiasm, or substance of the responses differs systematically across demographic groups. Research published at ICLR 2025 formalized this approach for large-scale testing, demonstrating methods to quantify the probability of unbiased responses across counterfactual prompt sets.</p><p>The SCOPE dataset, developed for systematic fairness evaluation, illustrates the scale at which this testing can be conducted. It contains over 241,000 prompts organized into more than 120,000 counterfactual pairs spanning nine bias dimensions and over 1,500 demographic groups. While you do not need to replicate this scale, the framework is useful. Your counterfactual tests should vary along the dimensions most relevant to your use case and cover the demographic groups most likely to be affected by your deployment.</p><p>Beyond counterfactual testing, examine your model&#8217;s default representations. When given open-ended prompts about professions, activities or social roles, what demographic patterns does the model default to? If asked to &#8220;write a story about a nurse,&#8221; does the model default to female characters? If asked to &#8220;describe a successful entrepreneur,&#8221; does it default to particular age, gender or ethnic characteristics? These defaults reveal learned associations that may not surface in direct questioning but shape the model&#8217;s output in pervasive ways.</p><p>Tools like LangFair and BEATS (Bias Evaluation and Assessment Test Suite) can automate portions of this work. LangFair handles use-case-specific assessments, including fairness-through-unawareness checks, counterfactual generation, and toxicity calculations directly from model outputs. BEATS offers 29 metrics spanning demographic, cognitive, and social biases. However, 2025 data revealed that nearly 38% of outputs from top industry models still exhibited some form of bias, underscoring that tooling alone does not eliminate the problem. It surfaces it for human judgment.</p><h4>Slice-Based Evaluation for Group Fairness</h4><p>Whereas bias testing assesses whether a model treats different groups differently on the same task, group fairness evaluation assesses whether a model performs equitably across the population of users it actually serves. This requires what practitioners call slice-based evaluation, which means breaking down aggregate performance metrics by meaningful subgroups rather than relying on overall averages.</p><p>Aggregate performance metrics can mask significant disparities. A model might achieve 95% accuracy overall while performing at 85% for users in certain geographic regions, age groups, or language communities. If you only report the aggregate number, you never see the gap.</p><p>Start by identifying the dimensions along which performance might vary in your specific context. Language and dialect are almost always relevant. Geography, age, technical sophistication, and accessibility needs (such as reliance on screen readers or simplified interfaces) are frequently important. For each dimension, measure core performance metrics, including accuracy, helpfulness ratings, error rates, and response quality, and compare across groups.</p><p>Recent empirical research adds an important nuance. Access-level fairness (does the model respond at all?) is necessary but insufficient. Researchers have demonstrated that models can provide zero-refusal-rate equity, responding to all demographic groups equally, while still exhibiting systematic disparities in interaction quality. One study found that GPT-4 expressed significantly higher hedging toward younger male users, while another model showed broader sentiment variation across identity groups. The model responds to everyone, but not everyone gets the same quality of response.</p><p>This means your group fairness evaluation should go beyond binary measures (did the model respond? was the response toxic?) and assess qualitative dimensions such as response helpfulness, specificity, confidence and tone.</p><h4>Human Review for Edge Cases and Ambiguity</h4><p>Automated evaluation handles scale. Human review handles judgment. Both are necessary, and neither is sufficient on its own.</p><p>Establish a structured human review process for outputs that automated systems flag as borderline, for categories where automated classifiers have known limitations, and for novel failure modes that your automated pipeline was not designed to detect. Human reviewers should be diverse in their backgrounds and perspectives, trained on the specific harm categories relevant to your deployment, and empowered to escalate issues that do not fit into existing categories.</p><p>Pay particular attention to intersectional cases, outputs where the harm emerges from the combination of multiple factors rather than any single dimension. A response might be neither toxic nor biased in isolation but become problematic in the context of a vulnerable user, a high-stakes decision, or a specific cultural setting. These cases resist automated classification and require human judgment informed by a genuine understanding of the affected communities.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/evaluating-generative-ai-practical?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/evaluating-generative-ai-practical?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>How to Interpret Results</h2><p>Raw evaluation results are necessary but not sufficient for decision-making. The critical skill is interpretation, which means understanding what scores mean, where they mislead, and how they should inform action.</p><h4>One bad score does not tell the whole story</h4><p>A model that scores poorly on a generic safety benchmark may perform well in your specific deployment context because your guardrails, system prompts and use case constraints mitigate the weaknesses the benchmark measures. Conversely, a model that scores well on benchmarks may fail in production because your use case exposes vulnerabilities that the benchmarks do not cover. Benchmarks are starting points for investigation, not final verdicts.</p><h4>Distinguish severity, frequency, and user impact</h4><p>A hallucination rate of 15% sounds alarming as a headline number. But the operational significance depends entirely on what the model is doing and who is affected. A 15% hallucination rate in a creative brainstorming tool is a different risk than the same rate in a medical information system. When interpreting results, always ask three questions. How severe is the potential harm from this failure? How frequently does it occur in realistic usage patterns? And how many users are exposed to the risk?</p><h4>Some harms are rare but catastrophic</h4><p>Others are common but subtle. Toxic outputs may occur in only 0.1% of interactions but generate outsized reputational and legal consequences. Mild representational biases may appear in 30% of outputs, individually causing minimal harm but cumulatively shaping perceptions over time. Your evaluation framework must capture both types. The rare-but-severe failures justify intensive safety testing and robust guardrails. The common-but-subtle failures justify ongoing monitoring and systematic measurement.</p><h4>Context changes everything</h4><p>Stanford&#8217;s 2026 AI Index revealed a striking finding about context sensitivity. When a false statement is presented as something another person believes, models handle it well. When the same false statement is presented as something the user believes, performance collapses. The model&#8217;s behavior depends not just on what it is asked but on the framing, the conversational context, and the relationship dynamics implied by the prompt. Your evaluation should test across the contexts your users actually encounter, not just clean benchmark conditions.</p><h4>Watch for trade-offs between dimensions</h4><p>Recent research cited in the Stanford report found that improving one dimension of responsible AI can degrade another. Improving safety sometimes reduces accuracy. Improving privacy can reduce fairness. These trade-offs are real and not always intuitive. When you observe an improvement in one evaluation dimension, check whether it comes at a cost elsewhere. Optimizing for a single metric while ignoring others is how organizations create new problems while solving old ones.</p><h2>How to Prioritize Fixes</h2><p>Not every finding from an evaluation requires the same response, and not every response needs to happen before launch. The goal is a structured prioritization framework that directs resources where they will do the most good while maintaining an honest assessment of residual risk.</p><h4>Rank by severity, likelihood, and exposure</h4><p>Severity measures the potential harm of a failure. Likelihood measures how frequently the failure occurs under realistic conditions. Exposure measures the number of users or stakeholders who could be affected. A failure that is severe, likely and widely exposed demands immediate attention. A failure that is severe, but extremely rare, and narrowly exposed may warrant monitoring rather than urgent remediation.</p><h4>Separate &#8220;must fix before launch&#8221; from &#8220;monitor after launch&#8221;</h4><p>Some findings are deployment blockers. If your model produces toxic outputs at a meaningful rate in response to common user queries, this must be resolved before production. If your model exhibits mild representational biases in a narrow category of edge-case prompts, this can be documented, monitored, and addressed through iterative improvement after deployment. The distinction is not a license to ignore issues but a recognition that demanding perfection before deployment means nothing ships, and the value of the AI system goes unrealized while incremental improvement is indefinitely deferred.</p><h4>Match the intervention to the problem</h4><p>Different failure modes require different fixes. Toxicity and safety failures often respond well to guardrails, which are output filters, content classifiers and system-prompt constraints that operate as a layer on top of the base model. Hallucination problems may require retrieval-augmented generation, grounding mechanisms, or citation requirements that change the architecture rather than just the output filtering. Bias and representational harms may require fine-tuning on more balanced data, adjusting training procedures, or redesigning prompt templates. Group fairness issues may demand changes to the product design itself, such as offering alternative interaction modes for underserved populations or adjusting how the model handles diverse languages and dialects.</p><p>Consider this hierarchy of interventions, ordered by increasing effort and impact. Product-level changes, such as restricting the model&#8217;s scope or adding human review to high-stakes outputs, are fast to implement and often highly effective. Guardrails and output filtering provide broad protection but can reduce helpfulness if too aggressive. Prompt engineering and system-level constraints can reshape model behavior without retraining. Fine-tuning adjusts model behavior at a deeper level but requires additional data and compute. Full retraining or model replacement is the most resource-intensive option and is typically reserved for fundamental inadequacies that cannot be addressed through other means.</p><h4>Document residual risk honestly</h4><p>After remediation, some risks will remain. Document them. Specify what was tested, what was found, what was fixed, what remains, and how the remaining risks will be monitored. This documentation serves multiple purposes. It provides transparency for governance and compliance review. It establishes the baseline against which future improvements will be measured. And it creates institutional memory that prevents the next team from repeating the same evaluation work without building on what was already learned.</p><h2>What Metrics Miss</h2><p>Even a rigorous evaluation program based on the methods described above has fundamental limitations. Acknowledging these limitations is not a reason to abandon measurement. It is a reason to complement measurement with broader forms of evaluation.</p><h4>Metrics capture what can be measured, not necessarily what matters most</h4><p>Toxicity classifiers can detect overtly harmful language. They struggle with subtle dehumanization, patronizing tone, or cultural insensitivity that does not register as &#8220;toxic&#8221; on a classifier but causes genuine harm to affected users. Bias metrics can detect differential treatment across demographic groups. They cannot capture whether the treatment any group receives is good enough in absolute terms. A model that is equally unhelpful to everyone is &#8220;fair&#8221; by many metrics but valuable to no one.</p><h4>Evaluation reflects the evaluator&#8217;s perspective</h4><p>The harm categories you test for, the demographic groups you include, and the severity thresholds you set all reflect choices made by the evaluation team. If your team lacks diversity of background and perspective, your evaluation will have corresponding blind spots. This is not a criticism of the people involved. It is a structural limitation that can only be addressed by deliberately broadening the perspectives that inform evaluation design.</p><h4>Static evaluation misses dynamic harm</h4><p>Benchmarks capture model behavior at a point in time. Real-world deployment is dynamic. User populations shift. Adversarial techniques evolve. Cultural norms change. An evaluation that declares a model safe in January may not account for a new adversarial attack technique that emerges in March or a shift in user demographics that exposes previously untested failure modes. This is why NIST AI 600-1 and the broader AI RMF emphasize continuous monitoring as a complement to pre-deployment testing. Evaluation is not a one-time gate. It is an ongoing practice.</p><h4>Context, workflow, and user population change the meaning of results</h4><p>A 5% hallucination rate means something very different when the model is summarizing meeting notes for internal use than when it is generating patient discharge instructions. The same bias pattern carries different weight when the model serves as an optional brainstorming tool versus a primary decision-support system. Evaluation results must always be interpreted in the context of how the model is actually used, by whom, and with what stakes.</p><h2>The Bridge to Deeper Evaluation</h2><p>The practical tests described in this article will significantly improve your ability to identify and address the most common failure modes in generative AI. But there are dimensions of impact that operational evaluation alone cannot capture.</p><p>Who benefits from your AI system, and who bears the risks? Are the communities most affected by potential harms represented in your evaluation process? How does your system interact with existing power structures, and does it amplify or mitigate existing inequities? What happens downstream when the outputs of your system are used as inputs to other decisions, other systems, other people&#8217;s lives?</p><p>These are questions that a benchmark cannot answer. They require engagement with affected communities, interdisciplinary analysis spanning technical, legal, social and ethical expertise, and a willingness to examine not just model outputs but also the systems of power and incentives within which models operate. Researchers increasingly describe this as a socio-technical evaluation, recognizing that AI systems are not isolated technical artifacts but components of complex social systems whose impacts can only be understood in context.</p><p>The Stanford 2026 AI Index highlights this gap. Frontier model developers consistently report capability benchmarks. Responsible AI benchmarks are reported sporadically. And the deeper questions of social impact, distributional effects and long-term consequences are rarely addressed in standardized evaluation. Responsible AI is not keeping pace with AI capability, with safety benchmarks lagging and incidents rising sharply.</p><p>Closing that gap requires moving beyond the question of &#8220;does this model produce harmful outputs?&#8221; toward the question of &#8220;does this model produce equitable outcomes in the real-world context where it operates?&#8221; The former is a testing problem. The latter is a governance problem, a design problem, and ultimately a question about organizational values.</p><p>Methods for conducting this deeper evaluation include stakeholder engagement frameworks, participatory evaluation approaches and techniques for assessing downstream and cumulative impacts. These socio-technical assessments are where Trusted AI moves from risk mitigation to genuine responsibility.</p><h2>Where to Start</h2><p>If you are reading this and your organization does not have a structured evaluation program for generative AI, here is a pragmatic starting point.</p><p><strong>First, inventory your deployed generative AI systems and classify them by risk level. </strong></p><p>High-stakes applications affecting individuals&#8217; access to services, employment, credit, healthcare, or safety warrant the most rigorous evaluation. Internal productivity tools warrant a lighter but still structured assessment.</p><p><strong>Second, for your highest-risk systems, implement the basic versions of each test described above. </strong></p><p>Assemble a set of prompts for safety testing. Run a grounded question-answering evaluation for hallucination rates. Conduct counterfactual prompt testing for at least the most salient demographic dimensions. Break out your performance metrics by user segment rather than relying on aggregate numbers.</p><p><strong>Third, establish a triage process for findings. </strong></p><p>Use the severity-likelihood-exposure framework to separate deployment blockers from monitoring priorities.</p><p><strong>Fourth, document everything. </strong></p><p>Your evaluation methodology, your findings, your remediation decisions, and your residual risk assessment. This documentation is the foundation of governance and the basis for demonstrating due diligence to regulators, auditors, and stakeholders.</p><p><strong>Fifth, commit to iteration. </strong></p><p>Your first evaluation will be imperfect. Your prompt sets will have gaps. Your thresholds will need calibration. That is expected. The organizations that build effective evaluation programs are not the ones that get it perfect on the first pass. They are the ones who treat evaluation as a continuous practice, learning from each cycle and improving their methods over time.</p><p>The question is not whether generative AI carries risks. The evidence on that point is overwhelming. The question is whether your organization has the evaluation practices to identify those risks before they become incidents, and the governance structures to act on what you find.</p><p>The organizations that answer yes will be the ones that capture AI&#8217;s full potential while managing its real consequences. The organizations that answer &#8220;we&#8217;ll get to it&#8221; are the ones contributing to next year&#8217;s incident count.</p><p>Start now. Measure what matters. And build from there.</p><p>But recognize that model-level evaluation, however rigorous, is only half the work. The harder question is not how your model behaves in a test harness. It is how your system affects the people and communities it touches, who benefits, who bears the risk, and whether the structures around your AI are designed to hear from those who experience its consequences firsthand. </p><p>That is where evaluation becomes governance, and where governance becomes genuine accountability.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/evaluating-generative-ai-practical/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/evaluating-generative-ai-practical/comments"><span>Leave a comment</span></a></p>]]></content:encoded></item><item><title><![CDATA[Stop Guessing: How to Evaluate Modern LLM Applications ]]></title><description><![CDATA[A practical framework for measuring what matters across prompts, retrieval, and tool behavior]]></description><link>https://trustedai.recodework.com/p/stop-guessing-how-to-evaluate-modern</link><guid isPermaLink="false">https://trustedai.recodework.com/p/stop-guessing-how-to-evaluate-modern</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Mon, 11 May 2026 14:20:14 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!7-5e!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7-5e!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7-5e!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg 424w, https://substackcdn.com/image/fetch/$s_!7-5e!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg 848w, https://substackcdn.com/image/fetch/$s_!7-5e!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!7-5e!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!7-5e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg" width="1456" height="991" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:991,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2261275,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/197160288?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!7-5e!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg 424w, https://substackcdn.com/image/fetch/$s_!7-5e!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg 848w, https://substackcdn.com/image/fetch/$s_!7-5e!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!7-5e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcde155d-da16-4a9f-a20e-971b6e6add89_5846x3980.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Every team building with large language models (LLMs) hits the same inflection point. The prototype works. The demo impresses stakeholders. Early users are enthusiastic. And then the system goes to production, where it encounters real queries from real users operating in real conditions, and something goes wrong. Not catastrophically wrong, usually. Wrong in the quiet, corrosive way that erodes trust over weeks and months. Answers that are almost right but miss a critical detail. Responses that cite internal documents but pull from the wrong version. Tool calls that fire when they should not, or fail to fire when they should.</p><p>The instinct is to fix the prompt.</p><p>This approach is understandable. Prompts are the most visible, most accessible lever in any LLM application. Changing a prompt takes minutes, requires no infrastructure changes, and often produces an immediate visible improvement on the specific failure that triggered the fix. Teams iterate on wording, add instructions, restructure examples, and test against the offending query. The new version handles that case better. Problem solved.</p><p>Except that the problem remains unsolved. It has been patched, and in production systems, there is a meaningful difference between patching and solving. The organizations that are deploying LLM applications successfully at scale have learned this the hard way and have built something fundamentally different in response. They have built evaluation systems.</p><p>This issue explores what systematic evaluation looks like for modern LLM applications, why it matters for enterprise governance, and how to move beyond prompt tinkering into a disciplined practice that actually improves outcomes.</p><h2>Why Prompt Tinkering Hits a Wall</h2><p>Prompt engineering is a legitimate discipline. Good prompts matter. The structure of instructions, the quality of examples, and the specificity of formatting guidance all influence how a model behaves. But prompt engineering, as the primary mechanism for improving production LLM systems, has severe limitations that most teams discover only after investing significant effort.</p><p>The first limitation is visibility. When you change a prompt and test it against the query that failed, you learn exactly one thing: whether the new prompt handles that specific query better. This does not address whether it can handle the other 10,000 queries your system processes daily. You do not learn whether the fix introduced regressions in cases that previously worked. Prompt changes propagate through the entire system, affecting every interaction, and without systematic measurement, the net effect is unknowable.</p><p>The second limitation is misattribution. Production failures in LLM applications often appear as prompt problems when they are actually retrieval, orchestration, or data problems. A customer service system that provides incorrect policy information may appear to have a prompt issue, when the real problem is that the retrieval pipeline is returning an outdated policy document. A financial analysis tool that produces inconsistent numbers may seem to need better instructions when the actual issue is that the tool-calling mechanism is passing malformed parameters to the calculation API. Research published in 2025 found that, under realistic conditions, roughly 70% of the passages retrieved by RAG systems did not directly contain the correct answer. When your retrieval is that noisy, no prompt will reliably produce correct outputs.</p><p>The third limitation is false confidence. Prompt iteration creates a strong psychological signal of progress. Each iteration produces a visible improvement on the test case that motivated it. Teams accumulate a growing collection of &#8220;fixed&#8221; cases and naturally conclude that the system is getting better. But without systematic measurement across a representative set of queries, this confidence is unfounded. The system may be getting better at the specific cases the team has examined while getting worse in ways nobody has noticed yet.</p><p>The mature approach recognizes prompt quality as one variable in a complex system and evaluates all variables together.</p><h2>What &#8220;Evaluation&#8221; Should Mean for LLM Applications</h2><p>Evaluation is not a test you run before deployment. It is a continuous practice that spans the entire lifecycle of an LLM application. This distinction matters because LLM applications are fundamentally different from traditional software in ways that make one-time testing insufficient.</p><p>Traditional software behaves deterministically. Given the same input, it produces the same output. You can write tests that verify specific behaviors and have confidence that passing tests will continue to pass unless someone changes the code. LLM applications do not work this way. The model itself is probabilistic. The data flowing through the system is constantly changing. User behavior evolves. External APIs and data sources update. A system that performs well today may degrade tomorrow for reasons unrelated to any changes your team made.</p><p>Effective evaluation for LLM applications operates at two timescales and two levels of granularity.</p><p>The two timescales are offline evaluation and production monitoring. Offline evaluation runs controlled experiments against curated test sets before changes are deployed. Production monitoring tracks system behavior continuously against live traffic. Both are essential. Offline evaluation tells you whether a change is likely to improve things. Production monitoring tells you whether it actually did, and whether anything else has shifted in the meantime.</p><p>The two levels of granularity are component-level evaluation and end-to-end evaluation. Component-level evaluation examines individual pieces of the system in isolation. How good is the retrieval? How well does the model follow instructions? Does the tool-calling mechanism select the right tools? End-to-end evaluation examines the final output that the user actually sees. Is the answer correct? Is it complete? Is it grounded in the provided context? These are complementary lenses. Component evaluation helps you diagnose problems. End-to-end evaluation helps you measure user experience.</p><p>The organizations getting real value from LLM evaluation maintain all four of these capabilities simultaneously. They run offline tests before deploying changes, continuously monitor production behavior, examine individual components when diagnosing issues, and measure the final user-facing output to assess overall quality.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>Evaluating Prompts Systematically</h2><p>With the caveats about prompt tinkering established, prompts still deserve careful evaluation. The goal is to move from ad hoc testing against individual examples to systematic measurement against a representative set of inputs.</p><p>Effective prompt evaluation examines three dimensions. The first is instruction following. Does the model actually do what the prompt tells it to do? If the prompt says to respond in JSON format, does the output parse as valid JSON? If the prompt says to limit the response to three sentences, does the model comply? If the prompt says to decline questions outside a defined scope, does the model respect that boundary? These are verifiable, often automatable checks.</p><p>The second dimension is answer quality. Given the information available in the context, how good is the response? This is harder to measure automatically because &#8220;quality&#8221; depends on the use case. For a customer service application, quality might mean accuracy, completeness, and appropriate tone. For a code generation tool, quality might mean correctness, efficiency, and adherence to coding standards. For a summarization system, quality might mean faithfulness to the source material and appropriate compression. Defining what quality means for your specific application is a prerequisite that many teams skip.</p><p>The third dimension is robustness across variation. A prompt that works well for straightforward queries but breaks down on edge cases, ambiguous inputs, or adversarial phrasings is not production-ready. Evaluation should include the easy cases that the system should handle routinely, the hard cases that test the boundaries of system capability, and the adversarial cases that probe for failure modes.</p><p>The practical mechanism for prompt evaluation is a fixed test set of representative queries, sometimes called a &#8220;golden set&#8221; or &#8220;eval set.&#8221; This test set should include examples from each category of expected input, annotated with expected outputs or, at a minimum, with criteria for acceptable outputs. When evaluating a prompt change, you run the new prompt against the entire test set and compare the results to those of the previous version.</p><p>Side-by-side comparison is the workhorse technique. Present the outputs from two prompt versions to evaluators (human or automated) without revealing which version produced which output, and ask them to assess quality. This approach controls for the bias that comes from knowing which version is &#8220;new&#8221; and which is &#8220;old.&#8221;</p><p>LLM-as-a-judge, where you use one model to evaluate the outputs of another, has become a standard technique for scaling prompt evaluation beyond what human review can handle. The approach works well for many evaluation criteria, but it comes with documented limitations. Research has shown that LLM judges exhibit position bias (preferring whichever answer appears first), verbosity bias (rating longer answers higher regardless of quality), and what researchers have termed &#8220;situational preference,&#8221; where the same judge gives inconsistent ratings on difficult cases. Even top-performing models fail to maintain consistent preferences in nearly a quarter of challenging evaluations. Calibrate your automated judges against human judgment on a sample of cases, and do not treat their scores as ground truth.</p><h2>Evaluating Retrieval</h2><p>If your application uses retrieval-augmented generation, and most enterprise LLM applications do, retrieval quality is likely the single most important factor in system performance. A perfect prompt cannot compensate for a retrieval pipeline that surfaces the wrong documents. And retrieval failures are particularly insidious because they are invisible to end users. The user sees a fluent, confident answer. They have no way to know that the answer was generated from irrelevant or outdated source material.</p><p>Retrieval evaluation examines three properties. The first is relevance. Of the documents or passages retrieved for a given query, how many are actually relevant to answering it? This is the most fundamental retrieval metric, and poor relevance is the most common retrieval failure. Studies of production RAG systems have found that retrieved passages frequently do not contain the information needed to answer the query correctly. When this happens, the model either hallucinates an answer, synthesizes something plausible but wrong from tangentially related material, or acknowledges it cannot answer. And hallucination is the most common outcome.</p><p>The second property is coverage. Even if retrieved documents are relevant, do they contain all the information needed for a complete answer? Partial retrieval is a subtle failure mode. The system might find a document that mentions the customer&#8217;s pricing tier, but miss the document that describes the exception applicable to their specific contract. The resulting answer is partially correct and therefore more dangerous than a clearly wrong answer, because partial correctness is harder to detect.</p><p>The third property is noise. Retrieval systems often return a mix of relevant and irrelevant passages. The model must then determine which passages to rely on and which to ignore. Research has demonstrated that RAG systems are sensitive to the ordering and composition of retrieved context. Simply reordering the same set of retrieved documents can change the model&#8217;s answer. Noisy retrieval puts an unfair burden on the model to perform information triage that it was not designed for.</p><p>Measuring these properties requires ground truth. For each query in your evaluation set, you need to know which documents or passages contain the correct answer. Building this ground truth is labor-intensive but essential. Without it, you cannot distinguish between a prompt problem and a retrieval problem when the system produces an incorrect answer.</p><p>Practical retrieval metrics include precision at k (the fraction of the top k retrieved documents that are relevant), recall (the fraction of all relevant documents that were retrieved), and mean reciprocal rank (the average position of the first relevant document in the result list). These are standard information retrieval metrics that predate LLMs by decades, and they remain the right tools for the job.</p><p>One pattern that experienced teams watch for is the interaction between retrieval quality and prompt quality. When retrieval is good, providing relevant, complete, low-noise context, even a mediocre prompt produces acceptable results. When retrieval is poor, even an expertly crafted prompt produces unreliable results. This interaction means that investing in retrieval quality often yields higher returns than investing in prompt refinement, particularly for systems that have already undergone initial prompt optimization.</p><h2>Evaluating Tool-Calling Behavior</h2><p>As LLM applications grow more capable, they increasingly rely on tool calling to perform actions, retrieve structured data, execute calculations, and interact with external systems. Agentic architectures, where the model orchestrates multi-step workflows across multiple tools, are becoming the standard pattern for complex enterprise applications. And tool calling introduces an entirely new category of failure modes that prompt-level evaluation cannot capture.</p><p>Tool-calling evaluation examines several dimensions. </p><h4>The first is tool selection</h4><p>Given a user query, does the model choose the correct tool? In systems with many available tools, the model must interpret the user&#8217;s intent and map it to the appropriate capability. Errors here include calling the wrong tool entirely, calling a tool when the query should have been answered directly from the model&#8217;s knowledge, or failing to call any tool when one was clearly needed. Research benchmarks have found that even top-performing models achieve only about 50% accuracy on multi-step tool-use tasks, highlighting the room for improvement.</p><p>A concrete example makes this tangible. Consider an internal financial analysis agent that has access to a revenue lookup API, a forecasting model, a currency conversion tool, and a report generation service. A user asks, &#8220;How did our EMEA revenue compare to the forecast in Q3?&#8221; The correct behavior is to call the revenue lookup API with the EMEA region filter and Q3 date range, then call the forecasting tool with the same parameters, then compare the two results. A tool selection failure might look like the model calling the currency conversion tool first (because the query mentions a non-US region) or skipping the forecasting tool entirely and answering based on its parametric knowledge of what forecasts &#8220;typically&#8221; look like. Both failures produce fluent, confident responses. Both are wrong in ways that a prompt evaluation would never detect.</p><h4>The second dimension is parameter correctness</h4><p>Even when the model selects the right tool, it must construct the correct parameters. Staying with the same example, suppose the model correctly calls the revenue API but passes &#8220;Q3 2025&#8221; as a string when the API expects start and end dates in ISO format. Or it passes the region code &#8220;EMEA&#8221; when the API expects &#8220;EU&#8221; plus separate calls for the Middle East and Africa. The API might return an error, or, worse, a valid but incorrect result, perhaps defaulting to global revenue when it receives an unrecognized region code. Parameter errors are particularly dangerous precisely because the tool often executes successfully, the model receives a response, and it incorporates that response into its answer with full confidence. The error is buried in the mechanics of the call rather than surfaced as an obvious failure. The user sees a number that looks reasonable, but it is wrong.</p><h4>The third dimension is sequencing</h4><p>Many real-world tasks require multiple tool calls in a specific order, where the output of one call informs the input to the next. In the EMEA revenue example, the model needs to complete the revenue lookup before it can compare it to the forecast. If it calls both tools simultaneously with independent parameters, it might compare Q3 actual revenue against a Q2 forecast, or against a forecast generated with different regional assumptions. Sequencing errors also include making redundant calls that waste resources without improving the answer, or failing to pass the output of one call as input to the next when the tools are designed to work in a chain.</p><h4>The fourth dimension is failure handling</h4><p>What does the model do when a tool call fails? Production systems routinely encounter timeout errors, rate limits, malformed responses, and service outages. If the revenue API is down, does the agent tell the user it cannot retrieve the data right now, or does it quietly produce an answer based on whatever information it has in context, which might be last quarter&#8217;s cached results or its own training data? A well-designed system should retry when appropriate, fall back to alternative approaches when possible, and communicate limitations transparently when it cannot fulfill the request. A poorly designed system either hallucinates a response as if the tool call had succeeded, or fails silently, or enters a retry loop that burns tokens and time without ever surfacing the issue.</p><p>Evaluating tool-calling behavior requires test cases that cover each of these dimensions, including explicit failure cases. Your evaluation set should include scenarios where the correct action is not to call any tool, where a tool call should fail, and the model should handle the failure gracefully, and where multiple tools must be orchestrated in sequence. The negative and edge cases are often more revealing than the happy-path tests that most teams start with. A useful practice is to log every tool call in production, including the parameters passed and the responses received, and periodically audit a sample for correctness. This audit often reveals patterns of parameter errors or unnecessary calls that would be invisible without systematic inspection.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/stop-guessing-how-to-evaluate-modern?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/stop-guessing-how-to-evaluate-modern?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>System-Level Evaluation</h2><p>Component-level evaluation of prompts, retrieval, and tool calling is essential for diagnosis, but it is insufficient to measure what the user actually experiences. System-level evaluation examines the application's complete output, from query to final response, and assesses whether the system accomplished its purpose.</p><p>System-level evaluation should measure at least four properties.</p><h4>The first is correctness</h4><p>Is the information in the response factually accurate? This seems straightforward, but it is operationally challenging because it requires some source of truth to evaluate against. For systems that answer questions from a defined knowledge base, correctness can be assessed against it. For systems that perform calculations or data lookups, correctness can be verified against the underlying data. For systems that generate creative content or provide subjective guidance, correctness may need to be reframed in terms of appropriateness or alignment with organizational guidelines.</p><h4>The second is faithfulness</h4><p>Even if the response is factually accurate, is it grounded in the context that was provided? A system might produce a correct answer that it derived from its parametric knowledge rather than from the documents it was supposed to reference. This matters because parametric knowledge can be outdated, and because users of enterprise applications need to trust that answers are based on authoritative sources rather than the model&#8217;s general training. Faithfulness evaluation compares the claims in the response against the retrieved context and flags any claims not supported by the provided material.</p><h4>The third is usefulness </h4><p>A response can be correct and faithful while still being unhelpful. It might be too vague, too verbose, poorly structured, missing the key information the user needed, or pitched at the wrong level of technical detail. Usefulness is inherently subjective and use-case-specific, which makes it harder to automate. But it is arguably the most important dimension because it directly determines whether users derive value from the system.</p><h4>The fourth is operational performance</h4><p>Production systems must meet practical constraints around latency, cost, and reliability. An answer that takes 30 seconds is often worse than a slightly less comprehensive answer that arrives in three seconds. A system that costs $5 per query to run will not scale to enterprise volumes. Operational metrics, including response time, token consumption, error rates, and infrastructure costs, are essential inputs to system-level evaluation. Cost-aware evaluation is becoming increasingly important as organizations discover that the most accurate system configuration can be four to ten times more expensive than a balanced one, and most production use cases do not require maximum accuracy on every call.</p><h4>Safety and compliance represent a fifth dimension</h4><p>They are particularly important for enterprise deployments. Does the system respect content policies? Does it decline requests outside its defined scope? Does it avoid generating content that could create legal, regulatory, or reputational risk? These constraints are typically defined by organizational policy and tested through targeted adversarial evaluation, including red-teaming exercises that attempt to elicit prohibited behaviors.</p><h2>Building an Evaluation Loop</h2><p>Prompt evaluation, retrieval evaluation, tool-calling evaluation, and system-level assessment are most effective when integrated into a continuous loop rather than treated as periodic activities.</p><p>The foundation of the evaluation loop is a representative benchmark set. This is a curated collection of test cases that reflects the full range of inputs your system encounters in production. Building this set is one of the highest-value investments a team can make, and it is never truly finished. Good benchmark sets share several characteristics.</p><h4>They are representative</h4><p>The distribution of test cases should roughly match the distribution of real queries. If 60% of your production traffic involves policy questions, 20% involves account inquiries, and 20% involves troubleshooting, your benchmark set should reflect those proportions.</p><h4>They are annotated</h4><p>Each test case should include criteria for acceptable responses. For factual queries, this means the correct answer. For more subjective queries, this means a rubric that defines what &#8220;good&#8221; looks like. Annotations are expensive to create but essential for automated evaluation.</p><h4>They are versioned</h4><p>As you add new test cases, fix incorrect annotations, or adjust evaluation criteria, you need to track those changes so you can compare results across versions of the benchmark set and determine whether improvements are real or artifacts of benchmark changes.</p><h4>They grow from production failures</h4><p>Every time the system fails in production, and the failure is identified, the offending query should be added to the benchmark set with an annotation indicating the correct behavior. Over time, this process builds a benchmark set specifically tuned to your system&#8217;s failure modes, making it increasingly useful for detecting regressions.</p><p>With a benchmark set in place, you need metrics for each layer of the system. Retrieval metrics such as precision and recall at the retrieval layer, instruction following and quality scores at the prompt layer, selection accuracy and parameter correctness at the tool-calling layer, and correctness, faithfulness, and usefulness at the system level. These metrics should be computed automatically whenever a change is proposed, providing immediate feedback about whether the change improves or degrades performance.</p><p>The final element of the evaluation loop is failure review. On a regular cadence, the team should examine recent production failures, classify them by root cause (prompt, retrieval, tool calling, data quality, or something else), and convert them into new test cases. This practice serves two purposes. It continuously improves the benchmark set and builds organizational knowledge of the system&#8217;s failure patterns, which in turn informs architectural and strategic decisions.</p><h2>What Mature Teams Do Differently</h2><p>The gap between teams that struggle with LLM quality and teams that manage it well is not primarily about talent or tooling. It is about whether evaluation is embedded in how the team works or bolted on as a periodic exercise.</p><p>The teams that get this right have made evaluation a reflex, not an event. When an engineer changes a prompt, the benchmark suite runs before the change is merged, just as unit tests run before a code change is merged. The results show up in the review itself. Regressions are visible immediately. Nobody has to remember to run the eval suite because the pipeline runs it for them. This is not a high bar to clear technically, but it requires a decision that evaluation is important enough to be a gate rather than a suggestion. Most teams that have not made that decision are still running evaluations manually, sporadically, and only when something has already gone wrong.</p><p>Longitudinal tracking matters more than most teams initially appreciate. A single evaluation tells you where the system stands today. A trend line over weeks and months tells you something far more valuable. It reveals a gradual degradation that no single snapshot would catch. It shows whether a series of small, individually harmless changes has compounded into a meaningful regression. It gives your AI Council or risk committee something concrete to review beyond anecdotes. The teams that produce these trend lines are the ones that can answer the question &#8220;Is our AI getting better or worse?&#8221; with data rather than intuition.</p><p>One of the most consequential shifts happens when teams start using evaluation data to inform product decisions rather than treating it exclusively as a model-tuning signal. A pattern I have seen repeatedly is that evaluation reveals a category of queries where the system consistently underperforms, and the engineering instinct is to fix the model&#8217;s handling of those queries. But when product leaders see the same data, they sometimes reach a different conclusion. Maybe those queries should be routed to a human. Maybe the interface should guide users away from ambiguous phrasing. Maybe the feature needs tighter scoping. The cheapest, most effective fix for a struggling LLM feature is often a product change rather than a model change. Teams that wall off evaluation data inside the engineering function miss this entirely.</p><p>The open-source ecosystem has matured substantially in the past eighteen months. Frameworks like RAGAS, DeepEval, and TruLens now provide standardized metrics for retrieval quality, faithfulness, and answer relevance, reducing the cold-start effort required to build evaluation infrastructure. But frameworks are scaffolding, not solutions. They need to be populated with test data that represents your actual traffic, configured with metrics that reflect your actual quality standards, and reviewed by people who understand both the technical system and the business domain. A team that installs RAGAS but never builds a representative benchmark set has a tool, not a practice.</p><p>Finally, the teams that sustain evaluation over time are the ones that connect it to governance. Evaluation metrics are the evidence base for compliance assertions under the EU AI Act. They are the data that risk committees need to approve or restrict AI deployments. They are the foundation of any credible claim that your AI systems perform as intended. When evaluation is positioned as a governance capability rather than an engineering chore, it earns the organizational investment it needs to remain viable, and it attracts cross-functional attention that makes it useful beyond the engineering team.</p><h2>Connecting Evaluation to Trusted AI</h2><p>Trusted AI requires two complementary capabilities: governance frameworks that define what AI systems should do, and observability infrastructure that reveals what AI systems are actually doing. Evaluation is where these capabilities converge for LLM applications.</p><p>Governance frameworks establish the standards. The system should provide accurate information. It should cite authoritative sources. It should decline to answer questions outside its scope. It should not generate harmful content. These are policy decisions that reflect organizational values and risk appetite.</p><p>Evaluation infrastructure verifies those standards in practice. It measures accuracy against defined benchmarks. It checks faithfulness against retrieved sources. It tests adherence to scope against adversarial inputs. It monitors safety constraints against known attack patterns.</p><p>Without governance, evaluation has no criteria. Without evaluation, governance has no evidence. The organizations building truly trustworthy LLM applications understand this relationship and invest accordingly in both.</p><p>The practical implication is clear. If your organization is deploying LLM applications in production, you need more than good prompts and a capable model. You need a systematic evaluation practice that covers every layer of the system, runs continuously, and produces evidence that your AI systems meet the standards you have set for them.</p><p>Stop guessing and start measuring. The difference between the two is the difference between AI that you hope works and AI that you know works.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/stop-guessing-how-to-evaluate-modern/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/stop-guessing-how-to-evaluate-modern/comments"><span>Leave a comment</span></a></p><p></p><p></p>]]></content:encoded></item><item><title><![CDATA[Your AI Passed Testing. Will It Survive Production? ]]></title><description><![CDATA[Experiments, staged rollouts, and runtime guardrails that turn deployment risk into repeatable control]]></description><link>https://trustedai.recodework.com/p/your-ai-passed-testing-will-it-survive</link><guid isPermaLink="false">https://trustedai.recodework.com/p/your-ai-passed-testing-will-it-survive</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Sat, 02 May 2026 15:40:06 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!u5xk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!u5xk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!u5xk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg 424w, https://substackcdn.com/image/fetch/$s_!u5xk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg 848w, https://substackcdn.com/image/fetch/$s_!u5xk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!u5xk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!u5xk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg" width="1000" height="443" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/eb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:443,&quot;width&quot;:1000,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:527597,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/196186371?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!u5xk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg 424w, https://substackcdn.com/image/fetch/$s_!u5xk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg 848w, https://substackcdn.com/image/fetch/$s_!u5xk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!u5xk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb800a59-ea23-418f-84ac-a594674433e4_1000x443.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Every AI team eventually learns the same hard lesson. The model that performed beautifully in testing does something unexpected the moment it encounters real users, real data and real stakes.</p><p>A customer service chatbot that passed every benchmark starts inventing refund policies that do not exist. A recommendation engine that showed strong offline accuracy begins surfacing bizarre results when exposed to holiday traffic patterns it never saw in training. A fraud detection model that cleared validation with flying colors starts flagging legitimate transactions at twice the expected rate because the distribution of real-world inputs drifted from the training set within weeks of deployment.</p><p>These are not hypothetical scenarios. They are production realities that organizations encounter with alarming regularity. In 2024, Air Canada was ordered by a tribunal to compensate a passenger after its AI chatbot fabricated a bereavement discount policy that contradicted the airline&#8217;s actual terms. A Chevrolet dealership&#8217;s chatbot was manipulated into offering a $76,000 Tahoe for one dollar. Taco Bell&#8217;s drive-through AI agent entered an order for 18,000 cups of water because it could not recognize the conversational absurdity of the request. McDonald&#8217;s AI-powered hiring chatbot frustrated applicants with poor responses while simultaneously exposing personal data through elementary security failures in its underlying infrastructure.</p><p>The common thread across these failures is not that the systems were poorly built. Many of them cleared internal testing. The problem is that offline evaluation, no matter how rigorous, cannot fully replicate the complexity of production. Real users probe boundaries that test scripts do not anticipate. Data distributions shift in ways that static datasets cannot represent. Adversarial inputs appear that no red team imagined. And the consequences of failure in production are measured not in accuracy points on a validation set but in regulatory enforcement, reputational damage, and eroded customer trust.</p><p>This is the fundamental challenge of production AI. And it demands a fundamentally different approach to managing change.</p><h2>The Limits of Offline Testing</h2><p>The traditional software development model follows a reassuring sequence. Developers write code, testers verify it against requirements, and if the tests pass, the code ships to production. The behavior of traditional software is deterministic. Given the same inputs, it produces the same outputs. If it works correctly in testing, it will work correctly in production, assuming the environment is properly configured.</p><p>AI systems break this model in ways that have profound implications for how we deploy and manage them.</p><p>First, AI systems are probabilistic rather than deterministic. A large language model given the same prompt twice may produce different outputs. A classification model operating near a decision boundary may oscillate between categories for borderline inputs. This inherent variability means that testing can characterize the distribution of behaviors but cannot guarantee the behavior for any specific interaction.</p><p>Second, AI systems are deeply sensitive to the data they encounter. A model trained on data from one time period, geography, or user population will often perform differently when exposed to data from another. This phenomenon, known as data drift, is not a bug to be fixed but an intrinsic characteristic of systems that learn statistical patterns from examples. The patterns shift because the world shifts.</p><p>Third, AI systems can exhibit emergent behaviors that their designers did not anticipate. When users interact with a chatbot in unexpected ways, when adversarial actors probe for vulnerabilities, or when edge cases compound in novel combinations, the system may produce outputs that were never observed during development. The Chevrolet chatbot incident illustrates this perfectly. The system was never designed to make binding purchase offers, yet a creative prompt convinced it to do exactly that.</p><p>Fourth, the interaction between AI systems and their users creates feedback loops that do not exist in testing environments. Users adapt their behavior in response to AI outputs. They learn what the system responds to. They find workarounds. They discover exploits. Testing environments are sterile by comparison.</p><p>For all of these reasons, the question facing AI teams is not whether problems will emerge after deployment. They will. The question is whether the organization has the infrastructure and processes to detect those problems quickly, contain their blast radius, and respond before they become incidents. This requires moving from a mindset of &#8220;test and release&#8221; to a discipline of continuous evaluation in production, which practitioners increasingly refer to as online evaluation.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>The Production Operating Model for AI Change</h2><p>Mature engineering organizations have long recognized that deploying software safely requires more than testing alone. The practices of shadow deployment, canary releases, A/B testing, and progressive rollout are well established in software engineering. What is relatively new is their adaptation for AI systems, where the challenges are different, and the stakes are often higher.</p><p>These deployment patterns share a common philosophy. They separate validation from exposure. They limit the blast radius of failures. And they create decision points where teams can evaluate production evidence before committing to a broader rollout. Together, they form an operating model for managing AI change that balances the need for improvement with the imperative for control.</p><h4>Shadow Deployment: Testing Without Exposure</h4><p>Shadow deployment is the lowest-risk starting point for introducing any significant AI change into production. The concept is straightforward. The new model or prompt version receives real production traffic, but its outputs are logged rather than served to users. The existing system continues to handle all user-facing interactions exactly as before.</p><p>This approach is powerful because it exposes the new system to the full complexity of production inputs without risking users. Teams can compare the shadow model&#8217;s outputs against the production model&#8217;s outputs, looking for divergences, errors, and unexpected behaviors. They can evaluate latency and resource consumption under real load. They can identify edge cases that test datasets missed.</p><p>The critical requirement for effective shadow deployment is a robust evaluation layer. Without it, shadow mode produces nothing more than a growing pile of logs that nobody examines. Organizations need automated scoring systems that compare outputs against defined quality criteria, anomaly detection that flags unusual patterns, and human review processes for a representative sample of shadow outputs. For large language model applications, this often means using an automated judge, a separate model that scores candidate outputs against the baseline, and gating progression on whether the candidate meets or exceeds quality thresholds.</p><p>Shadow deployment is not free. It roughly doubles the compute required for inference, since every request is processed by both the production model and the shadow candidate. For organizations running large-scale AI systems, this cost can be high. Selective shadow deployment, where a representative sample of traffic is mirrored rather than the full stream, can reduce costs while still providing a meaningful signal.</p><p>The duration of a shadow deployment depends on the significance of the change and the volume of traffic. For a major model version upgrade, teams might run shadow mode for days or weeks to accumulate sufficient evidence across diverse traffic patterns. For a minor prompt adjustment, a shorter window may suffice. The key is to define exit criteria in advance: which metrics must the shadow model meet, and over what time period, before proceeding to the next stage?</p><h4>Canary Releases: Controlled Exposure at Small Scale</h4><p>Once a candidate model has demonstrated acceptable performance in shadow mode, the next step is controlled exposure to actual users. A canary release routes a small percentage of production traffic to the new model while the majority continues to be served by the existing system.</p><p>The name originates from the coal-mining practice of carrying canaries underground to detect toxic gases. The birds would react to danger before conditions became lethal for the miners. Similarly, a canary release exposes a small subset of users to the new model, allowing problems to be detected before they affect the broader user base.</p><p>The mechanics of a canary release differ from a random traffic split in an important way. Canary deployments typically assign users deterministically to the canary group, meaning the same user consistently receives responses from the same model throughout the canary period. This consistency matters because it allows teams to track user-level behavioral signals, such as satisfaction, engagement, and complaint rates, across the canary cohort compared to the control population.</p><p>A well-structured canary release follows a stepwise progression. Start with 1% of traffic to verify that the infrastructure is functioning correctly. Increase to 5% and monitor for quality regressions. If metrics remain within acceptable bounds, expand to 20%, then 50%, and eventually 100%. Each step should have explicit gating criteria that must be satisfied before the next increase. And at every stage, the team must be able to route traffic back to the baseline model immediately if problems emerge.</p><p>For AI systems, the metrics tracked during a canary release go beyond those monitored in traditional software deployments. Teams should monitor changes in output quality, measured through automated evaluations or user feedback signals. They should track latency percentiles, since AI inference times can vary significantly across model versions. They should monitor cost per request, because token consumption patterns can shift when models or prompts change. They should watch for changes in refusal rates, output length distributions, and error patterns. And they should be alert to fairness metrics that might indicate the new model treats certain user populations differently from the baseline.</p><p>Automated rollback is not optional for production canary deployments. Teams must define explicit thresholds. If error rates exceed a certain threshold, if latency exceeds a defined bound, or if user complaint rates spike, the system should automatically route 100% of traffic back to the baseline without requiring human intervention. The window between detecting a problem and containing it is the window of exposure, and automation shrinks that window from hours to seconds.</p><h4>A/B Testing: Measuring Impact with Statistical Rigor</h4><p>While canary releases prioritize safety during rollout, A/B testing prioritizes measurement. An A/B test randomly assigns users to one of two or more model variants and statistically compares business outcomes across groups. The goal is not just to verify that a new model is safe to deploy, but to quantify whether it meaningfully delivers better results.</p><p>A/B testing is most valuable when the question is not whether a model works, but which of several alternatives performs best with respect to a specific business objective. Should the recommendation engine optimize for click-through rate or session duration? Does a more conservative prompt reduce hallucination rates without degrading user satisfaction? Is the trade-off between a more accurate but slower model worth the latency cost?</p><p>These are questions that offline evaluation cannot answer definitively because they depend on user behavior in context. A/B testing provides an experimental framework for answering them with data rather than assumptions.</p><p>Effective A/B testing for AI systems requires careful attention to several factors. Sample size and duration must be sufficient to detect meaningful differences, accounting for the natural variability in AI system outputs. Randomization must be robust to ensure that treatment groups are comparable. And the metrics being measured must be clearly defined and aligned with business objectives, not just technical performance measures.</p><p>One challenge specific to AI systems is that effects may take time to manifest. A model change that improves immediate user engagement might degrade long-term retention. A chatbot that provides more comprehensive answers might increase satisfaction for individual interactions but reduce the number of queries per session. Designing experiments that capture these longer-horizon effects requires thoughtful metric selection and sufficient experimental duration.</p><h4>Progressive Rollout: Putting It All Together</h4><p>In practice, these deployment patterns are not alternatives. They are stages in a progressive rollout strategy that moves from zero exposure to full deployment through a series of controlled steps, each generating evidence that informs the next.</p><p>The sequence typically follows a pattern. Shadow deployment verifies that the candidate model performs acceptably under production traffic without exposing users. Canary release introduces controlled user exposure at a small scale, with automated rollback if problems emerge. A/B testing, either concurrent with or following the canary phase, provides statistical evidence of the candidate&#8217;s impact on business metrics. Full rollout proceeds only after both safety and performance criteria are satisfied.</p><p>This progressive approach creates multiple opportunities to catch problems before they affect the full user population. It also creates a documented evidence trail that supports governance requirements, demonstrating that changes were validated systematically rather than deployed on faith.</p><h2>Guardrails: Runtime Protection That Operates Continuously</h2><p>Deployment patterns manage the risk of introducing change. Guardrails manage the risk of ongoing operation. Even after a model has been fully deployed through a rigorous rollout process, it remains exposed to inputs and conditions that can produce harmful, incorrect, or inappropriate outputs. Guardrails are the runtime controls that intercept these problems before they reach users.</p><p>The most effective guardrail architectures operate in layers, following the principle of defense in depth. No single layer catches everything, but multiple layers dramatically reduce the probability of failures reaching users.</p><h4>Input Validation: The First Line of Defense</h4><p>Before a user&#8217;s input ever reaches the AI model, input validation checks can intercept problematic requests. This includes detecting prompt-injection attempts, in which users embed instructions intended to override the model&#8217;s intended behavior. It includes scanning for personally identifiable information that should not be processed. It includes identifying patterns associated with adversarial manipulation. And it includes basic format and length validation that prevents resource exhaustion from malicious inputs.</p><p>Input validation is particularly important for large language model applications because the attack surface is the natural language prompt itself. Unlike traditional software, where inputs have defined schemas, LLM inputs are unconstrained text. This openness is what makes LLMs powerful, but it also makes them vulnerable to creative manipulation. The Chevrolet chatbot incident, where a user instructed the system to end every response with a binding offer statement, is a textbook example of prompt injection that basic input validation could have caught.</p><h4>Output Filtering: Catching Problems Before Users See Them</h4><p>After the model generates a response, but before it is delivered to the user, output filtering provides a second line of defense. This layer checks for toxic or harmful content, factual claims that can be verified against authoritative sources, personally identifiable information that may have been generated or repeated from training data, responses that fall outside the application's defined scope, and outputs that violate organizational policies or regulatory requirements.</p><p>Output filtering is not a replacement for model quality. A well-trained, well-prompted model should produce appropriate outputs in the vast majority of cases. But guardrails exist precisely for the cases where the model&#8217;s internal constraints are insufficient, the edge cases, the adversarial inputs, the unexpected data patterns that cause even well-designed models to produce outputs that should never reach a user.</p><h4>Topical and Behavioral Constraints</h4><p>Beyond content safety, many enterprise AI applications require models to remain within defined boundaries. A customer service agent should not provide medical advice. A financial planning tool should not make specific investment recommendations without appropriate disclaimers. A hiring assistant should not ask questions that violate employment law.</p><p>Topical guardrails enforce these boundaries by monitoring whether model outputs remain within the application's defined scope and by redirecting or blocking responses that stray outside it. These constraints are especially important in regulated industries, where the consequences of an AI system exceeding its intended scope can include regulatory penalties and legal liability.</p><h4>Rate Limiting and Resource Controls</h4><p>A category of guardrail that receives less attention but is equally important involves controlling the operational boundaries of AI systems. Rate limiting prevents individual users or sessions from consuming disproportionate resources. Token limits prevent runaway responses that waste compute and potentially expose the system to denial-of-service conditions. Cost controls ensure that unexpected surges in usage do not produce budget-busting bills before anyone notices.</p><p>These operational guardrails are the AI equivalent of circuit breakers in electrical systems. They exist to prevent cascading failures and to ensure that a problem with one user, one session, or one input does not compromise the system&#8217;s availability for everyone else.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/your-ai-passed-testing-will-it-survive?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/your-ai-passed-testing-will-it-survive?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Kill Switches: The Last Line of Defense</h2><p>Beyond layered guardrails, every production AI system needs the ability to be stopped. Not gradually. Not after a committee meeting. Immediately.</p><p>A kill switch is not an admission of poor design. It is a requirement for operating complex systems at scale. The question is not whether you will ever need to shut down an AI system in production. It is whether you can do so quickly, cleanly, and without causing additional damage.</p><p>Effective kill switches require several design considerations that many organizations overlook.</p><p>First, &#8220;stop&#8221; needs to be defined precisely for each AI system. For some applications, stopping means turning off the AI component entirely and falling back to a non-AI alternative, such as routing customer service requests to human agents. For others, it means switching the AI to a restricted mode, such as read-only operation or a simpler, more conservative model. For agentic AI systems that take autonomous actions, stopping may mean revoking tool permissions and halting queued jobs while preserving the ability to resume later.</p><p>Second, the kill switch must be accessible to authorized operators without requiring the engineering team that built the system to be involved. During an incident at 2 AM on a Saturday, the person with operational responsibility needs to be able to act without waiting for a developer to wake up and push a configuration change. This means kill switches should live in an operational control plane, not buried in application code.</p><p>Third, kill switches must be tested regularly. A kill switch that has never been exercised may not work when needed. Organizations should conduct periodic drills, the AI equivalent of a fire drill, to verify that shutdown procedures function as expected and that operational staff know how to execute them.</p><p>Fourth, kill switches should support graduated response levels. The ability to shift from full autonomy to approval-only mode, to read-only mode, to full shutdown gives operators the flexibility to match the response to the incident's severity. A problem that warrants restricting the system&#8217;s capabilities may not require taking it offline entirely.</p><p>The financial consequences of lacking kill switch capability are well documented. Knight Capital&#8217;s algorithmic trading failure in 2012, which resulted in $440 million in losses in 45 minutes, remains a cautionary tale about what happens when automated systems malfunction and operators cannot stop them quickly enough. Zillow&#8217;s iBuying algorithm, which contributed to $881 million in losses before the program was shut down, illustrates how AI systems can cause compounding damage when organizations are slow to intervene.</p><h2>From Heroic Oversight to Repeatable Controls</h2><p>A pattern emerges in organizations whose AI governance relies on the vigilance of individual experts rather than on systematic processes. Things work fine as long as the right person is paying attention. But people take vacations. They change jobs. They get pulled into other priorities. And when the expert is not watching, problems go undetected until they become incidents.</p><p>This is the difference between heroic oversight and repeatable controls. Heroic oversight depends on the knowledge, judgment, and availability of specific individuals. Repeatable controls are embedded in systems and processes that operate consistently regardless of who is on duty.</p><p>The deployment patterns and guardrail architectures described in this article are the building blocks of repeatable controls for AI systems. Shadow deployments systematically verify candidate models against production traffic, not because someone remembered to check. Canary releases limit exposure automatically through traffic routing rules, not because an engineer is watching a dashboard. Automated rollback triggers revert to safe configurations when predefined thresholds are exceeded, not because someone noticed a problem and escalated it. Guardrails intercept problematic outputs before they reach users on every request, not on a sampling basis when someone has time to review logs.</p><p>This shift from heroic oversight to repeatable controls is fundamentally a governance argument. Trust in AI systems does not come from having brilliant engineers who care deeply about quality, although that certainly helps. Trust comes from demonstrating that controls are in place, operate consistently, and are supported by documented evidence of their effectiveness.</p><p>This matters for regulatory compliance. The EU AI Act requires organizations deploying high-risk AI systems to implement risk management, human oversight, and technical documentation. Regulators will not be satisfied with assurances that talented engineers are monitoring the system. They will want evidence that governance processes exist, are followed, and produce measurable results. Progressive rollout procedures with documented gating criteria, automated guardrails with audit logs, and tested kill switch procedures with drill records provide exactly this kind of evidence.</p><p>It matters for board-level accountability. When the board asks how the organization manages AI risk, the answer should not be &#8220;we have really good people.&#8221; The answer should describe the systems, processes, and controls that ensure consistent governance regardless of individual availability. Surveys continue to show that only a small minority of boards discuss AI at every meeting, yet AI-related risks are becoming more frequent and consequential. The organizations that can present their AI governance as an operating system rather than a collection of ad hoc practices will be better positioned to satisfy board-level scrutiny.</p><p>And it matters for organizational scalability. An approach that depends on heroic oversight works when you have three AI systems. It does not work when you have 30 or 300. As organizations scale their AI portfolios, the only sustainable path is to embed governance into the infrastructure itself, making safe deployment the path of least resistance rather than an additional burden that teams are tempted to shortcut under deadline pressure.</p><h2>Building the Continuous Operating System for AI Change</h2><p>Trusted AI in production is not a single safety check performed before deployment. It is a continuous operating system for change, one that combines experimentation, staged rollout, and runtime guardrails so teams can improve systems without losing control.</p><p>This operating system has several essential components working together.</p><p>Offline evaluation remains important as a screening mechanism. It eliminates candidates that are clearly unsuitable before they consume production resources. But it is the beginning of the evaluation process, not the end.</p><p>Shadow deployment bridges the gap between offline evaluation and production exposure, validating candidate models against real-world traffic without risk to users.</p><p>Canary releases and A/B tests introduce controlled exposure, generating evidence about the candidate&#8217;s real-world performance while limiting the blast radius of any problems.</p><p>Automated rollback ensures that problems detected during staged rollout are contained quickly, without depending on human intervention during the critical window between detection and response.</p><p>Runtime guardrails provide continuous protection by intercepting problematic inputs and outputs, whether the model is newly deployed or has been running for months.</p><p>Kill switches provide the ultimate safety net, enabling rapid shutdown when guardrails alone are insufficient to contain an emerging problem.</p><p>And all of these components produce documented evidence, the audit trails, metric histories, and decision records that demonstrate governance in practice rather than governance on paper.</p><p>The organizations that build this operating system will have a significant advantage over those that do not. They will be able to iterate faster because their deployment infrastructure reduces the risk of each change. They will experience fewer incidents because layered controls catch problems at multiple points. They will satisfy regulatory requirements more easily because governance is embedded in their processes rather than bolted on after the fact. And they will earn stakeholder trust not through promises but through demonstrated, repeatable control.</p><h2>Starting Where You Are</h2><p>Not every organization needs to implement every component of this operating system immediately. The right starting point depends on your current AI maturity, the risk profile of your AI systems, and the resources available for infrastructure investment.</p><p>If you have no production monitoring today, start there. You cannot manage what you cannot see. Implement basic observability that tracks model performance, error rates, and output quality over time.</p><p>If you have monitoring but no staged rollout process, introduce canary releases for your highest-risk AI systems. Define gating criteria and rollback thresholds. Practice the discipline of evidence-based deployment decisions.</p><p>If you have a staged rollout but no runtime guardrails, add input validation and output filtering for your customer-facing AI applications. These layers provide immediate protection against the most common failure modes.</p><p>If you have guardrails but no kill switch capability, design and test shutdown procedures for your most consequential AI systems. Define what &#8220;stop&#8221; means for each system, who has the authority to stop it, and how operations continue when the system is offline.</p><p>And regardless of where you start, document what you build. The value of these controls is not just operational. It is evidentiary. The ability to demonstrate that governance processes exist, function as designed, and produce measurable results transforms AI governance from an aspiration into a competitive advantage.</p><p>The question facing every organization deploying AI in production is not whether to invest in these capabilities. The question is whether you build them before or after the incident, which makes the investment unavoidable.</p><p>The organizations that choose before will be the ones their customers, regulators, and boards trust to operate AI responsibly. And in a market where AI capabilities are increasingly commoditized, that trust may prove to be the most durable competitive advantage of all.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/your-ai-passed-testing-will-it-survive/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/your-ai-passed-testing-will-it-survive/comments"><span>Leave a comment</span></a></p>]]></content:encoded></item><item><title><![CDATA[Why Most LLM Evaluations Fail Before Production]]></title><description><![CDATA[How to build test sets, use synthetic data wisely, and trust LLM-as-a-judge without fooling yourself]]></description><link>https://trustedai.recodework.com/p/why-most-llm-evaluations-fail-before</link><guid isPermaLink="false">https://trustedai.recodework.com/p/why-most-llm-evaluations-fail-before</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Sat, 25 Apr 2026 16:27:50 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!btSV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!btSV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!btSV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg 424w, https://substackcdn.com/image/fetch/$s_!btSV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg 848w, https://substackcdn.com/image/fetch/$s_!btSV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!btSV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!btSV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:4657274,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/195368883?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!btSV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg 424w, https://substackcdn.com/image/fetch/$s_!btSV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg 848w, https://substackcdn.com/image/fetch/$s_!btSV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!btSV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfbca30-abce-419c-9dd6-a5e11220256c_3864x2576.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Every week, enterprise teams build impressive AI prototypes. The demos are polished. The leadership reviews go well. And then the system hits production, and the problems start. Customer-facing outputs hallucinate. Edge cases that never appeared in testing show up constantly in the real world. Confidence scores look great on paper but mean nothing when the stakes are high.</p><p>The culprit is rarely the model itself. It is the evaluation.</p><p>This pattern plays out at organizations of every size and in every industry. A financial services team builds an AI assistant that answers customer questions about account features. In testing, it performs beautifully on the curated Q&amp;A pairs the team assembled. In production, a customer asks about a discontinued product feature, and the system confidently fabricates a non-existent policy. A healthcare technology company deploys an AI-powered clinical documentation tool. It passes internal review with flying colors. Within weeks, clinicians report that the system struggles with the shorthand and fragmented notes that characterize real clinical workflows, producing summaries that omit critical details.</p><p>In each case, the model was capable. The evaluation was not.</p><p>The teams that ship reliable AI systems are not the ones with the fanciest models. They are the ones with the best evaluation discipline. They treat evaluation not as a gate to pass through once but as an operating capability that compounds over time. And they build that capability deliberately, starting with how they construct test sets, extending through how they use synthetic data and automated judges, and culminating in a repeatable system that catches regressions before customers do.</p><p>This post lays out how to build that capability. If you are deploying LLMs in production today, or plan to in the next twelve months, the quality of your evaluation pipeline will determine whether your AI program earns trust or erodes it.</p><h2>The Problem with &#8220;Good Enough&#8221; Evals</h2><p>Most teams begin evaluating too late, with datasets that are too small, too clean, or too disconnected from the conditions their system will actually face. This is not a failure of intent. It is a failure of sequencing. Evaluation is treated as a pre-launch checkpoint rather than a foundational engineering practice.</p><p>The pattern is familiar. A product team identifies a compelling use case. Engineers select a model and build a prototype. The prototype works well on a handful of curated examples. Stakeholders see the demo and approve the project for production. Somewhere in the final weeks before launch, someone assembles a test set from whatever data is convenient, runs a quick evaluation, and declares the system ready.</p><p>The problem is that this approach optimizes for false confidence. A test set of 50 or 100 clean, well-structured examples will almost certainly make any competent LLM look good. Real production traffic is messy. Users phrase things in ways no product manager anticipated. Inputs contain typos, ambiguity, contradictions, and context that the model was never designed to handle. The gap between demo performance and production performance is not a rounding error. It is often the difference between a system that works and one that fails in ways that damage customer trust.</p><p>Research consistently confirms this gap. Generic benchmarks, while useful for broad model comparisons, struggle to predict how a model will behave on your specific task, with your specific data, in your specific domain. OpenAI cofounder Andrej Karpathy captured this well when he noted that the evaluation landscape is in crisis, with previously reliable benchmarks like MMLU having outlived their usefulness and no single replacement adequate for the diversity of real production workloads. The implication for enterprise teams is stark. If you are relying on vendor-reported benchmarks or a small internal test set to make deployment decisions, you are flying blind.</p><p>The antidote is to treat evaluation as a first-class engineering discipline, one that starts before you write your first prompt and continues long after the system reaches production. That begins with how you build your test sets.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>Build Test Sets from Failure, Not Theory</h2><p>The best evaluation datasets do not come from brainstorming sessions or theoretical coverage matrices. They come from real prompts, real workflows, and real mistakes. The distinction matters because the inputs that cause problems in production are rarely the ones that engineers imagine during development.</p><h4>Start with a golden set</h4><p>Every evaluation pipeline needs a small, high-quality foundation of test cases reviewed and labeled by people who understand both the task and the downstream risk. This golden set does not need to be large. Thirty to fifty examples are a viable starting point for many use cases. What it needs to be is representative, carefully labeled, and grounded in real usage patterns rather than hypothetical ones.</p><p>A golden set should include examples that cover the primary use case your system is designed to handle, as well as boundary cases that test the limits of acceptable performance. It should include examples where the correct answer is ambiguous, because those are the cases where production systems stumble most visibly. And it should include examples that reflect the actual distribution of inputs your system will see, not a cleaned-up version of that distribution.</p><h4>Expand from real failures</h4><p>Once your system is handling real traffic, even in a limited pilot, you will begin accumulating the most valuable evaluation data you can get: the cases where things went wrong. Every hallucination, every off-topic response, every instance where a user had to rephrase their question or escalate to a human represents a failure mode that belongs in your test set.</p><p>This approach inverts the traditional logic of evaluation. Instead of starting with what your system should handle and testing whether it does, you start with what your system actually struggles with and test whether you have fixed it. This failure-first methodology produces test sets that are inherently aligned with production risk because they are drawn directly from production experience.</p><h4>Categorize by risk, not just topic</h4><p>Not all test cases carry equal weight. A customer service bot that occasionally recommends the wrong FAQ article is a nuisance. The same bot producing inaccurate information about billing, refund policies, or contractual terms is a liability. Your test set should reflect this distinction. Organize cases by the severity of downstream impact, and weight your evaluation metrics accordingly.</p><p>For teams operating in regulated industries, this risk-based categorization is not optional. The EU AI Act&#8217;s tiered approach to AI regulation maps directly to evaluation rigor. High-risk applications require more extensive testing, more granular metrics, and more thorough documentation. But even outside regulated contexts, understanding which failures matter most allows you to allocate evaluation resources where they will have the greatest impact on trust and business outcomes.</p><h4>Maintain living test sets</h4><p>A test set that was comprehensive six months ago may be dangerously incomplete today. User behavior evolves. Product requirements change. New features introduce new failure modes. The best evaluation programs treat their test sets as living artifacts, with defined processes for reviewing and expanding coverage on a regular cadence.</p><p>Practically, this means establishing a feedback loop between production monitoring and test set curation. When your monitoring detects a new class of errors or a shift in input distribution, those findings should be incorporated into your evaluation dataset within days, not months. The teams that execute this well typically assign explicit ownership of test set maintenance to a specific role or team, rather than leaving it as a shared responsibility that nobody prioritizes.</p><h4>The 5 D&#8217;s of evaluation dataset design</h4><p>Recent research on practical LLM evaluation frameworks has converged on a useful set of principles for building robust test sets. Your dataset needs a <strong>defined </strong>scope that aligns with the specific tasks the model performs. It should be <strong>demonstrative</strong> of production usage, mimicking the inputs and scenarios your system will actually encounter. It should include sufficient <strong>diversity</strong> to cover the range of user populations, input styles, and edge cases relevant to your domain. It needs to be properly <strong>documented</strong>, with clear metadata about labeling criteria, annotator qualifications, and known limitations. And it needs to be <strong>durable</strong>, designed for maintenance and evolution rather than one-time use. These principles apply whether you are building a test set for a customer service chatbot, a document analysis pipeline, or a clinical decision support tool.</p><h2>Use Synthetic Data to Widen Coverage</h2><p>Once you have a solid foundation of real-world test cases, synthetic data becomes a powerful tool for expanding coverage. The keyword is &#8220;expanding.&#8221; Synthetic data is most valuable when it fills gaps that would be difficult, expensive, or time-consuming to fill with real examples. It should complement your golden set, not replace it.</p><h4>Where synthetic data excels</h4><p>There are several scenarios where synthetically generated test cases add significant value. The first is volume. If you need hundreds or thousands of test cases to achieve statistical confidence in your evaluation metrics, generating synthetic variations of real examples is far more efficient than waiting for organic production data to accumulate.</p><p>The second is coverage of rare but important scenarios. Every production system encounters edge cases that appear infrequently but carry outsized risk when they do. A medical information system may rarely encounter questions about drug interactions for patients on five or more medications, but when it does, the quality of the response is critical. Synthetic data allows you to systematically generate these long-tail scenarios rather than hoping they appear in your production logs.</p><p>The third is adversarial testing. Synthetic data is particularly useful for generating inputs that users would rarely produce organically, but that reveal important vulnerabilities. Prompt injection attempts, deliberately ambiguous queries, inputs with misleading context, and requests that test the boundaries of your system&#8217;s safety guardrails are all more efficiently created through generation than collection.</p><h4>Where synthetic data falls short</h4><p>The limitations of synthetic data are real and important to understand. The most significant is that LLM-generated test cases tend to reflect the distribution and style of the generating model, not the distribution and style of your actual users. Research on synthetic test collections has demonstrated that synthetic queries exhibit measurably different patterns from real user queries, with implications for the extent to which evaluation results transfer to production conditions. If your entire test set is synthetic, you risk optimizing for a version of reality that does not match the one your system actually operates in.</p><p>Synthetic data also struggles with subtle, domain-specific nuances that matter most in high-stakes applications. A synthetically generated medical question may be well-formed and topically relevant, but miss the way actual patients describe symptoms, omit relevant context, or fail to capture the emotional undertone that affects how a response should be framed. For these reasons, expert-labeled examples remain essential for the portions of your evaluation that directly inform deployment decisions.</p><h4>A practical framework for combining real and synthetic data</h4><p>The most effective approach for enterprise teams is a layered one. Your golden set of human-labeled examples forms the foundation. These are the cases against which you validate your automated evaluation methods and calibrate your confidence. Synthetic data forms the next layer, providing breadth and volume across scenarios not covered by your golden set. The outermost layer is production data, continuously feeding back into both your golden set and your synthetic generation strategy.</p><p>The ratio between these layers depends on your maturity and risk profile. Early-stage evaluation programs might operate with 50 golden examples, 500 synthetic variations, and a nascent production feedback loop. Mature programs at scale might maintain 500 golden examples, thousands of synthetic cases across multiple dimensions, and a fully automated pipeline that incorporates production findings daily.</p><p>The critical discipline is to never evaluate high-stakes decisions solely against synthetic data. When the outcome of a model response could affect a customer&#8217;s finances, health, employment, or legal standing, the evaluation that gates deployment must include real examples reviewed by qualified humans. Synthetic data can help you map the risk landscape, but human judgment must anchor the decisions that matter most.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/why-most-llm-evaluations-fail-before?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/why-most-llm-evaluations-fail-before?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Treat LLM-as-a-Judge as a Tool, Not the Truth</h2><p>As LLM evaluation scales, human review of every output becomes impractical. This is the operational reality driving the rapid adoption of LLM-as-a-judge. In this methodology, one language model evaluates the outputs of another according to predefined criteria. The approach offers dramatic efficiency gains. At scale, automated judges can evaluate thousands of outputs in the time it would take a human reviewer to evaluate dozens, at a fraction of the cost.</p><p>But efficiency without accuracy is worse than useless. It is actively dangerous because it creates the illusion of quality assurance without the substance. Understanding when and how to trust LLM judges is one of the most consequential evaluation decisions enterprise teams face today.</p><h4>What the research tells us</h4><p>The evidence on LLM-as-a-judge is nuanced. Strong models used as judges can achieve roughly 80% agreement with human evaluators on general instruction-following tasks, which is broadly comparable to the level of agreement between human evaluators themselves. This is a meaningful finding. It means that for many routine evaluation tasks, automated judges are not just cheaper substitutes for human review. They are statistically comparable alternatives.</p><p>However, this aggregate finding masks important variation. In specialized domains such as medicine, law, finance, and technical subject matter, agreement between LLM judges and subject matter experts declines considerably, with some studies reporting rates as low as 60-70%. This degradation is not random. It is systematic, occurring precisely in the domains where evaluation accuracy matters most.</p><h4>Known biases to guard against</h4><p>LLM judges exhibit several well-documented biases that practitioners must account for. Position bias causes judges to favor responses based on their order of presentation rather than their quality. Simply swapping the order of two candidate responses in a pairwise comparison can shift accuracy by more than 10%. Verbosity bias leads judges to prefer longer, more formally structured responses even when shorter responses are more accurate or more appropriate. Self-preference bias causes models to assign higher scores to outputs that resemble their own generation patterns. And superficiality bias causes judges to overweight surface fluency at the expense of factual accuracy or adherence to instructions.</p><p>These biases are not theoretical curiosities. They are operational risks that can systematically distort your evaluation results. If your LLM judge prefers verbose responses, and you use that judge to compare model versions, you may inadvertently select the model that produces longer but less accurate outputs. If your judge exhibits position bias, you may draw incorrect conclusions from pairwise comparisons that would reverse with a different presentation order.</p><h4>How to use LLM judges responsibly</h4><p>The prescription is not to abandon LLM judges but to use them with appropriate discipline. Several practices separate responsible use from reckless reliance.</p><p>First, validate your judge against human labels before trusting it at scale. Build a validation set of 100 to 200 examples with human-generated quality scores, and measure agreement between your LLM judge and those human scores. If agreement is below 75% on the dimensions you care about, your judge is not ready for unsupervised use. Refine the rubric, adjust the prompt, or consider a different judge model before proceeding.</p><p>Second, match the judge to the task. LLM judges work best on structured, well-defined evaluation dimensions such as relevance, groundedness, tone adherence, and format compliance. They work poorly on ambiguous, subjective, or high-stakes judgments where reasonable evaluators would disagree. A practical heuristic is to ask whether you could train a competent human evaluator to apply your rubric consistently in under ten minutes. If the answer is yes, an LLM judge will likely perform well. If the answer is no, you need human evaluation for that dimension.</p><p>Third, design tight rubrics. Vague evaluation criteria produce unreliable results from both human and automated judges, but LLMs are particularly sensitive to rubric ambiguity. Instead of asking a judge to assess whether a response is &#8220;good,&#8221; specify the observable characteristics that constitute quality for your use case. Define what a score of 1, 3, and 5 looks like with concrete examples. The tighter the rubric, the more reliable the judge.</p><p>Fourth, implement mitigation strategies for known biases. Randomize response order in pairwise comparisons. Consider running evaluations twice with swapped positions and flagging cases where the judge&#8217;s preference reverses. Use ensemble approaches in which multiple judge instances evaluate the same output, and their scores are aggregated. These techniques add computational cost but substantially improve reliability. Research on ensemble-based &#8220;jury&#8221; models has shown that averaging scores across multiple judge instances can significantly reduce the impact of individual model biases. However, practitioners must weigh this benefit against the additional inference costs.</p><p>Fifth, maintain human oversight for high-stakes decisions. Even the best-configured LLM judge should not be the sole arbiter for deployment decisions, model selection for safety-critical applications, or evaluation of outputs that could cause material harm. Use automated judges to triage and prioritize, then route the most consequential decisions to qualified human reviewers. One data scientist described a practical implementation of this principle. His team automatically reruns evaluations that produce low scores, then routes confirmed failures to human review. This two-stage approach catches the cases where the judge itself hallucinated a low score, while still surfacing genuine quality issues for expert assessment.</p><h4>The cost calculus</h4><p>At scale, the economics of LLM-as-a-judge are compelling. Organizations running 10,000 or more monthly evaluations can realize substantial cost savings compared to fully human review while maintaining agreement rates that are statistically comparable to human-to-human consistency. But the savings only materialize if the judge is properly calibrated. An uncalibrated judge that produces plausible-looking but inaccurate scores does not save money. It creates technical debt in the form of false confidence that will eventually come due, typically at the worst possible moment.</p><h2>The Real Goal: A Repeatable Evaluation Loop</h2><p>Everything described above culminates in a single objective. Evaluation should not be a one-time checkpoint. It should be a repeatable, automated system that runs continuously as prompts change, models update, and product requirements evolve.</p><h4>Why repeatability matters</h4><p>LLM-powered systems are not static. Prompts get refined. Models are updated or swapped entirely. Retrieval pipelines are restructured. Context windows are adjusted. Each of these changes has the potential to introduce regressions that are invisible without systematic evaluation. A system that passed all tests last month may fail silently this month because a prompt change improved performance on the primary use case while degrading performance on an edge case that your users encounter daily.</p><p>This is fundamentally different from traditional software testing. When you update a conventional application, the change in behavior is deterministic. You can trace the code path and predict the impact. With LLMs, a seemingly minor prompt modification can produce cascading changes across the entire output distribution. Adding a single sentence to a system prompt might improve accuracy on one category of questions while introducing hallucinations in another. Swapping from one model version to another might improve reasoning quality while changing the tone in ways that conflict with your brand guidelines. Without a comprehensive evaluation across all the dimensions you care about, these tradeoffs remain invisible until users discover them.</p><p>The companies that execute this well have built evaluation into their development workflow, just as software teams build automated testing into their CI/CD pipelines. Every change to a prompt, a retrieval strategy, or a model version triggers a suite of evaluations that compare the new configuration against the previous one across all relevant dimensions. Regressions are flagged before they reach production, not after.</p><h4>Components of a mature evaluation loop</h4><p>A production-grade evaluation system includes several integrated components.</p><p>A versioned test set that is maintained alongside your application code, with clear processes for adding new cases, retiring outdated ones, and tracking coverage over time. Version control for your test set is just as important as version control for your code, because you need to understand not just how your system performs today, but how that performance has changed relative to previous evaluation runs.</p><p>An automated evaluation pipeline that can run on demand or on a schedule, producing standardized metrics that are comparable across runs. This pipeline should support multiple evaluation methods, including deterministic checks for format and structure, LLM-based judges for quality dimensions, and integration points for human review when automated methods are insufficient.</p><p>A dashboard or reporting mechanism that surfaces evaluation results to the people who need to act on them. Engineering teams need granular, dimension-level scores. Product managers need trend lines and regression alerts. Executives need summary metrics that convey overall system health. The same evaluation data should serve all three audiences at appropriate levels of abstraction.</p><p>A feedback loop that connects production monitoring to test set maintenance. When production observability surfaces new failure modes, input distribution drift, or degradation in user satisfaction metrics, those signals should flow into the evaluation pipeline. This closes the loop between what your system is doing in the real world and what you are testing for in your evaluation environment.</p><h4>The organizational dimension</h4><p>Technology alone does not create evaluation discipline. Someone needs to own the evaluation pipeline with the same seriousness as they do the production infrastructure. In many organizations, this responsibility falls between teams. Data scientists build the initial test set and move on. Engineers maintain the pipeline but do not curate the data. Product managers track user satisfaction but do not connect it back to evaluation metrics.</p><p>The organizations getting this right assign clear accountability for evaluation quality, invest in tooling that reduces friction, and treat evaluation coverage as a metric that is tracked and reported alongside system performance. They recognize that evaluation is not a tax on development speed. It is the mechanism that enables confident, rapid iteration. Teams that can evaluate quickly can ship quickly, because they have the evidence to know that their changes are improvements rather than regressions.</p><h2>From Evaluation to Evidence</h2><p>LLM evaluation is not an isolated technical practice. It is a critical component of the broader governance and observability infrastructure that enables safe, responsible AI at scale.</p><p>The governance frameworks establish what AI systems should do. They define acceptable use, risk thresholds, and accountability structures. But governance policies alone cannot tell you whether your system is meeting those standards in practice. That requires continuous, rigorous, production-aware evaluation. </p><p>Think about how this relates to the core principles of Trusted AI. Transparency requires that you can explain how your system behaves and what quality it delivers. A repeatable evaluation loop provides the evidence for that explanation. Accountability requires that someone is responsible for system outcomes. Clear evaluation metrics make accountability measurable rather than aspirational. Fairness requires that your system not produce biased or discriminatory outputs. Evaluation test sets designed to detect disparate outcomes make fairness auditable rather than assumed.</p><p>The EU AI Act&#8217;s requirements for high-risk AI systems include provisions for testing, validation, and ongoing monitoring that map directly to the evaluation practices described here. Organizations that build mature evaluation capabilities now will be positioned to demonstrate compliance as requirements take effect. Those who wait will find themselves retrofitting evaluation into systems that were designed without it, which is significantly more difficult and more expensive.</p><p>Beyond regulatory compliance, evaluation discipline connects directly to the business case for Trusted AI that we have explored throughout this publication. Organizations with mature AI governance report significant improvements in both regulatory compliance and stakeholder trust. Evaluation is the mechanism through which governance commitments become verifiable claims. When a board member asks whether your AI systems are performing as intended, when a regulator asks for evidence of ongoing monitoring, or when a customer asks how you ensure quality, the answer is your evaluation pipeline. Either you have one that produces credible evidence, or you have assertions that you cannot substantiate.</p><h2>Start Now, Start Small, Stay Disciplined</h2><p>If your organization does not yet have a structured LLM evaluation practice, here is where to begin.</p><p>Identify your highest-risk LLM deployment and build a golden set of 30 to 50 real-world test cases for it, including examples of known failures and edge cases. Label those cases with qualified reviewers who understand the domain and the downstream consequences of errors. Set up an automated evaluation pipeline, even a simple one, that can run those cases against your current system and produce a baseline score. Then commit to running that evaluation every time you make a change.</p><p>From that foundation, you can add synthetic data for coverage, LLM judges for scale, and production feedback loops for continuous improvement. But the foundation matters most. A small, high-quality, well-maintained test set that runs on every change will do more for your system&#8217;s reliability than a thousand synthetic examples that sit in a folder and are never revisited.</p><p>The teams that earn trust with AI in production are the ones that can show their work. Evaluation is how you show your work. Build that capability now, and you will be positioned not just to deploy AI, but to deploy AI that your customers, your regulators, and your own leadership can trust.</p><p>It&#8217;s not a question of whether your AI systems need rigorous evaluation. It&#8217;s whether you will build that discipline before or after your first production failure.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/why-most-llm-evaluations-fail-before/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/why-most-llm-evaluations-fail-before/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[The Wrong Scorecard: Why Most Organizations Are Failing at AI Evaluation]]></title><description><![CDATA[A decision framework for investing evaluation resources where they matter most]]></description><link>https://trustedai.recodework.com/p/the-wrong-scorecard-why-most-organizations</link><guid isPermaLink="false">https://trustedai.recodework.com/p/the-wrong-scorecard-why-most-organizations</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Sun, 19 Apr 2026 18:21:55 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-6RQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-6RQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-6RQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg 424w, https://substackcdn.com/image/fetch/$s_!-6RQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg 848w, https://substackcdn.com/image/fetch/$s_!-6RQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!-6RQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!-6RQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg" width="1456" height="1032" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1032,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:4762053,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/194654454?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!-6RQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg 424w, https://substackcdn.com/image/fetch/$s_!-6RQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg 848w, https://substackcdn.com/image/fetch/$s_!-6RQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!-6RQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a1598ce-91fc-4bfa-9d50-da4da2375c15_4921x3489.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Every enterprise AI team eventually arrives at the same uncomfortable realization. The model that looked brilliant in the demo is behaving unpredictably in production. Customer complaints are rising, but nobody can pinpoint exactly what is going wrong. The dashboard shows green across the board, but the metrics being tracked have nothing to do with the outcomes that actually matter.</p><p>This is the evaluation gap, one of the most consequential blind spots in enterprise AI today.</p><p>Evaluation is the connective tissue between governance and operational reality. Trusted AI requires both governance frameworks and continuous observability, and evaluation metrics are where those two imperatives converge. They translate abstract principles like fairness, transparency, and reliability into measurable signals that tell you whether your AI systems are performing as intended or drifting toward failure.</p><p>Yet most organizations approach AI evaluation with the same generic metrics, regardless of the use case. They track accuracy or F1 scores during development, declare victory, and move on. This is roughly equivalent to evaluating every employee in your organization with the same performance review, from the CFO to the front-line service representative. The measurements might be valid in isolation, but they are disconnected from what actually matters for each role.</p><p>The reality is that different AI use cases demand fundamentally different evaluation strategies. A customer support chatbot, a retrieval-augmented search system, an agentic workflow that executes multi-step tasks, a classification model that routes insurance claims, and a summarization engine that condenses legal documents all present distinct risk profiles and performance characteristics. Evaluating them with the same toolkit is not just inefficient. It is dangerous because it creates false confidence, obscuring real problems.</p><p>This post provides a practical framework for matching evaluation metrics to use cases. Think of it as a metric toolbox, organized around the dimensions that matter most for enterprise AI, and mapped to the use-case patterns that dominate real-world deployments. The goal is not academic completeness. It is decision-oriented clarity that helps you invest evaluation resources where they will have the greatest impact.</p><h2>The Six Dimensions of AI Evaluation</h2><p>Before mapping metrics to use cases, we need a shared vocabulary for what we are measuring. Enterprise AI evaluation spans six core dimensions, each capturing a different aspect of system performance and trustworthiness.</p><h4>1. Relevance</h4><p>Relevance measures whether an AI system's output actually addresses the question asked, and relevance failures are among the most common complaints from end users. A support chatbot that provides a technically accurate answer to the wrong question has failed on relevance, even though its content is correct. Relevance metrics evaluate the alignment between what a user needs and what the system delivers.</p><h4>2. Correctness</h4><p>Correctness assesses whether the AI system's output is factually accurate and faithful to its source material. In traditional machine learning, this maps to familiar concepts like accuracy and precision. In generative AI, correctness takes on additional dimensions. A large language model can produce fluent, confident, and entirely fabricated responses. Correctness metrics in the generative context must evaluate whether outputs are grounded in retrieved evidence, whether source documents support claims, and whether the system avoids hallucination.</p><h4>3. Safety</h4><p>Safety encompasses the guardrails that prevent AI systems from producing harmful, toxic, or policy-violating outputs. Safety metrics monitor for content that could damage customers, expose the organization to liability, or violate regulatory requirements. This dimension has become increasingly critical as generative AI systems interact directly with customers and the public.</p><h4>4. Bias</h4><p>Bias assesses whether an AI system's outputs or decisions exhibit systematic unfairness across demographic groups or protected characteristics. Bias metrics are essential for any AI application that influences decisions affecting individuals, from hiring and credit to healthcare recommendations and insurance underwriting. The EU AI Act and emerging regulations globally are making bias evaluation a compliance requirement, not merely a best practice.</p><h4>5. Latency</h4><p>Latency measures the time an AI system takes to produce its output. In many enterprise contexts, a technically perfect response that arrives too slowly is functionally useless. Customer support interactions have sub-second expectations. Real-time fraud detection systems operate in milliseconds. Even internal knowledge search loses adoption rapidly if retrieval takes more than a few seconds. Latency is not just a technical concern but a direct driver of user satisfaction and business value.</p><h4>6. Cost</h4><p>Cost tracks the computational and financial resources consumed by AI operations. With large language model inference costs varying by orders of magnitude depending on model selection, prompt design, and architecture choices, cost has become a first-class evaluation dimension. Organizations that ignore cost efficiency in their evaluation frameworks often discover that their AI systems are economically unsustainable at production scale, even when they perform well on every other dimension.</p><p>These six dimensions are not equally important for every use case. That is precisely the point. The art of AI evaluation lies in understanding which dimensions are primary for your specific application and which can tolerate more flexibility.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>Five Use-Case Patterns and Their Evaluation Priorities</h2><p>Most enterprise AI deployments fall into a handful of recurring patterns. While every implementation has unique characteristics, the evaluation priorities cluster in predictable ways. The following five patterns cover the vast majority of production AI systems and provide a foundation for building your evaluation strategy.</p><h4>Pattern 1: Conversational AI and Customer Support</h4><p>Conversational AI systems, including customer-facing chatbots, virtual assistants, and AI-powered support agents, represent perhaps the highest-visibility AI deployment pattern in the enterprise. These systems interact directly with customers, often handling thousands of conversations daily. When they fail, customers notice immediately, and the consequences are both measurable and public.</p><p>A fundamental tension shapes the evaluation priorities for conversational AI. These systems need to be helpful and responsive while simultaneously avoiding harmful, inaccurate, or off-brand responses. A support chatbot that provides wrong billing information can trigger regulatory complaints. One that generates toxic or inappropriate content can go viral on social media. And one that takes too long to respond drives customers to abandon the interaction entirely.</p><p>For this pattern, <strong>relevance</strong> and <strong>safety</strong> are primary metrics. Relevance in conversational AI is measured through answer relevance scoring, which evaluates whether the response addresses the user's actual intent, and through resolution rate, which tracks whether the interaction successfully resolved the customer&#8217;s issue. Industry benchmarks show that leading implementations achieve resolution rates above 65% for routine inquiries without human escalation. Customer satisfaction scores provide a lagging but essential validation layer, with live chat AI implementations averaging around 87% positive CSAT ratings according to recent industry data.</p><p>Safety evaluation in conversational AI requires monitoring for toxicity, policy violations, and brand-inconsistent responses. This means running every production output against content safety classifiers that flag harmful language, testing for prompt-injection vulnerabilities in which users attempt to manipulate the system into inappropriate behavior, and monitoring for data leakage in which the system inadvertently reveals confidential information.</p><p><strong>Correctness</strong> is a critical secondary metric. In support contexts, incorrect answers are often worse than no answer at all. The Air Canada case, in which a chatbot provided false refund information, and the company was held legally liable for the output, illustrates the stakes. Correctness evaluation should include factual accuracy checks against the organization's knowledge base and faithfulness scoring to verify that responses are grounded in approved content rather than generated from the model's data.</p><p><strong>Latency</strong> is also a secondary but important metric. Research consistently shows that customers expect initial responses within five seconds, and resolution times under two minutes are the benchmark for leading implementations. Latency monitoring should track both time-to-first-token for perceived responsiveness and total interaction duration.</p><p><strong>Bias</strong> and <strong>cost</strong> are important, but typically fall under optional monitoring for this pattern. Bias matters particularly in support contexts, where AI systems may inadvertently provide responses of varying quality based on customer characteristics, language patterns, or geographic indicators. Cost per interaction is a key business metric, but it rarely drives technical evaluation decisions in isolation.</p><h4>Pattern 2: Retrieval-Augmented Generation (RAG) and Knowledge Search</h4><p>RAG systems combine information retrieval with generative AI to answer questions grounded in organizational knowledge. They power internal knowledge bases, customer-facing documentation search, legal research tools, and enterprise question-answering applications. The distinctive characteristic of RAG is its two-stage architecture. A retriever finds relevant documents, and a generator produces a response based on those documents.</p><p>This architectural duality creates a unique evaluation challenge. Failures can originate in either stage, and diagnosing problems requires metrics that isolate retrieval quality from generation quality. A system might retrieve the perfect documents but generate a response that ignores or misrepresents them. Alternatively, the generator might perform flawlessly, but the retriever might surface irrelevant or incomplete context, leaving the generator unable to produce a useful answer.</p><p>For RAG systems, <strong>correctness</strong> is the primary, unambiguous metric, and it must be decomposed into its constituent parts. Faithfulness measures whether the generated response is factually consistent with the retrieved context. It is calculated by identifying the individual claims in the response and checking whether the retrieved documents support each claim. Research from StaStanford's Lab indicates that poorly evaluated RAG systems can produce hallucinations in up to 40% of responses, even when they access correct information, making faithfulness evaluation non-negotiable for production deployments.</p><p><strong>Relevance</strong> is the other primary dimension, and it applies to both stages of the pipeline. Context relevance (sometimes called context precision) assesses whether the retriever surfaces documents that are actually pertinent to the query. Answer relevance evaluates whether the generated response addresses the user's question. Context recall measures whether all the information needed to answer the query was retrieved. Together, these metrics form what practitioners call the RAG Triad, a framework for evaluating the three critical relationships in a RAG system between the query, the retrieved context, and the generated response.</p><p><strong>Safety</strong> serves as a critical secondary metric for RAG, particularly in regulated industries where the system may surface sensitive or confidential information. RAG-specific safety concerns include data leakage through retrieved context, responses that combine information from documents with different access levels, and hallucinated content that appears authoritative when presented alongside retrieved evidence.</p><p><strong>Latency</strong> matters significantly for RAG because the retrieval step adds a measurable delay. Total response time includes embedding the query, searching the vector database, re-ranking results, and generating the response. Each component should be monitored independently so that performance degradation can be attributed to the right stage. <strong>Cost</strong> is also worth tracking because RAG systems involve both retrieval infrastructure and generation costs, which scale differently with usage patterns.</p><p><strong>Bias</strong> is optional in most RAG deployments, but becomes important when the underlying knowledge base reflects historical biases or when the system is used to make decisions that affect individuals.</p><h4>Pattern 3: Agentic Workflows</h4><p>Agentic AI represents the newest and most complex deployment pattern. These systems go beyond generating text to actually performing multi-step tasks. They reason about what needs to be done, select and invoke tools, interpret intermediate results, and adapt their approach based on outcomes. Examples include AI agents that process insurance claims by gathering documents, validating information, and making routing decisions. Or agents that conduct competitive research by searching multiple sources, synthesizing findings, and producing structured reports. Or agents that manage IT operations by diagnosing issues, executing remediation steps, and verifying resolution.</p><p>The evaluation challenge for agentic systems is fundamentally different from simpler patterns because the system's behavior is non-deterministic and multi-step. A single task execution may involve dozens of decisions, tool calls, and intermediate outputs. Traditional metrics that evaluate only the final output miss critical failure modes in the reasoning chain. An agent might produce the correct final answer through an unsafe or unreliable process, or it might complete a task that looks correct but is actually the wrong task.</p><p>Recent research underscores this problem. A 2025 study evaluating enterprise agentic systems found that agent performance dropped from 60% on single-run evaluations to just 25% when measured across eight consistent runs, revealing massive reliability gaps that single-execution metrics completely miss. The same study found cost variations of up to 50x for similar accuracy levels across different agent implementations, demonstrating that accuracy-only evaluation produces dangerously incomplete assessments.</p><p>For agentic workflows, <strong>correctness</strong> and <strong>safety</strong> share primary importance. Correctness in the agentic context means task completion, the question of whether the agent accomplished the intended objective, combined with task adherence, which verifies the agent performed the right task and not just any task. Tool call accuracy measures whether the agent selected the appropriate tools and invoked them with correct parameters. Reasoning quality assesses whether the agent's intermediate reasoning steps were sound, not just whether the final output was acceptable.</p><p>Safety is uniquely critical for agentic systems because these are AI applications that take actions, not just produce text. An agent that executes the wrong API call, modifies production data incorrectly, or bypasses approval workflows can cause immediate operational harm. Safety evaluation must include guardrail compliance monitoring, permission boundary testing, and verification that the agent respects human oversight requirements. Amazon's valuation framework for its internal agentic systems operates across three layers, assessing the underlying model performance, the behavior of individual components, including tool use and reasoning, and the overall quality of task completion.</p><p><strong>Latency</strong> and <strong>cost</strong> are important secondary metrics for agentic workflows. Agents that take excessive time to complete tasks or consume disproportionate computational resources may be technically correct but operationally impractical. Cost-per-successful-task is emerging as a standard efficiency metric, combining accuracy with resource consumption into a single actionable measure.</p><p><strong>Relevance</strong> applies at the task level, ensuring the agent correctly interprets the user's intent. <strong>Bias</strong> is important when agents make decisions that affect individuals, but it is typically less prominent than in classification or support patterns.</p><h4>Pattern 4: Classification and Decision Support</h4><p>Classification models are the workhorses of enterprise AI. They route customer inquiries, flag fraudulent transactions, triage medical records, categorize compliance documents, score credit applications, and sort insurance claims. While they may lack the glamour of generative AI, classification systems often have the greatest impact because their outputs directly drive automated decisions or heavily influence human decision-makers.</p><p>The evaluation landscape for classification is more mature than for generative patterns, but maturity can breed complacency. Many organizations conduct evaluations during model development and then treat it as a solved problem, overlooking the inevitable performance degradation that occurs as production data drifts from the training distribution.</p><p>For classification, <strong>correctness</strong> is the dominant primary metric, but it must be measured with appropriate granularity. Overall accuracy is often misleading, particularly for imbalanced classes. A fraud detection model that achieves 99% accuracy by simply labeling everything as legitimate has learned nothing useful. Precision, recall, and F1 score broken down by class provide far more actionable insight. The specific balance between precision and recall should reflect business priorities. In fraud detection, high recall matters because missing a fraudulent transaction is costly. In spam filtering, high precision matters because false positives that block legitimate communications damage trust.</p><p><strong>Bias</strong> rises to primary importance for classification systems more than for any other pattern. This is because classification outputs frequently drive decisions that affect individuals, placing them squarely in the regulatory crosshairs. The EU AI Act classifies AI systems used in credit scoring, hiring, and insurance underwriting as high-risk, imposing stringent requirements for bias monitoring and documentation. Bias evaluation should examine model performance across protected characteristics, including race, gender, age, and geography, testing for both disparate impact in outcomes and disparate treatment in the decision process.</p><p><strong>Latency</strong> is a key secondary metric, particularly for real-time classification applications such as fraud detection or content moderation, where decisions must be made within milliseconds. Batch classification of documents or claims has more relaxed latency requirements, but throughput (the volume of classifications per unit of time) becomes the relevant operational metric.</p><p><strong>Cost</strong> matters as a secondary concern because classification models typically run at much lower per-inference costs than generative models. However, the high volume of classifications can still produce high aggregate costs.</p><p><strong>Relevance</strong> and <strong>safety</strong> are generally optional for classification patterns. The concept of relevance is subsumed into correctness for classification. Safety considerations are typically addressed through the bias dimension and business rules governing how classification outputs are used in downstream processes.</p><h4>Pattern 5: Summarization and Content Generation</h4><p>Summarization systems condense lengthy documents, transcripts, reports, and data sets into digestible formats. They generate executive briefings from earnings calls, produce clinical summaries from medical records, create meeting recaps from transcription data, and distill regulatory filings into actionable intelligence. Content generation extends this pattern to include drafting communications, producing reports from structured data, and creating documentation.</p><p>The unique evaluation challenge for summarization is that there is no single correct answer. A good summary is faithful to the source material, captures the most important information, avoids introducing unsupported claims, and presents content in a coherent structure. Multiple valid summaries can exist for the same source document, making evaluation inherently more subjective than for classification or factual question-answering.</p><p>For summarization, <strong>correctness</strong> is the primary metric, but it manifests differently than in other patterns. Faithfulness (sometimes called groundedness) verifies that the source material supports every claim in the summary. This is the most critical correctness dimension because a summary that introduces fabricated information is worse than no summary at all, particularly in high-stakes contexts like medical records or legal documents. Factual consistency scoring compares the claims in the summary against the source and flags any unsupported assertions.</p><p><strong>Relevance</strong> is the other primary metric, evaluated through coverage and information density. Coverage measures whether the summary captures the key points from the source material. Information density assesses whether the summary is appropriately concise or contains unnecessary filler. Together, these metrics answer whether the summary tells the reader what they most need to know.</p><p><strong>Safety</strong> functions as a secondary metric, primarily relevant when summaries are generated from sensitive source material or when they will be distributed to external audiences. Content safety checks should verify that summaries do not inadvertently expose confidential information, reproduce copyrighted content, or present information in misleading ways.</p><p><strong>Bias</strong> is a secondary consideration that becomes important when summarization systems process content about people or groups. A summarization system that systematically emphasizes negative information about certain demographics or downplays contributions from underrepresented groups introduces bias even if it is technically &#8220;accurate.&#8221;</p><p><strong>L&#8221;tency</strong> and <strong>cost</strong> are typically optional considerations for summarization, which often runs as a batch process rather than in real time. However, interactive summarization applications where users expect near-instant results, such as meeting recap tools, make latency a meaningful consideration.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/the-wrong-scorecard-why-most-organizations?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/the-wrong-scorecard-why-most-organizations?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>The Evaluation Matrix: Putting It All Together</h2><p>The following matrix synthesizes the analysis above into a decision framework. For each use-case pattern, metrics are classified as Primary (must measure; invest in automation), Secondary (should measure; review regularly), or Optional (measure if resources allow; monitor periodically).</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!57hC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!57hC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png 424w, https://substackcdn.com/image/fetch/$s_!57hC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png 848w, https://substackcdn.com/image/fetch/$s_!57hC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png 1272w, https://substackcdn.com/image/fetch/$s_!57hC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!57hC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png" width="970" height="312" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:312,&quot;width&quot;:970,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:34984,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/194654454?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!57hC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png 424w, https://substackcdn.com/image/fetch/$s_!57hC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png 848w, https://substackcdn.com/image/fetch/$s_!57hC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png 1272w, https://substackcdn.com/image/fetch/$s_!57hC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3b0076b-9bb0-4a22-93f0-d1a22c7780b0_970x312.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This matrix is a starting point, not a prescription. Your specific implementation context will shift priorities. A customer support chatbot in healthcare will elevate bias and correctness to primary status. A classification model used only for internal document routing may downgrade bias to optional. An agentic workflow operating in a regulated environment will treat every dimension as primary.</p><p>The key insight is directional. Not every metric deserves equal investment for every use case. Organizations that spread their evaluation resources uniformly across all dimensions for all systems end up with shallow coverage everywhere and deep coverage nowhere. The matrix helps you concentrate your evaluation investment where it matters most.</p><h2>From Metrics to Operational Practice</h2><p>Selecting the right metrics is necessary but not sufficient. The metrics must be operationalized through continuous monitoring, clear thresholds, and defined response procedures. Several principles guide this operationalization.</p><p>First, establish baselines before optimizing. You cannot set meaningful thresholds without understanding your system's current performance. Run your evaluation metrics against production data for a sufficient period to establish baseline distributions, then set alert thresholds based on statistically significant deviations from those baselines rather than arbitrary targets. A faithfulness score of 0.85 might be excellent for one RAG system and dangerously low for another, depending on the domain and the consequences of error. Baselines grounded in your actual production data provide the context that universal benchmarks cannot.</p><p>Second, automate evaluation wherever possible. Manual evaluation processes create bottlenecks, introduce inconsistency, and degrade over time as teams face competing priorities. Modern evaluation frameworks such as Ragas and DeepEval, along with purpose-built platforms from providers like Arize and Confident AI, enable automated evaluation pipelines that run continuously on production traffic. Integrate these into your CI/CD pipelines so that model updates are automatically evaluated before deployment. The most mature organizations treat evaluation gates with the same seriousness as security scans. No model change reaches production without passing defined evaluation thresholds.</p><p>Third, combine automated and human evaluation. Automated metrics provide scale and consistency, but they have known limitations. LLM-based evaluators, which use one language model to judge another's output, can disagree significantly with each other and with human assessments. Research has shown that different evaluator models can produce faithfulness scores ranging from 0% to over 80% on the same set of responses. Human evaluation remains essential for calibrating automated metrics, validating edge cases, and assessing subjective quality dimensions that automated tools struggle to capture. The practical recommendation is to run automated evaluation on 100% of production traffic and layer human evaluation on a statistically meaningful sample, focusing human attention on disagreements between automated evaluators and on the long-tail failure modes that automated systems tend to miss.</p><p>Fourth, monitor for drift continuously. AI system performance degrades over time as data distributions shift, user behaviors evolve, and the world changes. A model evaluated at deployment is not the same model six months later in terms of effective performance. Your evaluation framework must include ongoing drift detection that alerts you when performance metrics deviate from established baselines. This is particularly critical for classification systems where concept drift, a change in the underlying relationship between inputs and outcomes, can render a model progressively less accurate without any visible change in the input data itself. Seasonal patterns, regulatory changes, shifts in customer demographics, and competitive dynamics all contribute to drift that only continuous monitoring can detect.</p><p>Fifth, connect evaluation to governance. Evaluation metrics should feed directly into your AI governance reporting. When the AI Council or Risk Committee reviews AI system performance, they should see evaluation data that maps to the risk dimensions they care about. This connection ensures that evaluation is not an isolated technical activity but an integral component of organizational AI oversight. The evaluation matrix presented earlier provides a natural mapping between technical metrics and governance concerns. Faithfulness scores map to transparency and explainability. Bias metrics map to fairness requirements. Safety scores map to risk mitigation. Presenting evaluation data through this governance lens makes it actionable for the senior leaders who need to make decisions about AI program investments and risk appetite.</p><p>Finally, build evaluation feedback loops that drive improvement. Evaluation is not merely an audit function. The most valuable evaluation programs create tight feedback loops between metric findings and system improvements. When faithfulness scores decline in a RAG system, the evaluation data should indicate whether the problem stems from retrieval or generation quality, enabling targeted remediation. When a classification model shows emerging bias, the evaluation data should identify which segments are affected and whether the root cause is data drift or model architecture. Without these diagnostic loops, evaluation becomes a reporting exercise rather than a capability improvement engine.</p><h2>Getting Started: A Practical Sequence</h2><p>For organizations that have not yet built systematic evaluation capabilities, the sheer breadth of metrics can feel paralyzing. The temptation is either to measure everything superficially or to measure nothing at all. Neither approach serves you well. A practical starting sequence can help you build evaluation capabilities incrementally while delivering value at each stage.</p><p>Start with your highest-risk AI system. Identify the single AI application that carries the greatest potential for customer harm, regulatory exposure, or financial impact. This is where investment in evaluation will yield the greatest return in risk reduction. Use the matrix above to identify the primary metrics for that system&#8217;s se-case pattern, and build an automated evaluation for those dimensions first.</p><p>Next, establish your evaluation infrastructure. The tooling decisions you make for your first system will shape your evaluation capabilities for everything that follows. Invest in evaluation infrastructure that supports multiple use-case patterns rather than building point solutions for individual systems. Open-source frameworks like Ragas for RAG evaluation and DeepEval for broader LLM evaluation provide flexible foundations. Commercial platforms from Arize, Confident AI, and others offer more comprehensive capabilities, including production monitoring, drift detection, and integrations with governance reporting.</p><p>Then, extend evaluation to your broader AI portfolio. Once you have infrastructure and experience from your first system, systematically evaluate your remaining AI applications. Prioritize by risk tier, applying the most rigorous evaluation to high-risk systems and lighter-touch monitoring to lower-risk applications. This mirrors the risk-based governance approach recommended by both the EU AI Act and the NIST AI RMF.</p><p>Finally, build the organizational muscle for continuous improvement. Evaluation is not a one-time project. It is an ongoing operational discipline. Staff it accordingly. Train your teams on evaluation methodologies. Create feedback loops between evaluation findings and model improvement activities. And report evaluation results to governance forums with the same regularity and rigor you bring to financial reporting.</p><h2>The Metrics You Choose Reveal the Risks You Take Seriously</h2><p>Every evaluation strategy embodies a set of priorities, and by extension, a set of trade-offs. The metrics you choose to measure reflect the risks you consider most important. The metrics you choose not to measure represent risks you are implicitly accepting.</p><p>This is why the evaluation strategy should not be delegated entirely to technical teams. It requires input from business stakeholders who understand customer impact, from legal and compliance teams who understand regulatory exposure, from risk management professionals who understand organizational risk appetite, and from ethics advisors who understand the broader societal implications of AI deployment.</p><p>The organizations building the most resilient AI programs treat evaluation as a strategic capability, not a technical afterthought. They invest in evaluation infrastructure with the same seriousness they bring to model development. They staff the evaluation teams with the same talent they recruit for model building. And they hold themselves accountable to evaluation results with the same rigor they apply to financial metrics.</p><p>The metric toolbox presented here gives you a starting framework. The next step is yours. Audit your current AI systems against these dimensions. Identify the gaps between what you are measuring and what you should be measuring. Prioritize investments based on the risk profile of each use case. And build the evaluation muscle that will sustain Trusted AI as your AI portfolio grows.</p><p>Because in the end, the question is not whether your AI systems will face evaluation challenges. The question is whether you will discover those challenges through systematic measurement or through the far more expensive mechanism of customer complaints, regulatory findings, and public failures.</p><p>Measure what matters. Measure it continuously. And let the measurements drive action.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/the-wrong-scorecard-why-most-organizations/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/the-wrong-scorecard-why-most-organizations/comments"><span>Leave a comment</span></a></p>]]></content:encoded></item><item><title><![CDATA[Your AI is Running. But is It Working?]]></title><description><![CDATA[The observability stack that makes AI systems debuggable, auditable, and improvable]]></description><link>https://trustedai.recodework.com/p/your-ai-is-running-but-is-it-working</link><guid isPermaLink="false">https://trustedai.recodework.com/p/your-ai-is-running-but-is-it-working</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Thu, 09 Apr 2026 17:36:37 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!WpOW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!WpOW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!WpOW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg 424w, https://substackcdn.com/image/fetch/$s_!WpOW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg 848w, https://substackcdn.com/image/fetch/$s_!WpOW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!WpOW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!WpOW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:3416116,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/193709407?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!WpOW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg 424w, https://substackcdn.com/image/fetch/$s_!WpOW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg 848w, https://substackcdn.com/image/fetch/$s_!WpOW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!WpOW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a274845-d5c6-4500-80c3-cfa06a0836c9_4043x2275.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><p>You can monitor a traditional web application with dashboards that track error rates, latency percentiles, and throughput. When something breaks, you read the logs, find the stack trace, and fix the bug. The same input produces the same output. The system is deterministic. Debugging is hard, but the problem space is well understood.</p><p>AI systems break all of these assumptions.</p><p>The same prompt can produce different outputs on consecutive runs. A model that performed well last month may silently degrade as the data it encounters in production diverges from its training distribution. An agent that chains together retrieval, tool calls, and generation steps can fail at any point in the sequence, and the failure may not even look like an error. It may look like a confident, well-structured response that happens to be wrong.</p><p>This is why traditional application monitoring is necessary but insufficient for AI systems. You still need infrastructure metrics, error tracking, and latency measurement. But you also need a fundamentally different observability layer, one that captures the semantic content of AI interactions, the quality of outputs, the provenance of context, and the decisions made along the way.</p><p>This post examines what that observability stack looks like in practice. What to log, how to structure telemetry, how to capture feedback and evaluation signals, and how to do all of this without turning your logging pipeline into a privacy and compliance liability.</p><h2>Why AI Observability Is Different</h2><p>To understand why AI systems demand a different approach to observability, consider the specific properties that distinguish them from traditional software.</p><h4>Non-deterministic outputs</h4><p>Even with the temperature set to zero, model outputs can vary across providers, versions, and infrastructure configurations. Two requests with identical prompts may return substantively different responses. This means you cannot rely on simple assertions or on expected output matching what you would with a conventional API. Quality evaluation requires probabilistic assessment, not binary pass/fail checks.</p><h4>Prompt sensitivity</h4><p>Small changes to a prompt, sometimes a single word, can produce dramatically different results. A system prompt modification intended to improve formatting may inadvertently degrade factual accuracy. Without logging prompt versions alongside outputs, teams cannot correlate quality regressions to the changes that caused them.</p><h4>Model and data drift</h4><p>AI models do not remain static after deployment. Foundation models get updated by their providers. Fine-tuned models degrade when the distribution of production data shifts away from that of the training data. Retrieval-augmented generation systems depend on the quality and freshness of their knowledge bases, which are continuously updated. Traditional monitoring will tell you the system is running. It will not tell you the system is returning worse answers than it was a month ago.</p><h4>Tool use and multi-step reasoning</h4><p>Modern AI architectures increasingly involve agents that select and execute tools, retrieve context from external sources, and chain multiple inference calls together before producing a final response. A failure anywhere in this chain can compromise the output, but the failure may be invisible if you only inspect the final response. An agent might retrieve irrelevant documents, select the wrong tool, or misinterpret the tool's output, yet still produce a fluent, confident answer.</p><h4>Harder debugging</h4><p>When a traditional application returns an incorrect result, you can typically trace the logic path through deterministic code. When an AI system returns a hallucinated answer, the &#8220;bug&#8221; may reside in the prompt, the retrieved context, the model weights, the system instructions, or the interaction between all of these. Reproducing the issue often requires the exact combination of inputs, context, and model state that produced the original output.</p><p>These properties create a fundamental challenge. Standard logging tells you what happened at the infrastructure level. AI observability must tell you what happened at the semantic level, capturing enough information to understand not just whether the system responded, but whether it responded well.</p><p>According to Grafana&#8217;s 2026 Observability Survey, adoption of LLM-specific observability is accelerating but remains early. The percentage of organizations with no LLM observability on their radar dropped from 42% in 2025 to 29% in 2026, but only 14% are using it for production workloads. The gap between AI deployment and AI observability represents one of the most significant operational risks in enterprise technology today.</p><h2>What to Log</h2><p>The first question any AI observability program must answer is what to capture. Log too little, and you cannot debug problems or measure quality. Log too much, and you create performance overhead, storage costs, and privacy exposure. The right approach captures enough to reproduce behavior and assess quality without recording unnecessary raw content.</p><p>Here are the core event types that a production AI observability system should capture.</p><h4>Prompts and system instructions</h4><p>Every request to an AI model begins with some combination of system instructions, user input, and conversation history. Logging these inputs, or at minimum their hashed versions and associated version identifiers, is essential for debugging and for understanding how prompt changes affect output quality. The most practical approach is to log a prompt template version identifier rather than the full prompt text, then maintain a separate versioned prompt registry that allows reconstruction when needed.</p><h4>Retrieved context</h4><p>For retrieval-augmented generation systems, the documents or passages retrieved from vector stores or search indices are a critical determinant of output quality. Log the document identifiers, relevance scores, and retrieval metadata. If the model produces a hallucinated answer, you need to determine whether the retrieval step failed to surface the right information or whether the model ignored the relevant context it was given. Logging full document content is generally unnecessary and raises data governance concerns. Reference identifiers with scores are usually sufficient.</p><h4>Model outputs</h4><p>The model's response is the primary artifact of interest. For debugging and evaluation, teams need access to outputs. However, logging full output text in all cases can be expensive and may capture sensitive information. Many organizations implement tiered logging, capturing full outputs for a configurable sample of requests while logging only metadata such as output length, finish reason, and token counts for the remainder.</p><h4>Tool calls and function invocations</h4><p>When an AI system invokes tools or external functions, log the tool name, input parameters, return values, execution duration, and any errors. For agentic systems that make multiple tool calls in sequence, capture the ordering and dependencies among them. This is where observability for AI most closely resembles distributed tracing in microservices architecture, and it benefits from the same structural patterns.</p><h4>User actions and session context</h4><p>What did the user do after receiving the AI response? Did they accept a suggestion, edit it, reject it, escalate to a human, or abandon the session? These downstream actions are among the most valuable signals for measuring real-world quality, yet they are frequently overlooked. Log user interactions with sufficient context to correlate them back to the AI response that triggered them.</p><h4>Latency and performance metrics</h4><p>Time to first token, total response time, retrieval latency, and individual tool call durations all matter for user experience and cost management. Break latency into its component parts to identify bottlenecks. The retrieval step might cause a slow response, the model inference, or a downstream tool call, and the remedy differs in each case.</p><h4>Token usage and cost</h4><p>Token consumption directly translates to cost for API-based model deployments. Log input tokens, output tokens, and reasoning tokens separately for each request. Aggregate these by use case, user segment, prompt version, and model to understand cost drivers and identify optimization opportunities. Organizations running AI at scale frequently discover that a small number of use cases or prompt patterns account for a disproportionate share of their token spend.</p><h4>Errors and exceptions</h4><p>Beyond standard HTTP errors and timeouts, AI systems produce a category of failures that do not register as errors in traditional monitoring. A model that returns a refusal, a safety filter activation, a truncated response due to context window limits, or a response that fails a validation check should be logged as a structured event with sufficient context to diagnose the cause.</p><h4>Metadata</h4><p>Every logged event should carry standard metadata, including the model name and version, provider, environment, deployment identifier, and timestamp. This metadata enables filtering, correlation, and comparison across model versions, environments, and time periods.</p><p>The guiding principle is to log enough to reproduce and evaluate any interaction without capturing unnecessary raw content. For most organizations, this means a combination of structured metadata for every request, sampled full-content logging for a subset of requests, and on-demand full-content capture that can be activated for specific users, sessions, or time windows when debugging requires it.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>How to Structure Telemetry</h2><p>Knowing what to log is only half the challenge. How you structure that telemetry determines whether it is searchable, correlated, and actionable, or whether it becomes an expensive pile of unstructured text that no one can use effectively.</p><p>The dominant paradigm for structuring distributed system telemetry combines traces, spans, and events. This approach has been validated at scale in traditional observability and is now being adapted for AI workloads. The OpenTelemetry project has introduced experimental semantic conventions specifically for generative AI, defining standardized attributes for model calls, tool invocations, agent operations, and evaluation events. These conventions are emerging as the industry standard for AI telemetry.</p><p><strong>Traces</strong> represent end-to-end interactions. A single user request that triggers retrieval, model inference, tool execution, and response generation should be captured as a single trace that encompasses the entire chain. Every trace should carry a unique trace ID, a session ID that groups related interactions within a conversation, and a user or account identifier.</p><p><strong>Spans</strong> represent individual operations within a trace. A retrieval step is a span. A model inference call is a span. A tool invocation is a span. Each span carries its own start and end timestamps, status, and operation-specific attributes. Spans are nested to reflect parent-child relationships, allowing you to see that a model inference span spawned three tool call spans, one of which spawned its own model inference span.</p><p>The OpenTelemetry GenAI semantic conventions define specific span types and attributes to ensure AI telemetry is consistent and interoperable across providers and frameworks. Key attributes include <code>gen_ai.operation.name</code> for the type of operation, <code>gen_ai.request.model</code> for the model being called, <code>gen_ai.usage.input_tokens</code> and <code>gen_ai.usage.output_tokens</code> for token consumption, and <code>gen_ai.provider.name</code> for the model provider. Datadog, Splunk, Grafana, and other major observability platforms have begun supporting these conventions natively, allowing teams to instrument once and analyze across platforms.</p><p><strong>Events</strong> capture discrete occurrences within spans. The OpenTelemetry GenAI conventions define events for prompt inputs, model outputs, tool calls, and evaluation results. These events can carry structured payloads, but the conventions explicitly recommend that instrumentation not capture prompt and response content by default. Instead, content capture should be opt-in, with three recommended patterns. The first is not to record content at all, relying solely on metadata. The second is to record content directly on spans using attributes, which is suitable for pre-production environments where privacy constraints are relaxed. The third is to store content externally and record only references on spans, recommended for production environments where data volume and sensitivity require separate handling.</p><p>A well-structured telemetry schema for an AI interaction might look like this conceptually. The trace carries a trace ID, session ID, user ID, and environment. The root span represents the overall request, with attributes for the use case, request source, and prompt template version. Child spans represent retrieval with attributes for the number of documents returned and relevance scores. Another child span represents model inference with attributes for the model name, token counts, latency, and finish reason. Additional child spans represent tool calls, including their names, inputs, outputs, and durations. An evaluation event attached to the model inference span includes a score, the evaluator&#8217;s name, and the evaluation criteria.</p><p>This structure makes AI interactions searchable by any dimension. You can query for all requests that used a specific prompt version. You can filter for interactions where retrieval returned fewer than three documents. You can aggregate token usage by model and use case. You can correlate quality scores with specific prompt versions to measure the impact of changes.</p><p><strong>Prompt versioning</strong> deserves special emphasis. Every system prompt, user prompt template, and few-shot example set should be version-controlled and referenced in telemetry by a version identifier. When a quality regression occurs, correlating it with a specific prompt version change is often the fastest path to the root cause. Without versioning, teams resort to comparing timestamps against deployment logs, a brittle and error-prone process.</p><p><strong>Session and conversation management</strong> are equally important for conversational AI systems. A session ID groups related turns in a conversation, allowing you to analyze multi-turn interaction patterns, measure conversation-level success rates, and identify points of abandonment. Without session-level correlation, you can measure individual response quality but cannot understand the user journey.</p><h2>Feedback and Evaluation Signals</h2><p>Observability without quality measurement is like monitoring a factory&#8217;s power consumption without inspecting the products coming off the line. You know the system is running, but you do not know if it is producing good results.</p><p>Feedback and evaluation signals close this gap by providing continuous measurement of AI output quality. These signals come from three sources, each with distinct strengths and limitations.</p><p><strong>User feedback</strong> is the most direct signal of real-world quality, but it is also the sparsest. Explicit feedback mechanisms such as thumbs-up and thumbs-down buttons, star ratings, or correction interfaces capture user satisfaction, but typically only a small percentage of users provide it. Implicit feedback, including acceptance or rejection of suggestions, editing of generated content, escalation to human agents, and session abandonment, provides higher coverage but requires careful interpretation. A user who edits an AI-generated email may be making minor stylistic adjustments or may be correcting fundamental errors. The edit distance or type of change can help distinguish these cases, but ambiguity remains.</p><p>Structure user feedback as events attached to the relevant model inference span in your telemetry. Include the feedback type, value, timestamp, and any associated user action. This allows you to correlate feedback with specific model outputs, prompt versions, and retrieved context, enabling analysis of what drives quality from the user&#8217;s perspective.</p><p><strong>Automated evaluators</strong> provide the scale and consistency that human feedback cannot. These evaluators run programmatically against model outputs and produce structured scores on dimensions such as groundedness (does the output stick to provided context), relevance (does the output address the user's question), faithfulness (is the output factually consistent with source material), toxicity (does the output contain harmful content), and format compliance (does the output follow specified structural requirements).</p><p>Automated evaluators fall into two categories. Rule-based evaluators use pattern matching, heuristics, or deterministic checks to assess outputs. They are fast and consistent, but limited to well-defined criteria. LLM-based evaluators use a separate model to assess the primary model&#8217;s output, typically using a detailed rubric and chain-of-thought prompting to improve reliability. LLM-based evaluators can assess nuanced dimensions such as helpfulness and accuracy, but they also introduce their own costs and latency. A 2025 McKinsey survey found that 51% of organizations using AI experienced at least one negative consequence, underscoring the importance of systematic evaluation rather than reactive incident response.</p><p>The OpenTelemetry GenAI conventions include an evaluation event type that captures the evaluator name, score, score label, and the trace or response being evaluated. This standardization allows evaluation results from different sources and evaluators to be aggregated, compared, and analyzed within the same observability infrastructure.</p><p><strong>Human review and annotation</strong> provide the highest-fidelity quality signal, but at the highest cost. Structured review workflows where subject matter experts assess AI outputs for accuracy, appropriateness, and domain correctness generate labeled data that can be used to calibrate automated evaluators, identify failure patterns, and build regression test sets. The practical approach is to route a risk-weighted sample of production outputs for human review, prioritizing high-stakes use cases, low-confidence predictions, and cases flagged by automated evaluators.</p><p>The combination of these three feedback sources creates a quality measurement system that operates at multiple levels of fidelity and coverage. Automated evaluators run on every request, providing baseline quality monitoring. User feedback supplements this with a real-world signal. Human review provides periodic deep assessment and calibration. Together, they form a feedback loop that connects observability to continuous improvement.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/your-ai-is-running-but-is-it-working?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/your-ai-is-running-but-is-it-working?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Sensitive Data Handling</h2><p>Here is the challenge that makes AI observability fundamentally different from traditional application logging from a privacy perspective. In a conventional application, the sensitive data lives in database fields with well-defined schemas. You know which fields contain personal information because you designed the schema. In AI systems, sensitive data can appear anywhere. Users paste personal information into prompts. Retrieval systems surface documents containing confidential data. Models may generate outputs that include or infer sensitive information. The unstructured, free-text nature of AI interactions means that every prompt and every response is a potential vector for sensitive data leakage into your telemetry pipeline.</p><p>This is not a hypothetical concern. Research on telemetry pipelines has found that observability systems can inadvertently become repositories for personally identifiable information when they ingest the unstructured, information-rich prompts that users submit to AI systems. For organizations subject to GDPR, HIPAA, CCPA, or other data protection regulations, sensitive data in observability logs creates compliance exposure that extends well beyond the AI system itself.</p><p>Addressing this requires a defense-in-depth approach that applies controls at multiple layers.</p><h4>Redaction before persistence</h4><p>The most effective control is to detect and remove sensitive data before it reaches your logging backend. This should happen in the telemetry pipeline itself, not as a post-hoc cleanup process. Tools like Microsoft Presidio combine regular expression patterns with named entity recognition models to identify and redact names, email addresses, phone numbers, credit card numbers, social security numbers, and other PII categories. The OpenTelemetry Collector supports custom processors that can intercept traces and redact sensitive attributes before forwarding them to the observability backend. This approach, sometimes called &#8220;safe observability,&#8221; treats redaction as a first-class pipeline operation rather than an afterthought.</p><h4>Field-level allowlists</h4><p>Rather than trying to identify and remove all sensitive data from logged content, a more robust approach is to default to logging only structured metadata fields and explicitly allow specific content fields when needed. This inverts the typical logging model. Instead of logging everything and hoping your redaction catches sensitive data, you log only known-safe fields by default. A typical allowlisted event might include the trace ID, session ID, model name, prompt version, token counts, latency, finish reason, evaluation scores, and error codes, but not the raw prompt text, output text, or retrieved documents. This metadata alone is sufficient for most operational analysis.</p><h4>Tiered access and break-glass procedures</h4><p>Some debugging scenarios genuinely require access to full prompt and response content. Rather than logging this content for all requests, implement a break-glass mechanism that allows authorized personnel to access higher-fidelity logs under controlled conditions. This might involve a separate, access-controlled logging tier that captures the full content of a configurable sample of requests, with access gated behind approval workflows and audited via immutable access logs. The key is that full-content access is an exception requiring justification, not the default.</p><h4><strong>Pseudonymous identifiers</strong></h4><p>Replace user-identifying information with pseudonymous IDs in your telemetry. A hashed or tokenized user identifier allows you to correlate requests within a session and across sessions for the same user without exposing the user&#8217;s identity in the logging system. If you need to resolve a pseudonymous ID back to a real user, that resolution should require a separate lookup through an access-controlled service, creating an additional barrier against casual exposure.</p><h4>Encryption and access control</h4><p>Telemetry data at rest should be encrypted, and access should be restricted to roles that require it. This is standard practice for any sensitive data store, but it is worth emphasizing because observability platforms are often treated with less security rigor than production databases. If your AI logs contain prompts and responses, they deserve the same protection as your customer data, because they may very well contain customer data.</p><h4>Retention policies</h4><p>Define and enforce retention periods for different categories of telemetry data. Structured metadata might be retained for months or years to support trend analysis. Sampled full-content logs might be retained for days or weeks to support debugging. Evaluation data might be retained for the life of the model to support longitudinal quality analysis. Automated expiration reduces compliance risk by ensuring that sensitive data does not accumulate indefinitely. Critically, retention policies must apply to backups and replicas as well as primary storage. A deletion policy that applies to the primary store but not to the data warehouse or backup system provides only the illusion of compliance.</p><h4>Third-party logging boundaries</h4><p>Many AI systems integrate with external providers for model inference, retrieval, or tool execution. Understand what data is transmitted to these third parties and what they log on their end. Your redaction pipeline should sanitize data before it leaves your infrastructure, not after. This is particularly important for prompts sent to external model APIs, which may contain user data that you are contractually or legally prohibited from sharing with third parties.</p><p>The goal is not to eliminate logging. The goal is useful observability with data minimization enforced before data is stored. Security and compliance reviews will focus less on your dashboard and more on whether you can demonstrate that minimization, access control, retention, and deletion actually hold throughout the logging path.</p><h2>Operational Use Cases</h2><p>An observability stack is an investment. Like any investment, it must deliver returns that justify its cost. The following operational use cases demonstrate how AI observability translates into tangible business value.</p><h4>Debugging and root cause analysis</h4><p>When a user reports that the AI system gave the wrong answer, debugging without observability involves guesswork. With a properly instrumented stack, the response team can pull the trace for the specific interaction, inspect the prompt version and system instructions, examine the retrieved context and relevance scores, review the model&#8217;s raw output, and identify where the chain broke down. Was it a retrieval failure that surfaced irrelevant documents? A prompt issue that gave the model ambiguous instructions? A model limitation on a specific type of reasoning? Structured traces with semantic context transform debugging from an art into a systematic process.</p><h4>Incident response</h4><p>When AI quality degrades across a user population, observability enables rapid detection and triage. Monitoring dashboards built on evaluation scores, error rates, and user feedback can trigger alerts when quality metrics drop below defined thresholds. The ability to slice metrics by prompt version, model version, and time window allows teams to identify whether the degradation correlates with a specific change quickly. Organizations that have invested in AI observability report significantly faster mean time to resolution for AI-related incidents because they can isolate the contributing factor without resorting to trial-and-error rollbacks.</p><h4>Prompt tuning and optimization</h4><p>Prompt engineering is iterative work that benefits enormously from data. Observability enables teams to compare quality scores across prompt versions using production data rather than synthetic benchmarks. When you see that prompt version 3.2 produces higher groundedness scores but lower user satisfaction than version 3.1, you have actionable information to improve. Without this data, prompt optimization is driven by intuition and anecdote.</p><h4>Evaluation and regression testing</h4><p>Production observability data feeds directly into evaluation workflows. Sampled production traces become the test cases for regression suites. When a team proposes a prompt change or model upgrade, they can run the new configuration against real production scenarios and compare quality scores with the current baseline. This is how AI teams build the same kind of continuous integration and testing discipline that software engineering teams have relied on for decades, adapted for non-deterministic systems.</p><h4>Compliance and audit readiness</h4><p>Regulations such as the EU AI Act require organizations to maintain documentation on how their AI systems make decisions, what data they use, and how they are monitored. A well-structured observability stack provides the evidentiary foundation for compliance. Traces demonstrate the decision-making process. Evaluation logs demonstrate ongoing quality monitoring. Access logs demonstrate that sensitive data is appropriately protected. Organizations that build observability with compliance in mind from the start will find audit preparation far less burdensome than those that attempt to reconstruct this evidence after the fact.</p><h4>Cost management</h4><p>Token usage telemetry, aggregated by use case, model, and prompt version, provides the visibility needed to manage AI infrastructure costs. Teams can identify use cases with disproportionate token consumption, detect prompt patterns that unnecessarily inflate context windows, compare cost-per-quality-point across models, and set budget alerts that trigger when usage trends exceed forecasts. As AI workloads scale, cost management becomes a strategic concern, and observability is the only reliable mechanism for maintaining visibility into how the money is spent.</p><h2>Building the Foundation</h2><p>The observability stack for AI is not a single tool or platform. It is an architectural layer that integrates telemetry capture, structured storage, evaluation pipelines, privacy controls, and operational workflows into a coherent system. The organizations doing this well share several characteristics.</p><p>They treat observability as a first-class requirement, not an afterthought that gets bolted on after deployment. They design their telemetry schema before writing application code, ensuring that the instrumentation supports the analysis they will need. They adopt open standards, particularly OpenTelemetry&#8217;s GenAI semantic conventions, to avoid vendor lock-in and enable interoperability. They enforce privacy controls in the pipeline, not in policy documents. And they connect observability to action, using evaluation signals to drive prompt improvements, model selection decisions, and incident response.</p><p>The tooling landscape is maturing rapidly. Platforms such as Arize, Langfuse, Datadog LLM Observability, Splunk, and others now offer purpose-built capabilities for tracing AI interactions, evaluating output quality, and monitoring production behavior. The OpenTelemetry community&#8217;s work on GenAI semantic conventions is establishing an interoperability layer that allows organizations to instrument once and analyze across multiple backends.</p><p>But tools alone are not enough. The observability stack requires organizational commitment to instrumentation discipline, evaluation rigor, and privacy hygiene. It requires that engineering teams adopt new practices for prompt versioning and telemetry schema design. It requires governance teams to define retention policies and access controls that balance debugging utility and privacy protection. And it requires that leadership recognize observability as a strategic capability that protects the organization&#8217;s AI investments.</p><p>The organizations that build this foundation now will operate with a decisive advantage. They will debug faster, improve quality systematically, manage costs proactively, and demonstrate compliance with confidence. Those who defer will find themselves operating AI systems they cannot explain, audit, or improve with any rigor.</p><p>You would not operate a production database without monitoring. You would not deploy a web application without error tracking. The bar for AI systems should not be lowered. If anything, given the non-deterministic nature of these systems and the sensitivity of the data they process, the bar should be higher.</p><p>Start with structured metadata on every request. Add evaluation signals. Enforce redaction before persistence. Build from there.</p><p>The observability stack for AI is not optional infrastructure. It is the difference between deploying AI and operating it responsibly.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/your-ai-is-running-but-is-it-working/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/your-ai-is-running-but-is-it-working/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[From ML Monitoring to AI Observability: What Changes with LLMs and Agents]]></title><description><![CDATA[Closing the visibility gap that separates AI leaders from liabilities]]></description><link>https://trustedai.recodework.com/p/from-ml-monitoring-to-ai-observability</link><guid isPermaLink="false">https://trustedai.recodework.com/p/from-ml-monitoring-to-ai-observability</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Tue, 31 Mar 2026 00:19:30 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-wF4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-wF4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-wF4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg 424w, https://substackcdn.com/image/fetch/$s_!-wF4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg 848w, https://substackcdn.com/image/fetch/$s_!-wF4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!-wF4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!-wF4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg" width="1456" height="910" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:910,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:4759626,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/192643616?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!-wF4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg 424w, https://substackcdn.com/image/fetch/$s_!-wF4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg 848w, https://substackcdn.com/image/fetch/$s_!-wF4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!-wF4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe78d650e-431d-4ea7-8be4-4b94482bc813_6400x4000.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>For the better part of a decade, we built monitoring systems around a relatively stable set of assumptions. A model took structured inputs, produced numerical outputs, and could be evaluated using clean performance metrics, such as accuracy, precision, recall and F1 score. The discipline of ML monitoring matured around these signals, and it worked for traditional machine learning.</p><p>That era is not over, but it is no longer sufficient.</p><p>The shift from predictive models to large language models and autonomous agents has fundamentally altered what it means for an AI system to fail, how failures manifest, and what organizations need to see to catch problems before they reach customers. We are moving from a world of wrong numbers to a world of wrong words, wrong actions, and wrong reasoning chains that can look perfectly convincing on the surface.</p><p>Traditional monitoring tells you that your system is running. AI observability tells you whether your system is thinking correctly. That distinction is the central challenge for every organization deploying generative AI in production today.</p><p>This is not an incremental evolution. It is an architectural and philosophical shift in how we think about AI system health. And for organizations that have invested in traditional ML monitoring, the uncomfortable truth is that most of those investments address only a fraction of the risks posed by LLMs and agents.</p><p>The organizations that recognize this shift early and build observability capabilities accordingly will operate with a level of confidence that their competitors cannot match. Those who try to stretch their existing monitoring paradigm to cover generative AI will discover the gaps the hard way.</p><h2>Why the Shift Matters Now</h2><p>The urgency is real and measurable. Enterprise spending on generative AI reached $37 billion in 2025, more than tripling from the prior year. AI adoption across organizations has climbed to 72%, and agentic AI is the fastest emerging deployment pattern, with Gartner projecting that by 2028, more than a third of enterprise software applications will incorporate agentic AI capabilities.</p><p>Yet the infrastructure to govern these systems has not kept pace. A 2026 survey of over 1,200 cybersecurity and IT professionals found that 91% of organizations only discover what an AI agent did after it has already executed the action. Only 23% enforce AI security in-line at the point of action. And 37% of respondents reported that AI agents had caused operational issues within the past twelve months, with 8% experiencing incidents severe enough to cause outages or data corruption.</p><p>The monitoring tools that served us well for traditional ML were never designed for this use case. They were built for a different category of system, a different type of output, and a different failure profile. Understanding what has changed is the first step toward closing the gap.</p><h2>What ML Monitoring Was Designed For</h2><p>Traditional ML monitoring emerged from a well-defined problem space. You trained a model on structured, tabular data. The model produced a prediction, typically a number or a classification label. And you could evaluate that prediction against ground truth with mathematical precision.</p><p>The monitoring stack that evolved to support these systems focused on several core capabilities.</p><ol><li><p><strong>Data drift detection</strong> tracked whether the statistical properties of production data had shifted relative to the training set. If your credit scoring model was trained on applicant data from 2022 and the economic environment had materially changed by 2024, the input distributions would shift. Feature values would fall outside historical ranges. Monitoring systems could detect these shifts through statistical tests and alert teams before model performance degraded significantly.</p></li><li><p><strong>Feature monitoring</strong> observed critical input variables for outliers, missing values, or distributional changes. If a key feature suddenly started arriving as null for 30% of records, you needed to know immediately.</p></li><li><p><strong>Prediction monitoring</strong> continuously tracked standard performance metrics. Accuracy, precision, recall, ROC-AUC and calibration metrics provided quantitative signals about whether a model was still performing within acceptable bounds. You could set thresholds, trigger alerts, and know exactly what &#8220;degradation&#8221; looked like because it was expressed in numbers.</p></li><li><p><strong>Bias and fairness checks</strong> evaluated whether model outputs exhibited disparate impact across protected groups. These evaluations, while not always straightforward, operated on structured outputs that could be sliced, aggregated, and compared using well-established statistical methods.</p></li></ol><p>The entire ecosystem that grew around these requirements reflects the problem's underlying simplicity. Platforms like Arize, Evidently AI, and Fiddler built powerful capabilities for tracking model performance, detecting data and concept drift, and monitoring for bias in production. They integrated with model registries, CI/CD pipelines, and alerting systems to create a mature operational practice.</p><p>This entire paradigm rested on several key assumptions. Inputs were structured and predictable. Outputs were deterministic, meaning the same input would always produce the same output. Correctness was binary or at least quantifiable. The relationship between inputs and outputs could be expressed mathematically. And the model&#8217;s behavior was bounded by the feature space it was trained on.</p><p>These assumptions made monitoring tractable. You could build dashboards with clear metrics. You could set thresholds that meant something. And when a metric crossed a line, you had a reasonable basis for diagnosing what went wrong.</p><p>The problem is that virtually none of these assumptions hold for LLMs and agents.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>Why LLMs and Agents Are Different</h2><p>Large language models and agentic AI systems fundamentally break the traditional monitoring paradigm. The differences are not cosmetic. They require a different conceptual framework for understanding system health.</p><h4>Outputs are unstructured and non-deterministic</h4><p>A traditional model returns a number. An LLM returns natural language. The same prompt can produce materially different responses across invocations. There is no single &#8220;right answer&#8221; to compare against, and the quality of a response exists on a spectrum that is often subjective. Was the summary accurate? Was the tone appropriate? Did the response address the user&#8217;s actual intent? These questions cannot be answered with an F1 score.</p><h4>Context shapes everything</h4><p>In traditional ML, a model&#8217;s behavior is determined by its training data and the input features it receives. In LLM systems, behavior is additionally shaped by the prompt, the system instructions, the conversation history, any retrieved documents from a RAG pipeline, and the specific model version being called. A minor change to a system prompt can radically alter output behavior across thousands of interactions. Monitoring systems that do not capture this full context cannot diagnose problems when they arise.</p><h4>Failure modes are semantic, not statistical</h4><p>When a traditional model fails, it produces an incorrect result. When an LLM fails, it produces a confident, articulate, and entirely fabricated response. Hallucinations do not announce themselves. They look exactly like the correct outputs. The model does not return a low-confidence score or throw an error. It states falsehoods with the same fluency it uses for facts. Detecting this failure requires evaluating the meaning of outputs, not just their statistical properties.</p><h4>Agents add autonomy and real-world consequences</h4><p>When LLMs are embedded in agentic architectures, meaning systems that can plan, reason across multiple steps, call external tools, and take actions in the world, the stakes compound dramatically. An agent might execute a database query, send an email, modify a file, or trigger a financial transaction. Each step in a multi-step workflow represents a decision point at which the system could deviate from its intended behavior. And because agents chain these steps together, errors in early steps can cascade through the entire execution path. A single flawed reasoning step can produce downstream consequences that are difficult to trace and sometimes impossible to reverse.</p><p>The complexity is staggering. A single user request to an agent system may trigger 15 or more LLM calls across multiple chains and models, each with its own prompt context, tool interactions, and intermediate outputs. Traditional monitoring that tracks a single request-response pair is architecturally incapable of providing visibility into this kind of execution graph. You are no longer monitoring a function call. You are monitoring a decision-making process that unfolds over time and across systems.</p><h4>The evaluation problem becomes qualitative, not just quantitative</h4><p>When a traditional model misclassifies an input, you can compute the error and attribute it to specific features. When an LLM produces a response that is technically accurate but tonally inappropriate, or factually correct but misleading in context, or helpful but reveals information it should not have shared, the evaluation problem is fundamentally different. It requires human judgment, domain expertise, and often subjective assessment. Automated evaluation methods exist and are improving, but they introduce their own uncertainties and failure modes.</p><h4>The attack surface expands</h4><p>Traditional models had limited exposure to adversarial inputs. LLMs and agents face prompt injection attacks, where malicious instructions are embedded in user inputs or retrieved documents. They face memory poisoning, in which adversaries corrupt an agent&#8217;s stored context to influence its future behavior. Microsoft researchers have demonstrated scenarios in which an AI email assistant was compromised by a specially crafted email, causing it to forward sensitive correspondence to an attacker. These are not theoretical risks. They are production realities that traditional monitoring was never designed to detect.</p><p>These differences are not edge cases or academic concerns. They represent the core operating characteristics of the systems that enterprises are deploying at scale right now. And they demand a fundamentally different approach to understanding system behavior.</p><h2>What AI Observability Needs to Capture</h2><p>If monitoring asks, &#8220;Is this metric within threshold?&#8221; then observability asks, &#8220;What is actually happening inside this system and why?&#8221; The distinction matters because LLM and agent failures rarely present as clean metric violations. They present as subtle behavioral shifts, reasoning errors, and semantic degradation that require deep visibility to detect.</p><p>A mature AI observability practice for LLM and agent systems needs to capture several categories of signals that traditional monitoring does not address.</p><h4>Full trace capture across multi-step workflows</h4><p>Every LLM call, tool invocation, document retrieval, and reasoning step should be logged with sufficient context to replay the entire execution path. When an agent produces a problematic output, you need to be able to trace back through the chain of decisions that led to it. Which prompt was used? Which documents were retrieved? What tool calls were made? What intermediate reasoning did the model produce? Without this trace data, debugging agent failures becomes an exercise in guesswork.</p><p>Modern observability platforms are standardizing around OpenTelemetry (OTel) conventions for AI systems, providing common frameworks for instrumenting agent applications regardless of the underlying framework. This standardization is critical for organizations running multiple agent architectures, as it enables consistent visibility across heterogeneous deployments.</p><h4>Semantic quality evaluation</h4><p>Traditional metrics like latency and error rates still matter, but they are insufficient. An LLM can return a response in 200 milliseconds with a 200 status code that is entirely hallucinated. Observability systems must evaluate the meaning of outputs, assessing relevance, factual accuracy, coherence, and alignment with instructions. This often requires using evaluation models, sometimes called LLM-as-judge approaches, to score production outputs at scale.</p><h4>Prompt and context drift detection</h4><p>Just as data drift degrades traditional models, prompt drift and context drift degrade LLM systems. If the documents in your RAG pipeline become stale or if prompt templates are modified without proper testing, output quality can degrade in ways that are invisible to latency and error monitoring. Observability systems need to track changes in prompts, retrieved documents, and system instructions over time and correlate those changes with shifts in output quality.</p><h4>Cost and token economics</h4><p>LLM systems introduce a cost dimension that traditional ML monitoring rarely needed to address at the individual prediction level. Every API call consumes tokens, and token usage varies dramatically based on prompt length, context window utilization, and response verbosity. An agent that enters a reasoning loop can consume thousands of dollars in API costs within minutes. Observability systems must track token usage, per-interaction costs, and cost trends to prevent runaway spending and optimize system economics.</p><h4>Safety and policy compliance</h4><p>Production LLM systems need real-time evaluation for outputs that violate organizational policies, contain personally identifiable information, exhibit bias, or produce harmful content. Unlike traditional models, where bias can be assessed through periodic batch analysis, LLM outputs require continuous, per-interaction evaluation because the same system can produce compliant output for one prompt and policy-violating output for the next.</p><h4>User feedback integration</h4><p>Because LLM output quality is often subjective, user feedback becomes a critical observability signal. Thumbs up and thumbs down ratings, explicit corrections, and behavioral signals like whether users accepted or rejected a suggestion all provide ground truth that automated metrics cannot fully capture. Observability systems need to correlate this feedback with specific traces to build a continuous improvement loop.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/from-ml-monitoring-to-ai-observability?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/from-ml-monitoring-to-ai-observability?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Monitoring Versus Observability in Practice</h2><p>The distinction between monitoring and observability is not merely semantic. It reflects a fundamental difference in capability that has real consequences for how organizations detect and respond to AI system failures.</p><h4>Monitoring is reactive and metric-driven</h4><p>You define thresholds for known metrics. When a threshold is breached, an alert fires. This works well for predictable failures. If latency exceeds two seconds, you know something is wrong with the infrastructure. If error rates spike above 5%, you know something is failing technically. These are valuable signals, and they remain necessary. But they represent the floor of visibility, not the ceiling.</p><h4>Observability is exploratory and context-rich</h4><p>It provides the data and tools needed to ask questions you did not anticipate. Why did this particular agent session produce a harmful output when the previous thousand sessions were fine? What changed in the retrieved documents that caused the model to start hallucinating about a specific topic? Which step in the reasoning chain introduced the error that cascaded through the rest of the workflow?</p><p>Consider a practical example. A customer-facing AI assistant starts providing incorrect information about a product&#8217;s return policy. Traditional monitoring might show that latency and error rates are normal and that the system is technically healthy. Everything looks green on the dashboard. But the outputs are wrong.</p><p>An observability platform with full trace capture would reveal that the RAG pipeline is retrieving an outdated version of the return policy document. The model is responding correctly given the context it received. The problem is upstream, in the retrieval layer, and it is invisible to any monitoring system that only tracks infrastructure metrics. The observability platform can surface this because it captured the retrieved documents alongside the model&#8217;s response and evaluated whether the response was consistent with current policy.</p><p>This is not a contrived scenario. It represents exactly the kind of silent failure that erodes customer trust before anyone in the organization recognizes a problem exists.</p><p>Now consider an agentic example. A financial services firm deploys an agent that assists analysts with research by querying internal databases, summarizing documents, and generating preliminary reports. The agent begins producing reports that cite data from a deprecated internal system. The data is structurally valid but months out of date. Latency metrics show the agent is performing well. Error rates are at zero. Cost per interaction is within normal bounds. Every traditional monitoring metric is green.</p><p>But the reports are wrong. And they are being used to inform investment decisions.</p><p>An observability platform that traces the agent&#8217;s full execution path would show which data sources were queried, what data was returned, and how that data was incorporated into the final output. It would enable the team to correlate the shift in data source usage with a recent infrastructure change that redirected database queries. And it would surface the issue within hours, rather than weeks, because semantic evaluation would flag that the cited data no longer matched current reality.</p><p>The practical divide between monitoring and observability becomes even more pronounced with agentic systems. Traditional debugging approaches force teams to mentally reconstruct complex agent workflows from fragmented logs scattered across multiple systems. Comprehensive observability captures the entire execution path, every reasoning step, tool call, and intermediate output, enabling teams to replay sessions and pinpoint exactly where behavior deviated from intent.</p><h2>Common Failure Modes to Watch</h2><p>Organizations deploying LLMs and agents in production should expect and instrument for several failure modes that have no meaningful parallel in traditional ML systems.</p><h4>Hallucination and confabulation</h4><p>The model generates outputs that are factually incorrect, fabricated, or internally inconsistent, but presented with full confidence. This is not a bug to be fixed. It is an inherent characteristic of language models' text generation. ISACA&#8217;s analysis of 2025 AI incidents emphasized that hallucinations should be treated as safety risks, not quirks. Every high-impact AI system should be designed with the assumption that it will sometimes be confidently wrong. Observability systems must continuously evaluate factual grounding, particularly for systems that inform consequential decisions.</p><h4>Prompt injection and manipulation</h4><p>Adversaries embed instructions within user inputs or external data sources that override the system&#8217;s intended behavior. This can cause an LLM to ignore safety guidelines, leak system prompts, or execute unintended actions. In agentic systems, prompt injection becomes even more dangerous because a compromised model may have access to tools capable of taking real-world actions. Observability must include input scanning and output evaluation specifically designed to detect injection patterns and anomalous behavioral shifts.</p><h4>Cascading failures in multi-agent systems</h4><p>When multiple agents collaborate in a workflow, an error or compromise in one agent can propagate through the entire chain. A procurement workflow where a vendor verification agent accepts fraudulent credentials will cause downstream procurement and payment agents to process illegitimate transactions. By the time the error is detected at the final output, the damage is done. Observability must provide visibility into each agent&#8217;s behavior and the interactions between agents, enabling teams to identify where in the chain a failure originated.</p><h4>Context window overflow and degradation</h4><p>As conversation history or retrieved context grows, LLMs can exceed their effective context window, leading to degraded performance, lost instructions, or ignored context. The model does not throw an error when this happens. It simply starts performing worse, potentially dropping critical instructions or losing track of earlier conversation context. Observability systems should track context utilization and correlate it with output quality.</p><h4>Tool misuse and privilege escalation</h4><p>Agents with access to external tools may use them in unintended ways, either due to flawed reasoning or to adversarial manipulation. An agent designed to query a database for read-only reporting might, through prompt manipulation or reasoning errors, attempt to execute write operations. A 2026 survey found that AI agents have write access to collaboration tools in 53% of organizations, email in 40%, and code repositories in 25%. Observability must track tool usage patterns and alert on deviations from expected behavior.</p><h4>Stale knowledge and retrieval degradation</h4><p>RAG-based systems are only as good as their knowledge base. When underlying documents become outdated, are corrupted, or are indexed incorrectly, the model will faithfully generate responses based on bad information. This failure is particularly insidious because the model&#8217;s behavior appears internally consistent. It is answering correctly, given the context it received. Only by monitoring retrieval quality, document freshness, and source relevance can teams detect this class of failure.</p><h4>Cost runaway</h4><p>Agents that enter reasoning loops, repeatedly call expensive models, or generate unnecessarily verbose outputs can consume disproportionate resources. Without token-level cost tracking and anomaly detection, organizations may not discover runaway costs until the monthly invoice arrives.</p><h2>What Teams Should Do Next</h2><p>The gap between traditional ML monitoring and the observability requirements of LLM and agent systems will not close on its own. Organizations need to take deliberate action to build the visibility infrastructure that these systems demand.</p><h4>Audit your current monitoring stack against LLM and agent requirements</h4><p>Most organizations have invested in infrastructure monitoring, application performance management, and possibly traditional ML monitoring. Map these existing capabilities against the observability requirements outlined above. Identify the gaps explicitly. In most cases, you will find that latency, error rates, and infrastructure health are well-covered, while semantic quality, trace capture, prompt drift, and safety evaluation are absent entirely.</p><h4>Instrument for full trace capture from day one</h4><p>Do not wait until a production incident to discover you lack the data needed to diagnose it. Every LLM call, tool invocation, and reasoning step should be instrumented to produce structured trace data. OpenTelemetry&#8217;s emerging semantic conventions for AI systems provide a vendor-neutral foundation for this instrumentation. Building on open standards avoids lock-in and ensures that trace data can be consumed by whatever observability platform you ultimately select.</p><h4>Implement semantic evaluation in production, not just in testing</h4><p>Pre-deployment evaluation is necessary but not sufficient. Model behavior in production will diverge from its behavior in testing because real users provide inputs that test suites do not anticipate, retrieved documents change over time, and model providers update their systems. Production evaluation should assess output quality, factual grounding, safety compliance, and policy adherence on an ongoing basis.</p><h4>Establish cost monitoring and alerting at the interaction level</h4><p>Track token usage, API costs, and cost per interaction in real time. Set anomaly detection thresholds that trigger alerts when individual sessions or aggregate costs deviate from historical patterns. This is especially critical for agentic systems, where a single runaway session can incur high unexpected costs.</p><h4>Build human feedback into the observability loop</h4><p>Create mechanisms for end users and subject matter experts to flag problematic outputs and provide corrections. Route this feedback into your observability pipeline so it can be correlated with traces, used to identify systematic failure patterns, and incorporated into evaluation benchmarks. A March 2026 survey found that production corrections from subject matter experts represent the highest-value signal for improving agent reliability, because they capture real failure modes on real inputs.</p><h4>Appoint clear ownership for AI observability</h4><p>The same survey found that only 14% of organizations have successfully scaled an AI agent to organization-wide operational use, and the distinguishing factor among those that succeeded was not better models or bigger budgets. It was establishing an AI operations function that owned production monitoring, evaluation, and incident response before scaling began. Observability without ownership is just data. Ownership without observability is just hope.</p><h4>Integrate AI observability into your governance framework</h4><p>Observability data should feed directly into governance processes. Production monitoring data should inform risk assessments. Incident response procedures should leverage trace data to accelerate root cause analysis. Board reporting should include meaningful metrics about AI system health, not just deployment counts and cost figures. As I have written previously, governance tells you what your AI should do. Observability tells you what your AI is actually doing. Neither works without the other.</p><h2>The New Baseline</h2><p>The shift from ML monitoring to AI observability is not optional. It is the natural consequence of deploying fundamentally different types of AI systems in production. Organizations cannot govern what they cannot see, and the systems we are deploying now are orders of magnitude more complex, more autonomous, and more consequential than the predictive models that our monitoring practices were designed to support.</p><p>The traditional ML monitoring baseline of data drift detection, feature monitoring, and performance metrics remains necessary. Those capabilities do not become irrelevant. But they become a small fraction of what is required when your AI systems produce unstructured outputs, reason across multi-step chains, call external tools, and take actions in the real world.</p><p>The new baseline includes full execution tracing, semantic quality evaluation, prompt and context drift detection, cost economics tracking, safety and policy compliance monitoring, and human feedback integration. It requires tooling that can capture the full context of every AI interaction and surface problems in the meaning of outputs, not just in their technical delivery.</p><p>The observability market itself reflects this reality, projected to grow from $3.35 billion in 2026 to nearly $7 billion by 2031, with AI workloads as a primary driver of demand. The market for AI-specific observability is growing even faster, reflecting the urgency that organizations are beginning to feel.</p><p>But market growth alone does not solve the problem for any individual organization. The question for every enterprise deploying LLMs and agents is whether they will build observability capabilities proactively, as part of their deployment architecture, or reactively, after a production failure forces their hand.</p><p>The organizations that invested in traditional ML monitoring before it was standard practice gained a meaningful advantage over those that waited. The same dynamic is playing out now with AI observability, at higher stakes and faster clock speed.</p><p>There is a telling data point from a recent enterprise survey. Organizations that have successfully scaled AI agents to production did not spend more on AI overall. Their total AI budgets were comparable to those of stalled organizations. The difference was in the allocation. Successful scalers spent proportionally more on evaluation infrastructure, monitoring tooling, and operational staffing, and proportionally less on model selection and prompt engineering.</p><p>In other words, the winners are not winning because they have better models. They are winning because they can see what their models are doing.</p><p>This is the strategic insight that should inform every future AI investment decision. The model is the engine. Observability is the instrument panel. You would never fly an aircraft without instruments, especially one that is actively learning to navigate while airborne. The same logic applies to AI systems that make decisions, take actions, and directly touch customers and operations.</p><p>The systems have changed. The monitoring must change with them.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/from-ml-monitoring-to-ai-observability/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/from-ml-monitoring-to-ai-observability/comments"><span>Leave a comment</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Building an AI Control Set That Auditors Can Understand]]></title><description><![CDATA[How to translate AI risks into concrete controls, owners, and evidence that fit your existing security and privacy programs.]]></description><link>https://trustedai.recodework.com/p/building-an-ai-control-set-that-auditors</link><guid isPermaLink="false">https://trustedai.recodework.com/p/building-an-ai-control-set-that-auditors</guid><dc:creator><![CDATA[Jon Knisley]]></dc:creator><pubDate>Thu, 19 Mar 2026 19:59:55 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!PeXH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!PeXH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PeXH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg 424w, https://substackcdn.com/image/fetch/$s_!PeXH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg 848w, https://substackcdn.com/image/fetch/$s_!PeXH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!PeXH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!PeXH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:6275379,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/191479125?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!PeXH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg 424w, https://substackcdn.com/image/fetch/$s_!PeXH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg 848w, https://substackcdn.com/image/fetch/$s_!PeXH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!PeXH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F540b9be9-429d-4e79-b722-62515a3c2bdd_4938x3292.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><h2>Why Auditors Struggle with AI, and Why That Is Your Problem</h2><p>Your auditors know how to evaluate access controls, encryption standards, and incident response plans. They have spent years building expertise around SOC 2, ISO 27001, HIPAA, and PCI DSS. They understand the language of controls, evidence, and risk registers. Then you deploy an AI system, and the conversation breaks down.</p><p>This is not a criticism of your audit team. It is a structural problem. Most compliance and audit frameworks were built for deterministic software systems in which inputs produce predictable outputs, logic can be inspected in the source code, and failures follow patterns that humans can trace. AI systems violate nearly all of these assumptions. They learn from data rather than execute static instructions. They can produce different outputs from identical inputs. Their decision logic is often opaque even to the engineers who built them. And their behavior degrades over time as the real world drifts away from their training data.</p><p>The result is a dangerous gap. Organizations are deploying AI systems into consequential business processes. At the same time, their audit and compliance functions lack the vocabulary, control structures, and evidence frameworks needed to assess whether those systems operate responsibly. Research from PwC confirms that AI introduces risk categories not addressed by standard control matrices, from data drift and bias to explainability and misuse. Meanwhile, NIST has acknowledged that traditional IT security controls alone are insufficient for AI and launched its Control Overlays for Securing AI Systems (COSAiS) project in August 2025 to bridge this gap.</p><p>The good news is that you do not need to invent a new compliance program from scratch. You need to extend the one you already have. The organizations getting this right are translating AI risks into the same language their auditors already speak: controls with clear owners, defined testing procedures, and documented evidence. This post shows you how to do exactly that.</p><h2>Start from Your Current Control Framework, Not from AI Hype</h2><p>The most common mistake organizations make when approaching AI governance is treating it as an entirely new discipline that requires entirely new infrastructure. They hire an AI ethics team that operates independently from risk management. They build AI governance documentation that exists outside the enterprise control framework. They create review processes that run in parallel with, rather than are integrated into, existing compliance workflows.</p><p>This approach fails for two reasons. First, it creates duplicative work that neither the AI team nor the compliance team wants to maintain. Second, and more importantly, it makes AI governance invisible to the auditors who are already assessing your organization. If your AI controls live in a separate system from your SOC 2 or ISO 27001 controls, your auditors will not see them, will not test them, and will not give you credit for them.</p><p>The better approach starts with a simple question: what control framework are we already using? For most organizations, the answer will be one or more of the following: SOC 2 Trust Services Criteria, ISO 27001 Annex A controls, NIST SP 800-53 control catalog, NIST Cybersecurity Framework, or a combination mapped across these standards. The AICPA has documented approximately 80% overlap between SOC 2 and ISO 27001, and the mapping relationships across NIST frameworks are well established. Your AI controls should plug directly into this existing structure.</p><p>NIST itself has endorsed this integration-first approach. Its COSAiS project is developing AI-specific overlays for SP 800-53 precisely because, as the project description states, the overlays are designed to address specific risks associated with different types of AI usage in conjunction with an organization&#8217;s cybersecurity risk management program and existing control implementations. The overlays assume that foundational controls such as access management, account management, and authentication are already in place. They add AI-specific tailoring on top.</p><p>ISO/IEC 42001, the first certifiable AI management system standard, follows the same philosophy. It uses the Plan-Do-Check-Act methodology that organizations already apply through ISO 27001, and it is designed to integrate with other management system standards rather than replace them. As ISACA recently noted, ISO/IEC 42001 aligns AI governance with other management systems, making it easier to fold new AI controls, metrics, and reviews into existing audit cadences and evidence repositories.</p><p>The practical implication is straightforward. Open your current control register. Identify the control families that are most relevant to AI risk. Then, determine where you need to add AI-specific sub-controls or modify existing control descriptions to account for AI behavior. Do not build a parallel universe.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/subscribe?"><span>Subscribe now</span></a></p><h2>A Practical AI Risk Taxonomy Your Auditors Will Understand</h2><p>Before you can write controls, you need to agree on what risks those controls are meant to address. This is where many organizations stumble. Academic AI risk taxonomies can be extraordinarily detailed, with some frameworks defining over fifty distinct sub-threats across nine domains. That level of granularity is valuable for AI security researchers. It is overwhelming for an audit partner to understand your risk landscape during a one-hour walkthrough.</p><p>What you need instead is a practical taxonomy that maps AI-specific risks to categories your auditors already work with. The following six-category framework is designed to do exactly that. Each category is grounded in established risk domains, maps to recognized framework elements, and translates into controls that compliance professionals can evaluate.</p><h4><strong>Category 1: Data Integrity and Provenance</strong> </h4><p>This covers risks associated with the data that AI systems consume, learn from, and generate. It includes training data quality and representativeness, data poisoning by malicious actors, unauthorized use of protected or copyrighted data, inadequate data lineage and documentation, and privacy violations through data repurposing. Your auditors already understand data governance through SOC 2 processing integrity criteria and ISO 27001 information classification controls. AI data risks extend these concepts to encompass training pipelines, data labeling, and the provenance of third-party datasets.</p><h4><strong>Category 2: Model Reliability and Performance</strong></h4><p>This addresses whether AI systems produce outputs that are accurate, consistent, and fit for purpose. It includes model accuracy degradation over time, concept drift and data distribution drift, hallucination in generative AI systems, performance variability across different input populations, and failure to generalize beyond training conditions. For auditors, this maps to the reliability dimension of operational risk. The key difference from traditional software is that AI performance is probabilistic rather than deterministic, and it changes without anyone modifying the code.</p><h4><strong>Category 3: Fairness and Bias</strong></h4><p>This encompasses risks that AI systems produce discriminatory or inequitable outcomes across demographic groups. It includes representational harm from biased training data, allocational harm in which resources or opportunities are unfairly distributed, proxy discrimination via correlated features, and algorithmic amplification of existing inequities. This is the category that most frequently makes headlines and triggers regulatory action. It aligns with existing anti-discrimination obligations, employment law requirements, and fair lending regulations. Still, it also requires AI-specific testing and monitoring controls that most compliance programs have not yet developed.</p><h4><strong>Category 4: Security and Adversarial Resilience</strong></h4><p>This covers the attack surface unique to AI systems beyond traditional cybersecurity threats. It includes prompt injection and jailbreaking of large language models, adversarial inputs designed to cause misclassification, model theft or extraction through inference attacks, supply chain risks from third-party models and components, and shadow AI usage by employees outside sanctioned channels. Auditors understand security controls deeply. The extension here is recognizing that AI systems introduce novel attack vectors, such as data poisoning and prompt manipulation, that firewalls and encryption alone cannot address.</p><h4><strong>Category 5: Transparency and Explainability</strong></h4><p>This addresses the organization&#8217;s ability to explain how AI systems work and how they reach their outputs. It includes documentation of the model's purpose, limitations, and intended use; the ability to provide explanations for individual decisions when required; disclosure to affected individuals that AI is being used; auditability of the decision logic and contributing factors; and documentation of known limitations and failure modes. Research consistently shows that 83% of companies deploying AI consider explainability essential to their business. For auditors, transparency controls create the documentation trail they need to evaluate other controls. Without adequate documentation, nothing else is testable.</p><h4><strong>Category 6: Governance and Accountability</strong></h4><p>This covers the organizational structures that ensure responsible AI oversight. It includes ownership assignment for every AI system, defined approval and escalation processes, lifecycle management from development through retirement, incident response procedures specific to AI failures, and third-party AI vendor management. This category maps directly to the governance and oversight domains in every major compliance framework. The challenge is ensuring that AI-specific responsibilities are explicitly assigned rather than assumed to fall under existing IT governance.</p><p>This taxonomy is deliberately concise. It covers the essential risk landscape without requiring auditors to learn an entirely new discipline. Each category connects to existing compliance concepts while highlighting what is genuinely different about AI.</p><h2>Turning AI Risks into Explicit Controls, Owners, and Evidence</h2><p>A risk taxonomy tells you what can go wrong. Controls tell you what you will do about it. The discipline of writing effective controls is well established in the compliance world: each control should have a clear description, an assigned owner, a defined testing procedure, specified evidence requirements, and a link to the risk it mitigates. AI controls follow the same structure. They simply address different failure modes.</p><p>The following examples illustrate how risks from each taxonomy category translate into auditable controls. These are not exhaustive but demonstrate the pattern.</p><h4>Data Integrity Controls</h4><p>For training data quality, the control might state that all datasets used for model training or fine-tuning are documented in a data registry that records the source, collection method, date of acquisition, known limitations, and applicable usage rights. The owner would be the head of data engineering or an equivalent role. The evidence would include the data registry itself, acquisition records, and usage authorization documentation. The testing procedure would involve the auditor sampling entries from the registry, verifying completeness against the defined schema, and then tracing a sample of training datasets back to their source documentation.</p><p>For data privacy in AI contexts, the control might require that personal data used in AI training undergo a privacy impact assessment that evaluates consent basis, purpose limitation, and data minimization requirements specific to AI processing. The owner would be the privacy officer or data protection lead. Evidence would include completed privacy impact assessments and the decision register documenting approval or rejection of data use.</p><h4>Model Reliability Controls</h4><p>For performance monitoring, the control could specify that all production AI models are monitored against defined performance thresholds, with automated alerts when metrics breach acceptable ranges, and that monitoring reports are reviewed at a defined cadence. The owner would be the ML engineering lead or model operations team. Evidence would include monitoring dashboard configurations, alert threshold definitions, sample alert logs, and review meeting minutes. This is where observability platforms like Arize, Evidently AI, and Fiddler become part of your evidence trail, not just your engineering toolkit.</p><p>For drift detection, the control might require that statistical tests for data distribution drift and concept drift are run at defined intervals for all production models, with documented escalation procedures when drift exceeds defined thresholds. The owner would be the data science team lead. Evidence would include drift detection reports, threshold documentation, and records of remediation actions taken upon detection of drift.</p><h4>Fairness Controls</h4><p>For bias assessment, the control could state that before deployment, all AI systems that make or inform decisions affecting individuals are tested for disparate impact across defined protected characteristics, with results documented and reviewed by a designated approver. The owner would be the AI product owner, with review by the risk committee. Evidence would include pre-deployment bias test results, the definition of protected characteristics tested, acceptance thresholds, and the approver&#8217;s sign-off.</p><h4>Security Controls</h4><p>For prompt injection prevention, the control might specify that generative AI systems exposed to user inputs implement input validation, output filtering, and system prompt protection mechanisms, and that these mechanisms are tested through adversarial testing before deployment and at regular intervals. The owner would be the application security team. Evidence would include adversarial test plans and results, input validation configuration documentation, and records of periodic re-testing.</p><h4>Transparency Controls</h4><p>For model documentation, the control could require that every production AI system maintain a model card or equivalent documentation that describes its intended use, training data characteristics, performance metrics, known limitations, and conditions under which it should not be used. The owner would be the AI product owner. Evidence would include completed model cards and a review log showing they are updated at defined intervals. NIST&#8217;s COSAiS project has specifically identified model documentation as a foundational element of AI security overlays, meaning this control will become increasingly expected by auditors as standards mature.</p><h4>Governance Controls</h4><p>For AI system inventory, the control might state that the organization maintains a comprehensive inventory of all AI systems in production or development, including third-party AI services, with defined metadata fields including risk classification, owner, deployment status, and last review date. The owner would be the Chief AI Officer, CTO, or equivalent. Evidence would include the AI inventory register, verification of completeness against procurement records and code repositories, and last-review documentation.</p><p>The pattern across all of these controls follows a consistent formula. Identify the risk. Write a control statement that describes the required behavior. Assign an owner who is accountable for the control operating effectively. Define the evidence that proves the control is working. Specify how an auditor would test it. This is not conceptually different from how you already manage access control or change management controls. You are simply applying the same discipline to AI-specific risks.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/building-an-ai-control-set-that-auditors?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/building-an-ai-control-set-that-auditors?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h2>Plugging AI into Your Existing Security and Privacy Programs</h2><p>With risk categories defined and controls drafted, the next challenge is integration. AI governance should not create a parallel compliance structure. It should extend the structures you already operate. Here is how to map AI controls into the major frameworks.</p><h4>SOC 2 Integration</h4><p>SOC 2&#8217;s Trust Services Criteria provide natural extension points for AI controls. Security (Common Criteria) already covers access controls, system operations, and change management. AI extensions include access controls for model artifacts, training data, and inference endpoints, as well as change management procedures that encompass model retraining and deployment. Availability criteria extend to AI system uptime, failover, and graceful degradation when models underperform. Processing Integrity is perhaps the most relevant criterion, covering accuracy, completeness, and timeliness of system processing. AI-specific controls for model accuracy monitoring, drift detection, and output validation map directly here. Confidentiality and Privacy criteria also cover AI-specific concerns such as training data, model memorization, and inference-time data exposure.</p><p>The practical advantage is that your SOC 2 auditor already tests these criteria. By framing AI controls as extensions of existing criteria rather than novel requirements, you make them testable within your existing audit cycle. You are not asking your auditor to learn a new framework. You are asking them to evaluate additional controls within a framework they already know.</p><h4>ISO 27001 Integration</h4><p>ISO 27001&#8217;s Annex A controls similarly accommodate AI extension. Asset management controls expand to include AI models, training datasets, and configuration artifacts as information assets. Access control extends to model registries, training pipelines, and inference APIs. Supplier management is expanding to encompass third-party AI model providers, including evaluations of their security practices, training data handling, and model update procedures.</p><p>For organizations pursuing both ISO 27001 and ISO 42001, the integration is even more direct. ISO 42001 was designed to interoperate with ISO 27001, and certification bodies are increasingly offering combined audits. Deloitte has noted that the ISO 42001 approach to AI management builds upon control frameworks that many organizations already have in place, including data governance, IT, security, privacy, enterprise risk management, and internal audit.</p><h4>NIST SP 800-53 Integration</h4><p>For organizations using NIST SP 800-53, the path forward is being paved by NIST itself through the COSAiS project. The overlays will address four major use cases: adapting and using generative AI with large language models; automating business workflows with predictive AI; single- and multi-agent AI systems; and controls for AI developers. Each overlay will tailor the existing 800-53 controls to address AI-specific concerns such as model integrity, data provenance, adversarial robustness, and transparency. Organizations using 800-53 today should monitor this project closely, as it will provide the most authoritative mapping between traditional security controls and AI-specific requirements.</p><h4>Privacy Program Integration</h4><p>AI systems present privacy challenges that extend beyond traditional data processing. Models can memorize and regurgitate training data. They can infer sensitive attributes from seemingly innocuous inputs. They create new personal data through classification and prediction. Your privacy program should extend data protection impact assessments to cover AI training and inference. Consent management processes should address whether individuals consented to AI processing specifically, not just data collection generally. Data subject rights procedures should account for the right to an explanation and the right not to be subject to solely automated decision-making under regulations such as GDPR.</p><h4>The Unified Control Register</h4><p>The goal of this integration exercise is a single control register, or a unified view across registers, where AI controls sit alongside security and privacy controls. When an auditor opens your control register, they should see AI controls in context, mapped to the same risk categories, using the same ownership model, and producing the same types of evidence as every other control in your program.</p><h2>A Simple Template: One-Page AI Control Register for Auditors</h2><p>What follows is a template for a one-page AI control summary that you can hand to your auditor for any AI system. This is not a replacement for detailed control documentation. It is a communication tool that provides auditors with the entry point they need to understand how your AI governance aligns with their testing program.</p><p>The template captures six fields for each AI system. First, AI System Identification, including the system name, a brief description of its function, the risk tier classification (low, medium, high, or prohibited), the designated system owner, and the deployment date and last review date. Second, the Risk and Control Summary, which is a table mapping each applicable risk category to the specific control, the control owner, the evidence type, and the framework mapping (such as SOC 2 CC7.2 or ISO 27001 A.12.4). Third, Data Governance, documenting training data sources, privacy assessment status, data retention and deletion policies, and third-party data agreements. Fourth, Monitoring and Observability, specifying the performance metrics tracked, drift detection methods and frequency, alerting thresholds and escalation paths, and the last monitoring review date. Fifth, Testing and Validation, including pre-deployment test results with dates, bias and fairness assessment results, adversarial and red team testing results, and scheduled re-testing cadence. Sixth, Incident History and Remediation, recording any AI-specific incidents, root cause analysis, and remediation actions taken, and any resulting control modifications.</p><p>The risk and control summary table is the centerpiece. Here is what it looks like for a hypothetical customer service chatbot powered by an LLM.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!E8HR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!E8HR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png 424w, https://substackcdn.com/image/fetch/$s_!E8HR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png 848w, https://substackcdn.com/image/fetch/$s_!E8HR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png 1272w, https://substackcdn.com/image/fetch/$s_!E8HR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!E8HR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png" width="579" height="503" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:503,&quot;width&quot;:579,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:45927,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://trustedai.substack.com/i/191479125?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!E8HR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png 424w, https://substackcdn.com/image/fetch/$s_!E8HR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png 848w, https://substackcdn.com/image/fetch/$s_!E8HR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png 1272w, https://substackcdn.com/image/fetch/$s_!E8HR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6cee07c-3632-4da2-9cb0-dc1f0de8cf97_579x503.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This single page gives your auditor a complete map of the AI system, the risks it presents, the controls in place, who owns them, what evidence exists, and how it all connects to the frameworks they are already testing. It transforms a conversation that might otherwise devolve into a machine-learning tutorial into a structured compliance discussion in terms both parties understand.</p><p>One practical note on evidence. The shift from traditional software controls to AI controls often requires new categories of evidence that auditors have not previously evaluated. Model monitoring dashboards, drift detection reports, bias testing results, and adversarial test logs are not yet standard audit artifacts. When you introduce these to your auditor, provide context. A one-page guide explaining what each type of evidence shows, how it is generated, and what good looks like will save hours of back-and-forth during fieldwork.</p><h2>How to Pilot This for One AI Use Case in 30 Days</h2><p>If you have read this far, you understand the framework. The question now is how to start. Here is a 30-day pilot plan that takes one AI use case from uncontrolled to audit-ready.</p><h4><strong>Days 1 through 5: Select and scope</strong></h4><p>Choose one production AI system, ideally one that is high enough risk to be meaningful but well enough understood to be manageable. A customer-facing chatbot, a fraud detection model, or an AI-assisted underwriting system is a good candidate. Document its purpose, data inputs, decision scope, and affected stakeholders. Assign an owner if one is not already defined.</p><h4><strong>Days 6 through 10: Assess risks and map to existing controls</strong></h4><p>Walk through the six-category taxonomy with the system owner and relevant technical leads. For each category, identify which risks are applicable and which existing controls already provide partial coverage. Document gaps where no controls currently exist. Pull up your current control register and note where AI-specific extensions are needed.</p><h4><strong>Days 11 through 18: Draft controls and assign owners</strong></h4><p>For each identified gap, draft a control statement following the formula described in Section 3. Assign an owner for each control. Define the evidence each control should produce and confirm that the evidence is either already being generated or can be generated with reasonable effort. Do not let perfection be the enemy of progress. A basic monitoring control with weekly manual review is better than a fully automated pipeline that will take six months to build.</p><h4><strong>Days 19 through 25: Collect and organize evidence</strong></h4><p>Gather evidence for each control. This is where you discover whether your controls are actually operational or merely aspirational. If you cannot produce evidence that a control is working, the control is not working. Common findings at this stage include monitoring tools configured but not reviewed, documentation that exists but is outdated, and ownership defined on paper but not exercised in practice. Address what you can and document what needs remediation.</p><h4><strong>Days 26 through 30: Package and review</strong></h4><p>Complete the one-page AI control register template for your pilot system. Share it with your internal audit team or compliance lead and ask them to evaluate it as an external auditor would. Their feedback will reveal gaps in clarity, evidence sufficiency, and framework mapping. Incorporate their input and finalize the template.</p><p>At the end of 30 days, you will have a working example of an AI control set that auditors can evaluate, a template that can be replicated across your AI portfolio, and a concrete understanding of the gaps between your current governance posture and what audit-ready AI governance requires. Just as important, you will have started the conversation between your AI teams and your compliance teams in a shared language.</p><h2>The Bottom Line</h2><p>AI governance does not need its own compliance religion. It needs to speak the language that your organization&#8217;s risk infrastructure already understands. The control frameworks are already there. The audit relationships are already there. The evidence management systems are already there. What is missing is the translation layer that connects AI-specific risks to these existing structures.</p><p>NIST recognized this when it chose to build AI security overlays atop SP 800-53 rather than create something entirely new. ISO recognized it by designing ISO 42001 to integrate with ISO 27001. The organizations that move fastest will be those that follow this same logic: extend what exists, do not rebuild from nothing.</p><p>The window for proactive action is narrowing. The EU AI Act&#8217;s requirements continue to phase in through 2027. NIST is actively developing AI-specific control overlays that auditors will soon reference. SOC 2 auditors are already beginning to ask questions about AI systems within existing engagements. The organizations that have their AI control sets defined, documented, and integrated will answer those questions with confidence. The organizations that do not will find themselves in the reactive, expensive position of having to retrofit governance under pressure.</p><p>Start with one system. Build the template. Prove the pattern. Then scale it across your AI portfolio. The goal is not to create a perfect AI governance program overnight. The goal is to establish the structure that makes responsible AI auditable, scalable, and sustainable.</p><p>Policy alone cannot deliver trusted AI. You also need controls that auditors can test, evidence they can evaluate, and owners they can talk to. Build that bridge, and you will find that AI governance becomes less of a burden and more of a capability that accelerates everything else.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://trustedai.recodework.com/p/building-an-ai-control-set-that-auditors/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://trustedai.recodework.com/p/building-an-ai-control-set-that-auditors/comments"><span>Leave a comment</span></a></p>]]></content:encoded></item></channel></rss>