Evaluating Generative AI: Practical Tests for Safety, Bias and Fairness

How to assess toxicity, hallucinations, representational harms and group fairness

If your AI evaluation strategy still centers on accuracy scores and latency benchmarks, you are measuring the easy while ignoring the consequential.

Accuracy tells you whether a generative model can produce a correct answer. It tells you nothing about whether that model will fabricate a citation to a nonexistent court case, generate a recruitment summary that systematically favors one demographic over another, or respond to a vulnerable user with language that causes real harm. These are not hypothetical failure modes. They are documented incidents, and their frequency and severity are increasing.

Stanford HAI’s 2026 AI Index Report found that documented AI incidents rose to 362 in 2025, up 55% from 233 in 2024. Among organizations that experienced incidents, those reporting three to five incidents jumped from 30% to 50%. At the same time, organizations rating their incident response capability as “excellent” dropped from 28% to 18%. The gap between what we are deploying and what we can responsibly manage is widening, not closing.

The root cause is an evaluation gap. Almost all leading frontier model developers report results on capability benchmarks like MMLU and SWE-bench. But reporting on responsible AI benchmarks, those covering safety, bias, fairness and harmful outputs, remains sparse and inconsistent. We measure what models can do with great precision. We measure what models should not do with almost none.

Beyond the fundamentals of AI metrics and testing methodology, the socio-technical dimensions determine whether a generative AI system is safe to deploy and equitable in its impact. This article will define the four primary failure modes, walk through practical tests you can run for each, explain how to interpret results without overreacting or underreacting, and lay out a prioritization framework for deciding what to fix and when.

This is not academic theory. It is an operational playbook for the evaluation work that separates responsible AI programs from aspirational ones.

What Can Go Wrong in Generative Models

Traditional software fails in predictable ways. It throws an error, returns null, or crashes. Generative AI fails in ways that are far more insidious because the outputs often look plausible, fluent and authoritative even when they are wrong, harmful or biased. Understanding the taxonomy of failure modes is the first step toward evaluating them.

Toxic and harmful output

Generative models can produce content that is abusive, threatening, sexually explicit or otherwise harmful. This includes direct toxicity, where the model generates overtly offensive language, and indirect toxicity, where the model follows harmful instructions embedded in adversarial prompts or produces content that normalizes violence or self-harm. A customer-facing chatbot that generates a single toxic response over 10,000 interactions may seem statistically insignificant. Still, for the customer on the receiving end, it is the only interaction that matters. And when that interaction surfaces on social media, the statistical argument provides no protection.

Hallucinations and confabulation

Generative models produce content that is factually incorrect, fabricated or internally inconsistent while presenting it with the same confidence as accurate information. NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework, published in July 2024, identifies confabulation as one of twelve distinct risk categories for generative AI systems. The scale of the problem is significant. On Stanford’s AA-Omniscient Index, a benchmark designed to assess whether models will admit uncertainty rather than guess, hallucination rates across 26 top models ranged from 22% to 94%. In legal queries, Stanford researchers found hallucination rates between 69% and 88%. In medical case summaries, hallucinations reached 64% without mitigation prompts. These are not edge cases. They are systemic characteristics of how these models operate.

Representational harms

These occur when generative models produce outputs that reinforce stereotypes, erase or underrepresent certain groups, or associate demographic characteristics with negative attributes. A 2023 analysis of over 5,000 images created with generative AI tools found that they amplified both gender and racial stereotypes relative to real-world distributions. When a model consistently depicts engineers as men, nurses as women, or associates certain ethnicities with poverty, it is not merely reflecting existing data. It is reinforcing and amplifying patterns of exclusion. Representational harms are particularly dangerous because they are often subtle enough to pass casual review but pervasive enough to shape perceptions at scale.

Group fairness failures

Even when individual outputs appear acceptable, a model may systematically perform differently across demographic groups. This might manifest as higher error rates for certain languages or dialects, less helpful responses for users from particular backgrounds, or recommendation patterns that advantage some groups over others. Recent research has shown that even when models provide equal access (zero differences in refusal rates across demographic groups), they can still exhibit systematic disparities in interaction quality, including differences in sentiment, hedging, and the degree of confidence expressed in responses to users of different ages, genders, or nationalities.

These four failure modes are not mutually exclusive. A model that hallucinates disproportionately in responses to users from certain backgrounds combines confabulation with group fairness failure. A model that produces stereotyped content in response to adversarial prompts exhibits both representational harm and a safety vulnerability. Effective evaluation must address all four dimensions and their intersections.

Practical Tests You Can Run

Evaluation frameworks for generative AI have matured significantly over the past two years. What follows is not a comprehensive literature review but a practical guide to the tests that deliver the most actionable insight per unit of effort. Each section focuses on a specific failure mode and outlines approaches that range from straightforward (suitable for teams just beginning systematic evaluation) to sophisticated (appropriate for organizations with dedicated responsible AI programs).

Toxicity and Safety Testing

The foundation of safety evaluation is structured red-teaming, the practice of systematically probing a model with inputs designed to elicit harmful outputs. Stanford’s HELM Safety benchmark provides a standardized framework that includes HarmBench for jailbreak resistance, BBQ for social discrimination, SimpleSafetyTest for basic safety behaviors, XSTest for the balance between helpfulness and harmlessness, and the AnthropicRedTeam evaluation for resilience against adversarial probing.

For organizations building their own evaluation programs, start by assembling prompt sets that cover the categories of harm most relevant to your deployment context. A healthcare application faces different toxicity risks than an internal productivity tool. Your prompt library should include direct requests for harmful content to test baseline refusals, adversarial prompts that attempt to circumvent safety controls through role-playing or hypothetical framing, multi-turn conversations that gradually escalate toward harmful territory, and prompts that embed harmful instructions within a seemingly benign context.

Critically, do not limit testing to English. The Stanford 2026 AI Index found that HELM Arabic, a regionally developed model, outscored leading frontier models, and performance gaps widened significantly at the dialect level. If your users speak multiple languages, your safety testing must cover all of them. Models that perform well on English safety benchmarks may exhibit significantly degraded safety behavior in other languages.

Automated toxicity classifiers, such as Perspective API or specialized toxicity detection models, can score outputs at scale. But automated classifiers have known limitations, including higher false positive rates for text discussing marginalized groups and difficulty detecting subtle toxicity. Use automated scoring to identify patterns and flag concerning outputs, but always supplement with human review for final assessment.

Groundedness and Hallucination Testing

Testing for hallucination requires a different approach than testing for toxicity because the failure mode is not about harmful intent but about factual accuracy and calibration.

The most direct test is a grounded question-answering evaluation. Provide the model with a source document and ask questions whose answers are contained in that document. Measure how often the model introduces information not present in the source, contradicts statements in the source, or fabricates citations, statistics, or quotations. This approach tests factual fidelity in a controlled setting where the ground truth is known.

For open-ended generation, the evaluation is harder but not impossible. Select a representative sample of outputs and have subject matter experts verify factual claims. Track the rate of verifiable errors, the types of errors (minor factual inaccuracies versus completely fabricated information), and the model’s confidence calibration (whether appropriate uncertainty is expressed). Pay particular attention to how the model handles questions at the boundary of its knowledge. A model that confidently answers questions it should not be able to answer is more dangerous than one that occasionally gets facts wrong but signals uncertainty.

The AA-Omniscient benchmark approach is instructive here. It does not just ask whether models get facts right. It asks whether models will admit when they do not know something. That distinction matters enormously in enterprise deployments, where users may not have the expertise to verify model outputs independently.

Consider building domain-specific hallucination test sets that reflect your actual use cases. If your model generates financial summaries, create a set of ground-truth financial documents and measure confabulation rates against them. If it supports customer service interactions, compile verified product information, and test whether the model invents features, pricing, or policies that do not exist. Generic benchmarks establish a baseline. Domain-specific tests tell you whether the model is safe for your particular deployment.

Bias and Representational Harm Testing

Counterfactual testing is the workhorse methodology for detecting bias in generative models. The concept is straightforward. Take a prompt, change a single demographic attribute (a name, a pronoun, a nationality, a reference to religion or ethnicity), and compare the outputs. If the model produces meaningfully different responses when the only variation is a demographic characteristic, you have identified a potential bias.

For example, create pairs of prompts that describe identical professional qualifications but use names commonly associated with different ethnic backgrounds. Ask the model to evaluate the candidates, write recommendation letters, or suggest career paths. Measure whether the quality, enthusiasm, or substance of the responses differs systematically across demographic groups. Research published at ICLR 2025 formalized this approach for large-scale testing, demonstrating methods to quantify the probability of unbiased responses across counterfactual prompt sets.

The SCOPE dataset, developed for systematic fairness evaluation, illustrates the scale at which this testing can be conducted. It contains over 241,000 prompts organized into more than 120,000 counterfactual pairs spanning nine bias dimensions and over 1,500 demographic groups. While you do not need to replicate this scale, the framework is useful. Your counterfactual tests should vary along the dimensions most relevant to your use case and cover the demographic groups most likely to be affected by your deployment.

Beyond counterfactual testing, examine your model’s default representations. When given open-ended prompts about professions, activities or social roles, what demographic patterns does the model default to? If asked to “write a story about a nurse,” does the model default to female characters? If asked to “describe a successful entrepreneur,” does it default to particular age, gender or ethnic characteristics? These defaults reveal learned associations that may not surface in direct questioning but shape the model’s output in pervasive ways.

Tools like LangFair and BEATS (Bias Evaluation and Assessment Test Suite) can automate portions of this work. LangFair handles use-case-specific assessments, including fairness-through-unawareness checks, counterfactual generation, and toxicity calculations directly from model outputs. BEATS offers 29 metrics spanning demographic, cognitive, and social biases. However, 2025 data revealed that nearly 38% of outputs from top industry models still exhibited some form of bias, underscoring that tooling alone does not eliminate the problem. It surfaces it for human judgment.

Slice-Based Evaluation for Group Fairness

Whereas bias testing assesses whether a model treats different groups differently on the same task, group fairness evaluation assesses whether a model performs equitably across the population of users it actually serves. This requires what practitioners call slice-based evaluation, which means breaking down aggregate performance metrics by meaningful subgroups rather than relying on overall averages.

Aggregate performance metrics can mask significant disparities. A model might achieve 95% accuracy overall while performing at 85% for users in certain geographic regions, age groups, or language communities. If you only report the aggregate number, you never see the gap.

Start by identifying the dimensions along which performance might vary in your specific context. Language and dialect are almost always relevant. Geography, age, technical sophistication, and accessibility needs (such as reliance on screen readers or simplified interfaces) are frequently important. For each dimension, measure core performance metrics, including accuracy, helpfulness ratings, error rates, and response quality, and compare across groups.

Recent empirical research adds an important nuance. Access-level fairness (does the model respond at all?) is necessary but insufficient. Researchers have demonstrated that models can provide zero-refusal-rate equity, responding to all demographic groups equally, while still exhibiting systematic disparities in interaction quality. One study found that GPT-4 expressed significantly higher hedging toward younger male users, while another model showed broader sentiment variation across identity groups. The model responds to everyone, but not everyone gets the same quality of response.

This means your group fairness evaluation should go beyond binary measures (did the model respond? was the response toxic?) and assess qualitative dimensions such as response helpfulness, specificity, confidence and tone.

Human Review for Edge Cases and Ambiguity

Automated evaluation handles scale. Human review handles judgment. Both are necessary, and neither is sufficient on its own.

Establish a structured human review process for outputs that automated systems flag as borderline, for categories where automated classifiers have known limitations, and for novel failure modes that your automated pipeline was not designed to detect. Human reviewers should be diverse in their backgrounds and perspectives, trained on the specific harm categories relevant to your deployment, and empowered to escalate issues that do not fit into existing categories.

Pay particular attention to intersectional cases, outputs where the harm emerges from the combination of multiple factors rather than any single dimension. A response might be neither toxic nor biased in isolation but become problematic in the context of a vulnerable user, a high-stakes decision, or a specific cultural setting. These cases resist automated classification and require human judgment informed by a genuine understanding of the affected communities.

How to Interpret Results

Raw evaluation results are necessary but not sufficient for decision-making. The critical skill is interpretation, which means understanding what scores mean, where they mislead, and how they should inform action.

One bad score does not tell the whole story

A model that scores poorly on a generic safety benchmark may perform well in your specific deployment context because your guardrails, system prompts and use case constraints mitigate the weaknesses the benchmark measures. Conversely, a model that scores well on benchmarks may fail in production because your use case exposes vulnerabilities that the benchmarks do not cover. Benchmarks are starting points for investigation, not final verdicts.

Distinguish severity, frequency, and user impact

A hallucination rate of 15% sounds alarming as a headline number. But the operational significance depends entirely on what the model is doing and who is affected. A 15% hallucination rate in a creative brainstorming tool is a different risk than the same rate in a medical information system. When interpreting results, always ask three questions. How severe is the potential harm from this failure? How frequently does it occur in realistic usage patterns? And how many users are exposed to the risk?

Some harms are rare but catastrophic

Others are common but subtle. Toxic outputs may occur in only 0.1% of interactions but generate outsized reputational and legal consequences. Mild representational biases may appear in 30% of outputs, individually causing minimal harm but cumulatively shaping perceptions over time. Your evaluation framework must capture both types. The rare-but-severe failures justify intensive safety testing and robust guardrails. The common-but-subtle failures justify ongoing monitoring and systematic measurement.

Context changes everything

Stanford’s 2026 AI Index revealed a striking finding about context sensitivity. When a false statement is presented as something another person believes, models handle it well. When the same false statement is presented as something the user believes, performance collapses. The model’s behavior depends not just on what it is asked but on the framing, the conversational context, and the relationship dynamics implied by the prompt. Your evaluation should test across the contexts your users actually encounter, not just clean benchmark conditions.

Watch for trade-offs between dimensions

Recent research cited in the Stanford report found that improving one dimension of responsible AI can degrade another. Improving safety sometimes reduces accuracy. Improving privacy can reduce fairness. These trade-offs are real and not always intuitive. When you observe an improvement in one evaluation dimension, check whether it comes at a cost elsewhere. Optimizing for a single metric while ignoring others is how organizations create new problems while solving old ones.

How to Prioritize Fixes

Not every finding from an evaluation requires the same response, and not every response needs to happen before launch. The goal is a structured prioritization framework that directs resources where they will do the most good while maintaining an honest assessment of residual risk.

Rank by severity, likelihood, and exposure

Severity measures the potential harm of a failure. Likelihood measures how frequently the failure occurs under realistic conditions. Exposure measures the number of users or stakeholders who could be affected. A failure that is severe, likely and widely exposed demands immediate attention. A failure that is severe, but extremely rare, and narrowly exposed may warrant monitoring rather than urgent remediation.

Separate “must fix before launch” from “monitor after launch”

Some findings are deployment blockers. If your model produces toxic outputs at a meaningful rate in response to common user queries, this must be resolved before production. If your model exhibits mild representational biases in a narrow category of edge-case prompts, this can be documented, monitored, and addressed through iterative improvement after deployment. The distinction is not a license to ignore issues but a recognition that demanding perfection before deployment means nothing ships, and the value of the AI system goes unrealized while incremental improvement is indefinitely deferred.

Match the intervention to the problem

Different failure modes require different fixes. Toxicity and safety failures often respond well to guardrails, which are output filters, content classifiers and system-prompt constraints that operate as a layer on top of the base model. Hallucination problems may require retrieval-augmented generation, grounding mechanisms, or citation requirements that change the architecture rather than just the output filtering. Bias and representational harms may require fine-tuning on more balanced data, adjusting training procedures, or redesigning prompt templates. Group fairness issues may demand changes to the product design itself, such as offering alternative interaction modes for underserved populations or adjusting how the model handles diverse languages and dialects.

Consider this hierarchy of interventions, ordered by increasing effort and impact. Product-level changes, such as restricting the model’s scope or adding human review to high-stakes outputs, are fast to implement and often highly effective. Guardrails and output filtering provide broad protection but can reduce helpfulness if too aggressive. Prompt engineering and system-level constraints can reshape model behavior without retraining. Fine-tuning adjusts model behavior at a deeper level but requires additional data and compute. Full retraining or model replacement is the most resource-intensive option and is typically reserved for fundamental inadequacies that cannot be addressed through other means.

Document residual risk honestly

After remediation, some risks will remain. Document them. Specify what was tested, what was found, what was fixed, what remains, and how the remaining risks will be monitored. This documentation serves multiple purposes. It provides transparency for governance and compliance review. It establishes the baseline against which future improvements will be measured. And it creates institutional memory that prevents the next team from repeating the same evaluation work without building on what was already learned.

What Metrics Miss

Even a rigorous evaluation program based on the methods described above has fundamental limitations. Acknowledging these limitations is not a reason to abandon measurement. It is a reason to complement measurement with broader forms of evaluation.

Metrics capture what can be measured, not necessarily what matters most

Toxicity classifiers can detect overtly harmful language. They struggle with subtle dehumanization, patronizing tone, or cultural insensitivity that does not register as “toxic” on a classifier but causes genuine harm to affected users. Bias metrics can detect differential treatment across demographic groups. They cannot capture whether the treatment any group receives is good enough in absolute terms. A model that is equally unhelpful to everyone is “fair” by many metrics but valuable to no one.

Evaluation reflects the evaluator’s perspective

The harm categories you test for, the demographic groups you include, and the severity thresholds you set all reflect choices made by the evaluation team. If your team lacks diversity of background and perspective, your evaluation will have corresponding blind spots. This is not a criticism of the people involved. It is a structural limitation that can only be addressed by deliberately broadening the perspectives that inform evaluation design.

Static evaluation misses dynamic harm

Benchmarks capture model behavior at a point in time. Real-world deployment is dynamic. User populations shift. Adversarial techniques evolve. Cultural norms change. An evaluation that declares a model safe in January may not account for a new adversarial attack technique that emerges in March or a shift in user demographics that exposes previously untested failure modes. This is why NIST AI 600-1 and the broader AI RMF emphasize continuous monitoring as a complement to pre-deployment testing. Evaluation is not a one-time gate. It is an ongoing practice.

Context, workflow, and user population change the meaning of results

A 5% hallucination rate means something very different when the model is summarizing meeting notes for internal use than when it is generating patient discharge instructions. The same bias pattern carries different weight when the model serves as an optional brainstorming tool versus a primary decision-support system. Evaluation results must always be interpreted in the context of how the model is actually used, by whom, and with what stakes.

The Bridge to Deeper Evaluation

The practical tests described in this article will significantly improve your ability to identify and address the most common failure modes in generative AI. But there are dimensions of impact that operational evaluation alone cannot capture.

Who benefits from your AI system, and who bears the risks? Are the communities most affected by potential harms represented in your evaluation process? How does your system interact with existing power structures, and does it amplify or mitigate existing inequities? What happens downstream when the outputs of your system are used as inputs to other decisions, other systems, other people’s lives?

These are questions that a benchmark cannot answer. They require engagement with affected communities, interdisciplinary analysis spanning technical, legal, social and ethical expertise, and a willingness to examine not just model outputs but also the systems of power and incentives within which models operate. Researchers increasingly describe this as a socio-technical evaluation, recognizing that AI systems are not isolated technical artifacts but components of complex social systems whose impacts can only be understood in context.

The Stanford 2026 AI Index highlights this gap. Frontier model developers consistently report capability benchmarks. Responsible AI benchmarks are reported sporadically. And the deeper questions of social impact, distributional effects and long-term consequences are rarely addressed in standardized evaluation. Responsible AI is not keeping pace with AI capability, with safety benchmarks lagging and incidents rising sharply.

Closing that gap requires moving beyond the question of “does this model produce harmful outputs?” toward the question of “does this model produce equitable outcomes in the real-world context where it operates?” The former is a testing problem. The latter is a governance problem, a design problem, and ultimately a question about organizational values.

Methods for conducting this deeper evaluation include stakeholder engagement frameworks, participatory evaluation approaches and techniques for assessing downstream and cumulative impacts. These socio-technical assessments are where Trusted AI moves from risk mitigation to genuine responsibility.

Where to Start

If you are reading this and your organization does not have a structured evaluation program for generative AI, here is a pragmatic starting point.

First, inventory your deployed generative AI systems and classify them by risk level.

High-stakes applications affecting individuals’ access to services, employment, credit, healthcare, or safety warrant the most rigorous evaluation. Internal productivity tools warrant a lighter but still structured assessment.

Second, for your highest-risk systems, implement the basic versions of each test described above.

Assemble a set of prompts for safety testing. Run a grounded question-answering evaluation for hallucination rates. Conduct counterfactual prompt testing for at least the most salient demographic dimensions. Break out your performance metrics by user segment rather than relying on aggregate numbers.

Third, establish a triage process for findings.

Use the severity-likelihood-exposure framework to separate deployment blockers from monitoring priorities.

Fourth, document everything.

Your evaluation methodology, your findings, your remediation decisions, and your residual risk assessment. This documentation is the foundation of governance and the basis for demonstrating due diligence to regulators, auditors, and stakeholders.

Fifth, commit to iteration.

Your first evaluation will be imperfect. Your prompt sets will have gaps. Your thresholds will need calibration. That is expected. The organizations that build effective evaluation programs are not the ones that get it perfect on the first pass. They are the ones who treat evaluation as a continuous practice, learning from each cycle and improving their methods over time.

The question is not whether generative AI carries risks. The evidence on that point is overwhelming. The question is whether your organization has the evaluation practices to identify those risks before they become incidents, and the governance structures to act on what you find.

The organizations that answer yes will be the ones that capture AI’s full potential while managing its real consequences. The organizations that answer “we’ll get to it” are the ones contributing to next year’s incident count.

Start now. Measure what matters. And build from there.

But recognize that model-level evaluation, however rigorous, is only half the work. The harder question is not how your model behaves in a test harness. It is how your system affects the people and communities it touches, who benefits, who bears the risk, and whether the structures around your AI are designed to hear from those who experience its consequences firsthand.

That is where evaluation becomes governance, and where governance becomes genuine accountability.