Running Practical AI Risk Assessments on Real Projects
From Framework to Field: A Hands-On Guide to Evaluating AI System Risks
Every AI governance framework emphasizes risk assessment. But when you sit down with a real AI project, the pristine categories and abstract principles can feel disconnected from the messy reality in front of you. How do you actually determine whether your AI system poses “high” or “medium” risk? How do you quantify the likelihood of a model producing biased outputs? How do you know if your mitigations are sufficient after you have identified a risk?
This is where theory meets practice. To demonstrate the practical steps that translate governance principles into operational reality, this post walks through an end-to-end risk assessment of a real AI use case for a credit decisioning system. It also explores the AI System Impact Assessment (AIIA), a mandatory requirement under ISO 42001 that represents the most substantial documentation effort in building a compliant AI management system.
The goal is not to provide a template to copy blindly, but it’s to illustrate the thinking process that separates rigorous risk assessment from checkbox compliance.
Use Case: Credit Decisioning
AI-powered credit scoring represents one of the most consequential applications of machine learning in financial services. These systems analyze applicant data to predict creditworthiness, influencing who receives loans, at what rates, and under what terms. The decisions affect individuals’ ability to buy homes, start businesses, and manage financial emergencies.
Credit decisioning also sits squarely in the regulatory crosshairs. Under the EU AI Act, AI systems intended to evaluate the creditworthiness of natural persons or establish their credit score are explicitly classified as high-risk under Annex III. This designation triggers comprehensive requirements for risk management, data governance, transparency, human oversight, and technical documentation. Compliance obligations for these high-risk systems will be fully enforceable by August 2026, with organizations already expected to be building toward compliance.
For our purposes, let’s consider an AI credit decisioning system for a mid-sized regional bank. The system uses machine learning models trained on historical lending data, supplemented with alternative data sources, to generate credit risk scores and make preliminary approve/review/decline recommendations. Human loan officers retain final decision authority, but the AI handles initial screening and prioritization.
This use case captures the essential tensions in AI risk management: the efficiency gains and potential for fairer decisions on one side, and the risks of bias, opacity, and harm to individuals on the other.
The Foundation: Distinguishing Risk Assessment from Impact Assessment
Before diving into methodology, it’s essential to understand a critical distinction that many organizations overlook: the difference between an AI risk assessment and AI impact assessment.
An AI Risk Assessment (as required by ISO 42001 Clause 6.1) focuses on risks to your organization. It’s the operational disruptions, regulatory penalties, reputational damage, and financial losses. It asks: What could go wrong with this AI system, and how would it affect our business objectives?
An AI System Impact Assessment (AIIA) (addressed through ISO 42001’s control A.5 and detailed in ISO 42005) focuses on impacts to individuals, groups, and society. It’s the harm to fundamental rights, discrimination, erosion of autonomy, and broader societal effects. It asks: How could this AI system affect the people it touches?
You need both. They complement each other but serve different purposes and require different analytical approaches. The AIIA, in particular, represents the most substantial documentation effort when building an ISO 42001-conformant AI management system. While your organizational risk assessment might run 10 to 20 pages, a comprehensive AIIA can extend to 60 or 70 pages of detailed analysis.
The confusion between these assessments leads organizations to either duplicate effort or leave critical gaps. A robust risk management program treats them as two lenses on the same system, each revealing different aspects of the risk landscape.
Step 1: Define the Use Case and System Boundaries
Every risk assessment begins with a clear articulation of what you’re assessing. Vague scope leads to vague analysis. For the credit decisioning system, we need to document several critical elements.
Business Purpose and Problem Statement
The system aims to improve credit decision accuracy while reducing time-to-decision and enabling the bank to serve more customers, including those with limited traditional credit histories. The business case rests on three pillars: reduced default rates through better risk prediction, faster customer experience, and expanded access to credit for underserved populations.
System Architecture and Data Flows
The AI system ingests applicant information from loan applications, credit bureau data, bank account history (for existing customers), and alternative data sources, including utility payment records and rent payment history. A gradient boosting model generates a probability-of-default score, which is then combined with business rules to produce an initial recommendation. All recommendations flow to loan officers who make final decisions, with mandatory human review for applications near decision thresholds.
Stakeholder Identification
Direct stakeholders include loan applicants (whose financial lives are affected by decisions), loan officers (whose workflows and authority are changed), and the bank itself (which bears financial and reputational risk). Indirect stakeholders include regulators, communities affected by lending patterns, shareholders, and the broader financial system that depends on sound credit allocation.
Deployment Context
The system will operate within the United States, subject to the Equal Credit Opportunity Act (ECOA), Fair Credit Reporting Act (FCRA), and applicable state regulations. For customers with EU residency, EU AI Act requirements also apply.
This foundational documentation is not bureaucratic overhead. It establishes the parameters that will guide every subsequent assessment decision.
Step 2: Identify AI-Specific Risks
With the use case clearly defined, we turn to systematic risk identification. AI systems present risks that differ fundamentally from traditional software. They learn from data, adapt over time, and can exhibit emergent behaviors, and our risk identification must account for these unique characteristics.
Risk Category 1: Bias and Fairness
Risk Description: The model may produce systematically different outcomes for protected groups, leading to discriminatory lending patterns. This can occur through biased training data that reflects historical discrimination, proxy variables that correlate with protected characteristics, or model behaviors that amplify existing disparities.
Potential Manifestations:
Higher denial rates for minority applicants with equivalent creditworthiness
Differential pricing that charges higher rates to protected groups
Feature reliance on proxies like geography that correlate with race
Training data that underrepresents certain populations, leading to less accurate predictions for those groups
Risk Category 2: Transparency and Explainability
Risk Description: The model’s decision logic may be insufficiently interpretable, making it difficult to explain decisions to applicants, auditors, or regulators. This “black box” problem is particularly acute for gradient boosting and neural network models.
Potential Manifestations:
Inability to provide specific reasons for adverse credit decisions (required by FCRA)
Difficulty demonstrating compliance to examiners
Loan officer resistance due to the inability to understand and trust recommendations
Legal exposure from unexplainable adverse actions
Risk Category 3: Data Quality and Integrity
Risk Description: Model performance depends critically on data quality. Errors, missing values, outdated information, or data poisoning could compromise predictions. Alternative data sources introduce additional uncertainty about accuracy and completeness.
Potential Manifestations:
Inaccurate credit decisions due to erroneous input data
Model degradation as data distributions shift over time
Privacy violations from improper data handling
Security vulnerabilities enabling data manipulation
Risk Category 4: Model Drift and Degradation
Risk Description: AI models are not static. Their performance degrades over time as the relationship between input features and outcomes changes. Economic conditions, customer behavior, and data distributions are not fixed.
Potential Manifestations:
Declining prediction accuracy leading to increased defaults
Systematic bias emergence as demographic patterns shift
Failure to adapt to economic disruptions (recession, pandemic)
Silent degradation undetected until significant harm occurs
Risk Category 5: Operational and Integration Risks
Risk Description: Integrating an AI system into existing loan origination workflows introduces operational complexities. System failures, integration errors, or misalignment between AI outputs and human processes can cause harm.
Potential Manifestations:
System outages halting loan processing
Loan officers overriding AI recommendations inconsistently
Training gaps leading to improper use
Version control failures deploying untested models
Risk Category 6: Adversarial and Security Risks
Risk Description: AI systems can be targeted by adversarial attacks. Sophisticated actors may attempt to manipulate inputs, poison training data, or exploit model vulnerabilities.
Potential Manifestations:
Applicants gaming the system with synthetic identities
Data poisoning compromising model integrity
Model extraction attacks stealing proprietary algorithms
Prompt injection or manipulation in system interfaces
Step 3: Assess Impact and Likelihood
Having identified risks, we now assess each one across two dimensions: impact (severity of consequences if the risk materializes) and likelihood (probability of occurrence). This assessment requires both quantitative analysis, where possible, and informed qualitative judgment, where necessary.
Defining Our Assessment Scales
Before scoring, we establish consistent scales that provide meaningful differentiation.
Impact Scale (1-5):
Likelihood Scale (1-5):
Risk Scoring Matrix
These scores represent inherent risk, which is the risk level absent mitigation controls. The next step is to identify the controls we can implement to reduce these risks to acceptable levels.
Step 4: Design and Implement Mitigations
Risk treatment follows four classic options: avoid (eliminate the activity causing risk), mitigate (reduce impact or likelihood through controls), transfer (shift risk to another party through insurance or contracts), or accept (acknowledge the residual risk as within tolerance). For most AI risks, mitigation through controls is the primary strategy.
Effective AI risk controls fall into four broad categories: governance and oversight controls, technical and security controls, operational process controls, and transparency and accountability controls. Let’s examine how each applies to the credit decisioning system.
Mitigations for Bias and Discrimination Risk
Technical Controls:
Implement fairness-aware machine learning techniques during model development
Deploy bias detection algorithms that continuously monitor outcomes across protected groups
Use disparate impact testing with thresholds aligned to regulatory standards (e.g., 80% rule)
Implement SHAP (SHapley Additive exPlanations) or LIME analysis to identify feature contributions that may proxy for protected characteristics
Conduct regular retraining on representative, bias-audited datasets
Governance Controls:
Establish a cross-functional model risk committee with authority to halt deployment
Require documented fairness assessments before any model goes to production
Create escalation paths for bias concerns from loan officers or customers
Engage external auditors for annual fair lending reviews
Operational Controls:
Train loan officers on fair lending requirements and AI limitations
Implement mandatory human review for all adverse decisions
Establish appeal processes that don’t simply re-run the same model
Residual Risk Assessment: With comprehensive controls, impact remains at 5 (consequences of materialization still severe), but likelihood reduces to 2 (extensive monitoring and testing make undetected bias unlikely). Residual risk score: 10 (High, but within tolerance given extensive monitoring).
Mitigations for Explainability Gaps
Technical Controls:
Implement SHAP values to generate feature-level explanations for each decision
Build a parallel interpretable model (logistic regression) that provides human-readable reasons
Create reason code libraries that translate model outputs to FCRA-compliant explanations
Document model architecture and feature engineering in technical specifications
Governance Controls:
Require explainability review as part of model validation process
Establish minimum explainability standards for production deployment
Train compliance and legal teams on AI explanation capabilities and limitations
Operational Controls:
Develop adverse action notices that incorporate AI-generated explanations
Train loan officers to explain decisions using provided reason codes
Create escalation procedures for applicants seeking additional explanation
Residual Risk Assessment: Impact reduces to 3 (explanation capabilities now meet regulatory minimums) and likelihood reduces to 2 (systematic processes ensure explanations are generated). Residual risk score: 6 (Medium, acceptable).
Mitigations for Data Quality Issues
Technical Controls:
Implement data validation rules at ingestion points
Deploy anomaly detection on input data streams
Establish data quality metrics with automated alerting
Create data lineage documentation for all training and inference data
Implement data versioning for reproducibility
Governance Controls:
Establish data governance policies covering acquisition, use, and retention
Define data quality standards for each data source
Require vendor due diligence for third-party data providers
Operational Controls:
Conduct regular data quality audits
Establish incident response procedures for data quality failures
Train operations staff on data quality monitoring
Residual Risk Assessment: Impact remains at 4 (data errors can still cause significant harm) but likelihood reduces to 2 (extensive monitoring catches issues before impact). Residual risk score: 8 (Medium, acceptable).
Mitigations for Model Drift
Technical Controls:
Implement real-time monitoring of prediction distributions
Deploy automated drift detection with configurable thresholds
Establish champion/challenger testing for model updates
Create automated retraining pipelines with human approval gates
Monitor feature importance stability over time
Governance Controls:
Define model lifecycle management policies including retirement criteria
Establish performance thresholds that trigger mandatory review
Require documented approval for model updates
Operational Controls:
Conduct monthly model performance reviews
Establish quarterly validation against holdout data
Create emergency procedures for rapid model replacement
Residual Risk Assessment: Impact remains at 4 (drift can cause significant harm if undetected) but likelihood reduces to 2 (continuous monitoring provides early warning). Residual risk score: 8 (Medium, acceptable).
Step 5: Document Residual Risk and Risk Acceptance
After implementing controls, we reassess our risk landscape. Residual risk is what remains after mitigation. It’s the risk the organization consciously accepts when deploying the AI system.
Residual Risk Summary
Risk Acceptance Decision
For each residual risk above the organization’s risk tolerance threshold, formal acceptance is required. In our case, the bias/discrimination risk at a residual score of 10 remains elevated and requires explicit acceptance by senior leadership with documented rationale.
Risk Acceptance Statement (Bias/Discrimination):
The AI Credit Decisioning System presents residual bias risk that, despite comprehensive mitigations, cannot be eliminated entirely. We accept this residual risk because: (1) the implemented controls significantly reduce likelihood of undetected bias; (2) human oversight on all adverse decisions provides additional protection; (3) continuous monitoring enables rapid detection and response; (4) the potential benefits of improved credit access for underserved populations outweigh the carefully managed risks; and (5) ongoing external audits provide independent verification. This acceptance is contingent on maintaining all specified controls and is subject to quarterly review by the Model Risk Committee.
Step 6: The AI Impact Assessment (AIIA) Under ISO 42001
The organizational risk assessment addresses risks to the bank. But ISO 42001 requires a separate, more extensive analysis: the AI System Impact Assessment.
The AIIA, detailed in ISO 42005, focuses on impacts to individuals, groups of individuals, and society. It’s analogous to a Data Protection Impact Assessment under GDPR, but broader in scope and typically more detailed, often running 60 to 70 pages for complex systems.
AIIA Structure Under ISO 42005
ISO 42005 structures the AIIA into seven sections:
Section A: System Purpose and Overview Documents the AI system’s intended purpose, the problem it solves, and its role in the broader business process. For example, automated credit risk assessment to support lending decisions while maintaining human decision authority.
Section B: Data and Training Information This is one of the most substantial sections, requiring assessment of each dataset across twenty characteristics defined in ISO 42005, including origin, collection methods, representativeness, quality measures, consent basis, and potential biases. For a credit model using multiple data sources, this section alone can run 20 or more pages.
Section C: Algorithms and Model Information Documents the origin of algorithms (internal development, third-party, or off-the-shelf), development approach, model architecture, and key design decisions. It includes an explanation of why particular techniques were selected and their known limitations.
Section D: Deployment Environment Covers geographic deployment, language considerations, technical infrastructure, integration points, and environmental constraints. It addresses where and how the system operates in practice.
Section E: Interested Parties (Stakeholders) Maps all stakeholders affected by the AI system, both directly (e.g., applicants, loan officers) and indirectly (e.g., communities, regulators, competitors). It documents their interests, concerns, and relationships to the system.
Section F: Benefits and Harms Analysis Examines actual and potential benefits and harms to each interested party across defined AI perspectives. This structured taxonomy ensures careful consideration of impacts, including fairness, privacy, safety, economic effects, autonomy and dignity.
For each identified harm, the analysis must address: the nature of the harm, who is affected, the mechanism by which harm occurs, the severity and likelihood, and potential cascade effects. Similarly for benefits.
Section G: Failures and Foreseeable Misuse Documents potential system failures (technical, operational, human) and how misuse might occur both accidental and intentional. This section links to disaster recovery and business continuity planning while also considering how bad actors might exploit the system.
Key AIIA Findings for the Credit System
Without reproducing a full seventy-page assessment, key impact findings would include:
Beneficial Impacts:
Faster credit decisions benefiting applicants
Potentially expanded credit access for thin-file consumers
More consistent decision-making reducing arbitrary variation
Efficiency gains enabling competitive pricing
Harmful Impacts (Potential):
Credit denial affecting housing, transportation, and economic opportunity
Privacy concerns from alternative data use
Autonomy reduction as algorithmic decisions shape financial lives
Dignity harm from impersonal rejection processes
Systemic risk if model failures affect large populations simultaneously
Disproportionate Impact Risks:
Populations underrepresented in training data may receive less accurate predictions
Applicants without digital footprints may be disadvantaged by alternative data reliance
Communities with historical discrimination may face compounded harm
The AIIA must document not just that these impacts are possible, but the specific mechanisms through which they might occur, the affected populations, and the mitigation measures in place.
Step 7: Continuous Monitoring and Reassessment
Risk assessment is not a one-time exercise. AI systems evolve, environments change, and new risks emerge. Effective governance requires ongoing monitoring and periodic reassessment.
Monitoring Framework
Real-Time Monitoring:
Model performance metrics (accuracy, precision, recall)
Prediction distribution stability
Feature importance changes
Data quality indicators
System availability and latency
Periodic Reviews:
Monthly: Performance dashboard review by model owners
Quarterly: Model Risk Committee review of all production models
Annually: Full risk reassessment and AIIA update
Triggered: Reassessment upon significant model changes, regulatory developments, or incident occurrence
Key Risk Indicators (KRIs)
Incident Response
When risks materialize, a structured response is essential. For our credit system, incident response procedures include: immediate notification to the model risk committee, impact assessment to determine the scope of affected decisions, decision on model suspension vs. continued operation, root cause analysis, corrective action implementation, and regulatory notification where required.
Connecting to the Broader Governance Framework
This risk assessment does not exist in isolation. It connects to and supports the broader AI governance operating model discussed previously.
Executive Oversight: Risk assessment findings inform board-level AI risk reporting and strategic decisions about AI deployment.
Policy Integration: Identified risks drive policy requirements for data governance, model validation, and human oversight.
Operational Embedding: Mitigation controls are embedded in operational workflows rather than maintained as separate compliance activities.
Continuous Improvement: Assessment findings feed back into governance maturity, identifying areas where policies, capabilities, or resources need strengthening.
Making Risk Assessment Work in Practice
Several practical lessons emerge from this process. Invest time upfront in defining the AI system’s purpose and boundaries. Vague scopes produce vague risks. Engage diverse perspectives because technical teams, business owners, compliance staff, and affected stakeholders each see different risks. Be honest about uncertainty; better to acknowledge what you don’t know than manufacture false precision. Document your reasoning, not just your conclusions. And build in iteration. Your first assessment will have gaps, so plan for reassessment as you learn how the system behaves in production.
The Path Forward
A practical AI risk assessment bridges the gap between abstract governance principles and operational reality. It transforms questions like “Is this AI trustworthy?” into specific, answerable inquiries: What are the concrete risks? How likely are they to materialize? What controls reduce those risks to acceptable levels?
The methodology outlined here applies whether you’re assessing a credit decisioning system, a healthcare triage algorithm, a hiring screening tool, or any other AI application. The specific risks differ, but the analytical approach remains consistent.
Policy alone cannot deliver trusted AI. You need continuous, rigorous assessment of how your systems actually behave. The AI Impact Assessment under ISO 42001 provides the framework, and this methodology shows how to execute it.
Start with your highest-risk AI systems. Apply this process. Learn from what you find. The question is not whether your AI systems need risk assessment. The question is whether you will conduct it rigorously on your own terms or superficially under regulatory pressure.
Choose rigor.
It's interesting how this reminds me of balancing poses in Pilates, always tougher in practise.