Skip to main content Skip to Footer

LATEST THINKING


Credit risk model monitoring

Accenture can help banks develop effective credit risk model monitoring to gauge the performance of their analytical models.

Overview

Analytical models, which process quantitative data and provide quantitative output, are highly-valued strategic assets in financial institutions today. But for many banks, the job of monitoring the models may fall to an understaffed analytics team. Often there are no formal policies around credit risk model monitoring in place and poorly performing models remain in production.

When it comes to gauging model performance, simple statistics and infrequent thorough checks are not the best measure or approach. Instead, banks need to consistently evaluate model performance using comprehensive indicators:

  • Model discrimination.

  • System or population stability.

  • Characteristic stability.

  • Actual versus expected, or calibration.

  • Score distribution analysis.

  • Override analysis.

Effective monitoring allows institutions to ensure their credit risk models are performing as they should be. Accenture has extensive experience in all aspects of model development, validation and monitoring, having delivered more than 500 risk analytics solutions. We have a deep understanding of the regulatory implications for banks, and can help accelerate and simplify the implementation of a robust model monitoring process.

 

Download PDFDownload the full report [PDF, 962 KB]

Background

In today’s financial institutions, analytical models, which process quantitative data and provide quantitative output, are high-value organizational and strategic assets. Because credit risk models are needed to run the business and comply with regulations, poor governance or management of these models can expose an organization to the risks of less than optimal business decisions, regulatory fines and reputational damage.

To ensure that their models are meeting their business needs, organizations should develop a model monitoring standards document that includes:

  • Enterprise level model inventory. Take stock of the models in use and establish clear ownership of the maintenance and usage of the models.  

  • Robust data monitoring processes. Consider and create rules for each raw data field to help ensure the quality and consistency of the data.  

  • Governance structure. Make sure the model monitoring team works independently from the model development team. There also needs to be effective model audit processes and procedures in place, and senior management must be engaged and involved.

Ongoing monitoring of models is essential to evaluate whether changes in products, exposures, activities, customers or market conditions call for adjustment, redevelopment or replacement of the model, or to verify that any extension of the model beyond its original scope is valid. Any limitations or assumptions identified when a model is developed should be assessed as part of ongoing monitoring.

Analysis

When it comes time to gauge the performance of credit risk models, organizations often use a few simple metrics for regular monitoring purposes, such as the Kolmogorov-Smirnov statistic and the Gini Coefficient, leaving more comprehensive checks to be conducted infrequently. A better approach is to evaluate credit risk model performance on a regular basis using the following comprehensive indicators:

  • Model discrimination. The ability of the model to differentiate between events and non-events based on its input values, such as defaults and non-defaults.  

  • System or population stability. How different is the current data being scored by the model, compared to the data from the model development; is the model stable over time?  

  • Characteristic stability. How different is the distribution of the current population in each explanatory variable in the model compared to the population used for model development?

  • Actual versus expected, or calibration. Does the model deliver accurate point predictions?  

  • Score distribution analysis. Does the model generate large concentrations at particular deciles or credit grades?  

  • Override analysis. To what extent and why are there judgmental overrides over and above the raw model output?

Numerous metrics can be used to set up a comprehensive model monitoring system, but these should be incorporated into an effective management information system (MIS) program. Industry best practices suggest that the model risk monitoring MIS be thorough and timely, offer coverage, provide data/formula integrity and validation, use thresholds, provide an overall assessment, and include clear and relevant insights and commentary.

Recommendations

A strong credit risk model monitoring process is not only required by regulations, but has proven to be a potent competitive advantage for organizations that have taken the steps to:

  • Track model performance diligently.

  • Escalate and resolve model issues.

  • Involve senior management in decision-making.

  • Fine-tune models on a timely basis.

  • Maintain well-documented logs and rationales of changes.

Effective monitoring allows institutions to closely control and better empower the strategic risk management tools for day to day operational management as well as for purposes of calculating capital requirements.

Accenture has extensive experience in all aspects of model development, validation and monitoring. Our team has delivered more than 500 risk analytics solutions, and has a deep understanding of the regulatory implications of Basel II and III, Dodd-Frank, Federal Reserve and UK Prudential Regulation Authority (PRA) requirements. We have also developed proprietary assets that can help accelerate and simplify the implementation of a robust model monitoring process.

Authors

Larry Lerner is a managing director, Accenture Digital. Based in Washington, he leads Accenture Analytics and is a member of the group’s leadership team. He has extensive consultancy and enterprise experience in Financial Services and Analytics and has built industrialized underwriting, marketing and credit risk management capabilities. Over the years he has held several leadership roles in business intelligence, analytics, banking payment and capital markets enterprises and now guides clients on their journey to high performance.

Parvez Shaikh is a managing director, Accenture Digital. Based in Bengaluru, Parvez leads the Risk Analytics practice in India serving global Financial Services clients. His areas of experience include risk modeling and quantification, stress testing, pricing, valuation, economic capital, provisioning/loss forecasting and development/validation of risk rating scorecards, frameworks and methodologies. Parvez also has broad exposure to global financial services regulation with a focus on North America where he has held leadership roles in risk management among global financial services firms.

Tadhg O’Suilleabhain is a senior manager, Accenture Digital. Based in Dublin, Tadhg has over 13 years of enterprise and consultancy experience in risk management and analytics across global retail banking and capital markets sectors. His experience extends to risk model building and developing and validating Basel risk models (PD, EaD, LGD), impairment models and risk/reward models in both retail and non-retail settings. He also has extensive policy writing experience in both Basel and impairment model risk management settings.

Siddhartha Chatterji is a senior manager, Accenture Digital. Based in Gurgaon, India, Siddhartha has 12 years of global risk and marketing analytics experience. He has worked in the areas of Basel II analytics, risk scorecard development and lifecycle management covering retail and commercial risk, as well as core risk management roles in retail and commercial risk. He holds a B.A. (H) in Economics from St Stephens College Delhi, a Master’s Degree in Economics from the Delhi School of Economics and an MBA in Finance & Marketing from ISB Hyderabad.

Stay In The Know

Receive e-mails from Accenture featuring new content that matches your interests.

Visit the subscription center to make your selections and subscribe to New from Accenture.