Analytical models, which process quantitative data and provide quantitative output, are highly-valued strategic assets in financial institutions today. But for many banks, the job of monitoring the models may fall to an understaffed analytics team. Often there are no formal policies around credit risk model monitoring in place and poorly performing models remain in production.
When it comes to gauging model performance, simple statistics and infrequent thorough checks are not the best measure or approach. Instead, banks need to consistently evaluate model performance using comprehensive indicators:
System or population stability.
Actual versus expected, or calibration.
Score distribution analysis.
Effective monitoring allows institutions to ensure their credit risk models are performing as they should be. Accenture has extensive experience in all aspects of model development, validation and monitoring, having delivered more than 500 risk analytics solutions. We have a deep understanding of the regulatory implications for banks, and can help accelerate and simplify the implementation of a robust model monitoring process.
Download the full report [PDF, 962 KB]
In today’s financial institutions, analytical models, which process quantitative data and provide quantitative output, are high-value organizational and strategic assets. Because credit risk models are needed to run the business and comply with regulations, poor governance or management of these models can expose an organization to the risks of less than optimal business decisions, regulatory fines and reputational damage.
To ensure that their models are meeting their business needs, organizations should develop a model monitoring standards document that includes:
Enterprise level model inventory. Take stock of the models in use and establish clear ownership of the maintenance and usage of the models.
Robust data monitoring processes. Consider and create rules for each raw data field to help ensure the quality and consistency of the data.
Governance structure. Make sure the model monitoring team works independently from the model development team. There also needs to be effective model audit processes and procedures in place, and senior management must be engaged and involved.
Ongoing monitoring of models is essential to evaluate whether changes in products, exposures, activities, customers or market conditions call for adjustment, redevelopment or replacement of the model, or to verify that any extension of the model beyond its original scope is valid. Any limitations or assumptions identified when a model is developed should be assessed as part of ongoing monitoring.
When it comes time to gauge the performance of credit risk models, organizations often use a few simple metrics for regular monitoring purposes, such as the Kolmogorov-Smirnov statistic and the Gini Coefficient, leaving more comprehensive checks to be conducted infrequently. A better approach is to evaluate credit risk model performance on a regular basis using the following comprehensive indicators:
Model discrimination. The ability of the model to differentiate between events and non-events based on its input values, such as defaults and non-defaults.
System or population stability. How different is the current data being scored by the model, compared to the data from the model development; is the model stable over time?
Characteristic stability. How different is the distribution of the current population in each explanatory variable in the model compared to the population used for model development?
Actual versus expected, or calibration. Does the model deliver accurate point predictions?
Score distribution analysis. Does the model generate large concentrations at particular deciles or credit grades?
Override analysis. To what extent and why are there judgmental overrides over and above the raw model output?
Numerous metrics can be used to set up a comprehensive model monitoring system, but these should be incorporated into an effective management information system (MIS) program. Industry best practices suggest that the model risk monitoring MIS be thorough and timely, offer coverage, provide data/formula integrity and validation, use thresholds, provide an overall assessment, and include clear and relevant insights and commentary.
A strong credit risk model monitoring process is not only required by regulations, but has proven to be a potent competitive advantage for organizations that have taken the steps to:
Track model performance diligently.
Escalate and resolve model issues.
Involve senior management in decision-making.
Fine-tune models on a timely basis.
Maintain well-documented logs and rationales of changes.
Effective monitoring allows institutions to closely control and better empower the strategic risk management tools for day to day operational management as well as for purposes of calculating capital requirements.
Accenture has extensive experience in all aspects of model development, validation and monitoring. Our team has delivered more than 500 risk analytics solutions, and has a deep understanding of the regulatory implications of Basel II and III, Dodd-Frank, Federal Reserve and UK Prudential Regulation Authority (PRA) requirements. We have also developed proprietary assets that can help accelerate and simplify the implementation of a robust model monitoring process.