MRM
Model Risk
Risk infrastructure is absolutely saturated with complexity:
- Models themselves are relatively complex mechanisms
- They are constructed using complex processes
- They are fed by complex, multi-stage networks of ETL pipelines and intermediate databases
- They consume complex data, that has inevitably undergone several complex, parametrised transformations
- They are validated using complex measures
- And their very utilisation can involve great complexity
It doesn’t help that much of this complexity is actually hidden, at different levels – sometimes quite deep – buried in a flow of symbols, esoteric communication and the tacit knowledge of human modellers.
As a result of all this complexity, the potential for failure is always present. Not only that but a single failure can trigger a cascade, with risk and effects spreading through an organisation like a wave. Worse still, ‘failure’ is not always an obvious state – i.e. failures can happen silently, avoiding detection for years while the impact of errors accumulate. Some aspects of failure exist on a continuum which places pressure on the setting of thresholds and the interpretation of validation results.
Errors can enter the process at any point, or many points, in the model development process. Problems can form at the very earliest point of framing the requirement for a model, or in the very last stages of implementation. To compound matters, errors in development actions (and tooling) can interact, adding an extra layer of model risk complexity.
And, as the regulations themselves point out (see references below), even the utilization of a model is a source of model risk.
Clearly, this large, multi-faceted ‘surface’ of model-related vulnerabilities cannot be treated with a single resource or a single technique; an integrated system of policies and resources is required. Arguably, a certain culture is required
The term ‘model risk’ has emerged in recent years to capture the issue of failure arising from these intricate mechanisms, their development processes, and their utilisation. Comprehensive definitions can be found in the publications of several regulators, most notably: SR 11-7 link.
In response to the rising awareness of model-related risk, the discipline ‘Model Risk Management’ (MRM) has emerged to address these concerns in a structured, systematic way.
Calibration may appear on the surface to be a relatively uncomplicated process but it too is a source of model risk. In fact, we would argue that poorly-calibrated models are a material form of model failure. Logically, every aspect of calibration falls within the general scope of MRM.
At Radius we have a particular perspective on model risk that is grounded in systems theory. This perspective has significantly influenced the design of our modelling platform, more of which we will be announcing a little later in the year.
The term ‘model risk’ has emerged in recent years to capture the issue of failure arising from these intricate mechanisms, their development processes, and their utilisation. Comprehensive definitions can be found in the publications of several regulators, most notably: SR 11-7 link.
In response to the rising awareness of model-related risk, the discipline ‘Model Risk Management’ (MRM) has emerged to address these concerns in a structured, systematic way.
Calibration may appear on the surface to be a relatively uncomplicated process but it too is a source of model risk. In fact, we would argue that poorly-calibrated models are a material form of model failure. Logically, every aspect of calibration falls within the general scope of MRM.
At Radius we have a particular perspective on model risk that is grounded in systems theory. This perspective has significantly influenced the design of our modelling platform, more of which we will be announcing a little later in the year.
Trends and Challenges
Without a doubt, the most pronounced change in the overall landscape is the sheer volume of models. Banks’ model inventories have increased significantly over the past few years, driven partly by a broadening of scope but also just the awareness of model-based potential to manage a business.
The overall complexity of models is also increasing, and in some cases, increasing dramatically – advances in machine learning architectures and the increased availability of libraries has encouraged a proliferation of models with more structural depth (such as neural networks), and compositions of models. To a lesser extent, there has also been an increase in less ‘conventional’ techniques, including hierarchical Bayesian models; models with non-parametric structures; and graph-theoretic models.
Use of models in banks has expanded in line with other developments in the business environment. Beyond the more regulated use cases for models, such as for capital-setting and provisioning, banks’ use of predictive models for things like Customer Relationship Management, and KYC, has exploded.
A more recent innovation bears mentioning. The extraordinary capabilities of generative AI, such as that exhibited by LLMs (large language models such as the GPT series), will drive their adoption in banks. This development alone presents enormous problems for MRM; the potential for well-intentioned but low-quality modelling code to find its way in to the stack … is virtually unlimited. Despite their allure, and the promise of radically reduced development times, LLMs are something to be treated with caution until issues around alignment can be resolved.
Use of models in banks has expanded in line with other developments in the business environment. Beyond the more regulated use cases for models, such as pricing and capital-setting, banks’ use of predictive models for things like Customer Relationship Management, and KYC, has exploded.
A more recent innovation bears mentioning. The extraordinary capabilities of generative AI, such as that exhibited by LLMs (large language models such as the GPT series), will drive their adoption in banks. This development alone presents enormous problems for MRM; the potential for well-intentioned but low quality modelling code to find its way in to the stack … is virtually unlimited. Despite their allure, and the promise of radically reduced development times, LLMs are something to be treated with caution
Model development processes have also changed dramatically in recent years, and this has been paired with advances in model architecture. Exotic loss functions; regression on quantiles; advanced hyper-parameter optimisation; regularisation structures; meta-heuristic parameter estimation; complex sampling schemes; formal guarantees for prediction sets; and symbolic methods have combined to create a much richer landscape of techniques for utilisation by modellers. While this definitely widens the capability of a modelling exercise, it also creates obvious challenges for the validation and policy elements of MRM.
Another challenge to MRM is the use of open source languages and libraries. While the explosive growth in FOSS has been net positive, there are some serious issues associated with the security, stability and correctness of open source code. Further, the easy availability of pre-written code has established something of a trend for blind utilisation; today’s modellers are able to cut-and-paste from a vast universe of code repositories, and run this code without testing.
The above-mentioned changes to the modelling environment has created another issue: modeller skill. The depth of knowledge, experience and of technical skill required to build a model has actually fallen despite the increase in model complexity.
Other developments in the business environment have radically increased the breadth and heterogeneity of possible data sources for model-building. Data generated by social media platforms has rich predictive structure, but resists easy assimilation