Quality Assurance of Analytical Modelling: The importance of QA in Model Risk Management

Since the global financial crisis there has been greater recognition that models serve many important strategic purposes for financial services firms, but more importantly they pose many risks that must be recognised and addressed. The importance of quality assurance of analytical modelling in finance and in the public sector is well established. Yet outside these areas, analytical model QA is often not given the attention it demands.

Model Risk Management in the Finance Sector is now embedded in the regulations, guidance and expectations of supervisory bodies, which emphasise the need for audit and assurance of both individual models and the processes around their development and governance

It was addressed as a major concern by leading global banks in recent surveys and round tables conducted in Europe and the United States by McKinsey and Risk Dynamics. The overall number of models varied widely, ranging from 100 to 3,000 per bank; the number of full-time equivalent staff (FTEs) dedicated to model risk management and validation was also highly variable, with European banks dedicating an average of 8 FTEs per €100 billion of assets, while for US banks this average was 19.

Model risk management is NOT specific to finance

The importance of analytical model risk management is not limited to the finance sector.  During the Intercity West Coast franchise competition in 2012 (where a winner was announced, the competition subsequently cancelled, and bidders’ costs refunded before any contracts were signed) the UK Government identified errors in a model used to evaluate bids as being a major contributor to the failure of the process.  The total cost to the UK taxpayer of those spreadsheet model errors is estimated to be in excess of £50million. 

Quality Assurance of analytical modelling elevated by the Macpherson review

Following their experiences with that competition, the UK Government instigated the Macpherson review into quality assurance (QA) of analytical models across government, which reported in March 2013, and recommended that departments put in place the right processes and culture to support model development and quality assurance of those models.  Inherent in those recommendations was the recognition that analytical model governance and QA processes should recognise both the level of business risk associated with each model’s use, and the complexity of the model itself.  Complex models affecting major business decisions justify resource intensive QA.

Following the Macpherson review, a cross-government departmental working group on analytical modelling quality assurance was established and this developed “The Aqua Book: guidance on producing quality analysis for government” (published my HM Treasury).  Hartley McMaster consultants (including myself) reviewed and commented on the content of the Aqua book when it was in draft form.  Its subsequent publication led to many of the departments implementing their own analytical assurance frameworks in line with the Aqua guidance.  It was nice to see model QA finally being taken seriously!

Analytical Modelling Quality Assurance  brings new organisational needs

The increased emphasis on testing of analytical models has provided many opportunities for consultants to provide Analytical Modelling QA support to government departments and to a number of private sector companies. 

We’ve been lucky enough to provide Analytical Model Testing and Quality Assurance services ranging from reviewing entire QA processes to auditing individual models.  They have not been limited to spreadsheet models but have covered models developed in VBA, Visual Basic, C#, SQL, R, Python, SAS, SPSS and other proprietary modelling languages such as those available in Simul8 (simulation) and FICO XPRESS (optimisation). 

Yet quality assurance of high-risk analytical models is being ignored

In all industries, as critical business decisions are increasingly driven by analytical models, it is imperative for organisations to identify and understand their risks.

To be blunt about it: complex models influencing major decisions demand robust quality assurance as part of their risk management strategies.

As you can probably tell, I’m a bit of an evangelist when it comes to model QA.  I’ve consulted with many organisations, many of which have not even considered the need for model QA, despite having models that are high risk or business critical.  

Developing a quality assurance process for your analytical models? Here’s a start…

I believe that analytical model development requires the discipline exhibited by good software engineers.

While talking to one of my colleagues recently, she challenged me to put down in a mind map all the things I thought organisations needed to consider when performing model QA. 

I accepted the challenge, and my resulting multi-level mind map was detailed to say the least, covering many aspects of analytical model development, governance and use.

Here’s the simplified version showing the ten top-level stages:

These ten stages encompass the most important components of analytical model development and Quality Assurance and they act as a starting point for organisations considering how they can put a rigorous analytical modelling QA process in place.

In my next blog post I’ll take a brief look at each stage.

We’d be interested in what elements of model development and QA are important to you.  Feel free to get in touch!