Making Models | By Meredith F. Piotti
In this highly regulated and risk-focused environment, financial institutions are relying more on applying models to important procedures such as identifying risk within the financial institution, and complying with regulatory reporting requirements. However, using models can increase risk since there are a number of factors that could cause them to be inaccurate and lead to financial loss.
Understanding what model risk is, how to build better models and verifying their accuracy will reduce risk and losses at your financial institution while instilling best practices that can be refined and replicated to help the organization become more secure and successful.
Understanding Model Risk
Model risk is quite simply the risk that results from decisions made based on misused or incorrect model output reports. These errors can lead to financial loss, poor business and strategic decision making and long lasting damage to the financial institution’s reputation.
Model risk occurs when there are fundamental errors in the model that result from poor quality of input data, which of course, leads to poor quality output data on which decisions are based. Model risk also occurs when a model is used incorrectly or inappropriately, such as when models are used in a different way from what was originally intended.
How to Prevent Model Risk
There are steps you can take to reduce the chance of model risk. The most important step is to put an “effective challenge” process in place. This is done by ensuring that there is ongoing critical analysis by objective, informed parties who can identify model limitations and assumptions and apply appropriates changes. There are three areas to look at in the people who will be doing the effective challenge.
Incentive – Having a person who did not build the model lead the effective challenge will produce better results as they will have an objective view of the model, they will not hesitate to question of any assumptions in the model, and they will not be tempted to hide any flaws in the model that could be revealed and reflect poorly on them.
Competence – The people overseeing the model and raising the effective challenge must be competent and have knowledge of the model process. This can be accomplished through selecting people with this knowledge as well as by providing training on how the modeling works, what the outcomes should look like and what indicates potential problems.
Influence – You want the people who are overseeing the effective challenge process to have influence in the institution so they can provide leadership on implementing the needed changes and be heard by the board, management and staff.
vs. Outsourced Modeling
There is no question that your institution must create the best models possible to continue operating successfully. Depending on your institution’s size and available resources, there are two ways to get the modeling done: have your internal team build the model or outsource the work to experts. There are some things to consider when deciding to do the modeling in-house or to outsource.
Before undertaking modeling in your institution, you will need to assess if the team that would work on the model has the time and skills to build an accurate model that will produce solid output data. You will also want to consider how much control and customization is required for your modeling needs.
One of the main benefits to in-house modeling is the amount of control you will have because you are the one building the model. However, the drawback to in-house modeling is that you could be limited by time and availability of staff, the complexity of the model that is needed and not having the knowledge among your team to build an accurate model.
When developing your model in-house, there are important things to consider to make sure it's as effective as possible:
You need a clear statement of purpose that lays out what the model should be performing.
Make sure the components in the model work as intended, are appropriate for the business goal and are conceptually sound and mathematically correct.
Have a good assessment of data quality and relevance, and that the appropriate documentation is available.
Undertake periodic testing during development to make sure that the calculations within the model are accurate, that the model is stable and robust, that limitations are assessed and that there is constituent behavior over a range of inputs.
Outsourcing modeling allows your institution to seek out the best experts available to suit your needs and build the model for you without being limited by knowledge or complexity. It also means that the outside experts are working on the model, which frees up precious time for your staff to address other issues in the institution.
An outside expert brings a wealth of knowledge to the modeling process, as it’s their job to make the most accurate models possible. They not only get to know the particulars of your institution, they have also experienced and gathered a range of best practices from the many different financial institutions and can apply them to your needs.
Testing Your Model
Understanding if your model is working effectively is an ongoing process. To get an accurate understanding of the model’s accuracy and ensure that it reflects reality, you should seek out and utilize user feedback and insights and ask senior management to question assumptions and methods.
If the model is not working correctly and you are developing it internally, you may want to consider hiring a professional team to move the process forward. If you are already outsourcing the work, you can work with your current vendor or change vendors.
Validating Your Model
All models in the institution – as well as all parts of each model – should be subject to validation. Validation is a set of processes and activities intended to verify that the models are performing as expected and are in line with their design objectives and business uses. The frequency of the validation depends on the complexity of the model and how often it’s used. The rigor of the validation depends on the potential risk.
The validation should be performed by people who do not have oversight of the model. They should have the skills, experience and authority to accurately validate the model and ensure that changes are made throughout the institution.
The ongoing monitoring of model validation is one of the most robust periods of testing for your models. It should include checking for mathematical accuracy, ensuring that the model is performing as expected and assessing if it’s meeting regulatory requirements and best practices. It should be noted that one regulatory hot button right now is the override of assumptions, which can be used to manipulate outcomes. It is considered a best practice today by regulators to fully document any assumption override decisions.
The final step of the validation process is the outcomes analysis. In this step, you want to compare the model’s outputs to benchmarks and industry norms, but most importantly, to the actual outputs created by the institution. This will allow you to more accurately evaluate the performance of the model. You will also want to look for some early warning metrics that indicate any flaws in your model.
Building a Stronger Future by Building Better Models
Building, testing and validating accurate models is important to your financial institution’s financial well-being. By building a strong team around the modeling process, utilizing best practices and seeking expert consultants when needed, you can create strong models that will serve your financial institution well into the future.■
Meredith F. Piotti, CPA, CIA, is internal audit supervisor for Wolf & Co. She can be reached at 617-933-3353 or via email at firstname.lastname@example.org.