Every year, there are several executive surveys conducted by organizations that produce the primary auto finance conferences. As we emerge from the current crisis, I am reminded of the conferences of 2012 when we were recovering from the Great Recession. A well-known industry consultant was delivering the results of the executive survey, when he noted that leaders felt we were building models off of the past and not the future. I almost broke out laughing as I thought, “Brilliant! Please get me some data from the future so I can build better models.” While I’m not entirely sure the presenter got it, I do understand what the survey respondents meant; that is, if the data you are building models off of is not representative of the future state of things, the results can be horribly myopic.
In the coming months, many lenders will be updating their credit scoring, collections and loss forecasting models. They should do so with great caution, as we live in an age where it is too easy to produce models using software tools apart from knowledge of the limitations of the methods employed. Merely throwing historical data into new models without careful scrubbing could potentially produce results worse than no update at all.
At the outset of the COVID-19 crisis, many industry analysts developed projections based off of the Great Recession. Such forecasting was poorly conceived for a variety of reasons. First, recessions are the product of a fundamental weakness in the economy, which was not the case prior to 2020. Second, the Great Recession is an extreme case among recessions. Finally, decline and recovery during recessionary periods typically takes place over three to four years. In the last downturn, it took two years for unemployment to jump from six to nine percent, and another five years for it to come back down.
This pandemic is much more like a sudden market shock, the kind of which is seen during weather events such as major hurricanes. During Hurricane Katrina in 2005, devastation was massive, but the bounce-back was fairly quick in terms of its impact on auto finance companies. Lenders were given carte-blanche to offer forbearance by banks, rating agencies and investors. In addition, the court of public opinion demanded a compassionate response. The COVID-19 crisis is a very similar event (shock), except it occurred on a global scale. For lenders, this meant handing out deferments (payment extensions) at a much higher rate and suspending repossession activity.
Competent modelers must neutralize the effects of exaggerated forbearance from their data, or they risk producing models that are wildly optimistic under normal conditions. For most lenders, they cannot afford to throw out nine months of performance data, nor would I recommend they do so. There are some practical ways to account for these effects while still retaining most of the data.
Oftentimes, doing so does not come easy to many who are involved in analytics. There is a natural resistance to say anything that is not explicitly supported by the record. I have trained many data scientists over the last 20 years, and my number one pet peeve is when I ask them about a trend and they respond, “Well, that’s what the data said.” The data doesn’t say anything. It is there to be interpreted. Machine learning cannot do that, neither can software nor any other automated tool. There is no substitute for a person who has a contextual understanding of the problem they are trying to explain. As such, this article will discuss three primary pandemic-related pitfalls that are likely present in the data of most auto lenders, and what may be done to make this data useful in forecasting future events.
Pitfall One: Loss Levels
The typical non-prime lender defers two to three percent of their active balances on a monthly basis. Securitization data from 2020 suggests that this level increased to as much as five times due to the pandemic. While levels have come down significantly since last year, they are still elevated.
Many consumers who might have otherwise defaulted were given enough time to get back on track with regard to making regular payments. In my prior roles as a risk manager, we regularly tracked our post-deferment charge-off rates. We consistently observed default rates between 20 and 25 percent on that population. Think about that figure for a moment. With a well-executed deferment strategy (not just avoiding the inevitable) lenders can help a significant number of accounts avoid the default altogether. There is no doubt that many consumers who might have defaulted during the pandemic were given enough latitude to rehabilitate due to deferments.
The point of any predictive modeling exercise is to, as clearly as possible, distinguish between various outcomes (i.e., good and bad) using data attributes from an earlier period. When a servicing strategy is materially changed, the modeler must consider what that could mean in terms of which accounts will default in the new environment. As it relates to deferments, I recommend analysts do two things. First, remove accounts from the development sample that traditionally would not have needed a deferment. This may be done by using multivariate methods (cluster analysis or decision trees) to identify the historic profile of the deferment recipient so that the analyst may assess the true incremental need-based deferments that were awarded. Second, the analyst should over-sample post-deferment need-based charge-offs from the affected period in order to adjust the sample to what is likely to be observed in the future.
Pitfall Two: Loss Timing
Changes in servicing strategy frequently wreak havoc on risk models. For example, pulling the repossession assignment date forward will cause some accounts to default that might have cured, while pushing it out may save some accounts. The latter tends to also cause increases in delinquency, decreases in recoveries and a temporary flattening of loss curves. Shifting the timing of loss will also convolute roll rates and other loss forecasting tools, not to mention a distortion of seasonality.
A wholesale shift in deferment levels has these effects as well. The industry standard in deferment policies is to move two monthly payments to the end of the loan, with the requirement that the deferment must bring the account into a current status. During the COVID-19 crisis, policies may have been wildly stretched. This may result in more than two deferments within a twelve-month period (which should be a troubled-debt restructuring), as well as deferments covering more than two payments. Such disruption not only impacts timing curves but also credit scoring models. The standard scorecard development methodology calls for lenders to sample accounts that have a performance window (for default) of between twelve and twenty-four months. A large shift in deferment activity could push many defaulting accounts outside the performance window, thus degrading the sample.
Depending on the type of model the analyst is evaluating, the loss timing for defaulting accounts should be over-written with what would have occurred under normal circumstances. This is easier to do than adjusting the level of losses in the prior section as there is less ambiguity involved. In this case, the analyst should merely backdate the default to the first incidence of an out-of-policy deferment. This is not a perfect solution, as time to repossess and move through auction can vary greatly, but it is far better than modeling an anomalous trend where an account languished in non-payment for six or more months before being charged-off.
Pitfall Three: Vehicle Recovery Values
Lenders typically see vehicle recovery rates in the 60 to 65 percent range (as a percentage of gross default) in the first six months on books, as the loan is so new the vehicle hasn’t had much time to depreciate. At the 18-month mark, these same lenders may experience a 40 to 45 percent recovery. At 40 months that figure may drop to the mid-30s. Deferments affect timing, and timing affects recovery rate.
Some lenders have very complex vehicle recovery models, segmented by time on books and vehicle type. Others use a flat assumption based on average time to charge-off. In either case, vehicle recovery rates should be modified when redeveloping models in order to make certain the most reasonable numbers are relied upon. Unfortunately, this is made more difficult by the impact of repossession forbearance.
Many lenders across the country did not repossess any vehicles in the second and third quarters, and some are only slowly returning to the practice today. The lack of repossessions created a supply issue at auctions, driving prices much higher. This is a great example of why lenders should be wary of black box solutions or analysts who cannot venture an opinion that is not purely supported by data. One might look at 2020 performance and conclude that delaying the repossession was correlated with higher recovery values. That would be a correct analysis; however, there is an assumption of causality that is ridiculous. It is like saying wet sidewalks cause rain.
There are a variety of reasonable ways to address this issue. Some may throw out the recovery data from questionable periods, while others may estimate the amount of price inflation and back that out of the results. In addition, analysts with a sufficient amount of data may impute their historical recovery values on like vehicles with similar loss timing. A method I think is very interesting here involves setting inflated recoveries to missing, and imputing values based on the same methods used in accounting for missing values during a scorecard development. This is actually one of the areas where neural network models have proven to be very effective. Whichever method is selected, the results are likely to be more robust than modeling highly inflated recovery values as if they would be that way indefinitely.
Final Thoughts
One of the first steps an analyst should take when beginning a predictive modeling project is to do an exhaustive evaluation of what factors influenced yesterday’s data that are not likely to repeat. This must involve sitting down with both originations and servicing leaders to discuss what changed during the period from which the data was sampled. This is often outside the comfort zone of the analyst, who may have very little operational background; but, it is critically important in making sure the models are not constructed with outcomes that are unlikely to reoccur. The added benefit of this is that operations leaders are much more likely to buy-in to models that they believe considered important changes in the market or the lender’s strategy.
The COVID-19 pandemic led to an enormous shift in how auto loans are serviced. These shifts have produced results that are not likely to repeat in 2021 and beyond, creating pitfalls in the data. While I have identified three things to watch out for, there are likely many others. Anything that could cause a change in the number, timing or dollar loss of defaulting accounts should be considered, regardless of whether there was a macro-level crisis. Paying attention to pitfalls is one of the surest ways to build models today that are robust going forward.