I had successfully avoided jury duty my entire adult life, until April of last year. The courtroom was cramped and a bit run down. Fifty or sixty of us were jammed into rows of old wooden pews that looked like something out of a 19th century country church. I’m sure if there were only a few of us in the room, the air-conditioning system would have been adequate for late April in Texas, but with so many people jammed into such a tight space it was surely not. A couple people seemed very excited to do their civic duty, but most of us were looking for any reason to be disqualified so we could leave – I was in the latter group.
Before the jury selection process began, the judge explained the nature of the case, which involved someone who had injured a number of children. Both the prosecutor and the defense attorney were allowed to make introductions of their own, before proceeding to the jury selection and elimination process. The prosecuting attorney began by asking if any of us enjoyed watching television shows such as CSI, NCIS or Criminal Minds. Many in the room enthusiastically raised their hands as the prosecutor smiled and nodded. Then he began to explain that while those shows are amazing entertainment, much of the forensic technology presented does not actually exist.
The jurors who would ultimately be chosen were going to make important decisions concerning evidence, and the prosecutor wanted to make certain that everyone had the correct understanding of what should be expected from the technology of the day. Fantastic writing and Hollywood special effects have left the average person with a completely distorted view of what the technology is capable of. Ironically, this is true in the world of auto finance as well.
Analytic vendors everywhere are claiming amazing advances in predictive power using artificial intelligence (AI) and machine learning. Some lenders have even issued press releases proclaiming how they have integrated AI into their credit process. Terms like AI, machine learning and neural networks invoke images of the Skynet computer from The Terminator movies – thinking machines that learn new things on their own, accelerating far beyond what a human mind could conceive. Such machines do not yet exist; instead, what we have today is largely scripted intelligence (by a human) and automated data-mining.
Unfortunately, there is no prosecuting attorney to correct the public’s idea of what is realistic in data analytics. To the contrary, vendors have no interest in grounding your understanding of what is real and what is fiction. The message instead is “buy my solution because we really are that advanced!” Similar to the jurors in the courtroom, companies are making critical decisions on data. Given that so much is riding on the answer, lenders must gain an understanding of what is practical and what is merely marketing.
The right tool for the right job
Advancements in driverless vehicles, and the related potential for significant market disruption, have created a tremendous focus on artificial intelligence. Suddenly, lenders and vendors alike are claiming a secret sauce focused around this class of technology in much the same way the blockchain crowd has; but, the question remains – what is real and what is hype?
The answer lies in the problems these tools are tasked with solving. Most of us, at one time or another, have driven in a nail with a wrench. We also realize that there are better tools for that particular task. The same is true in predictive modeling. Some modeling techniques, such as linear regression, are ideal for problems where the output is continuous, such as age, income or temperature. Others, like logistic regression, are more appropriate for binary outcomes (i.e., pay-off vs. charge-off).
Artificial Intelligence was a term coined back in 1956 in the field of computer science. Research in that area has made incredible advancements in the last two decades due to the increase in processing speed and data storage capacity, in addition to the proliferation of big data – where billions of transactional records are produced on a daily basis. Machine learning, deep learning and neural networks are tools within the broader category of AI. Applications of this technology are widespread today in areas such as natural language processing, 3D image recognition and speech recognition.
The methods that fall under the umbrella of AI are ideally suited to problems where explicitly describing all of the rules needed to categorize something would be infeasible. For example, consider all of the rules one would have to write down in order for a computer to visually recognize a dog in a photograph. Given all the breeds, colors, shapes and sizes it would be nearly impossible for a human to come up with an exhaustive list that would accurately recognize a dog and distinguish it from another similarly shaped animal.
However, using complex algorithms and millions of photos labeled as simply “Dog” or “No Dog”, AI can rapidly sort through and find complex commonalities in the pixel patterns of images, eventually iterating (learning or training) to produce a reliable recognition model. From there, a human would create script on what the program should do with that recognition. The latter is an important aspect to take note of in understanding how AI operates today.
For example, visual AI technology can be trained to recognize a pedestrian, a stop sign or particular weather conditions in real time. The moral judgment of choosing to crash into a bicyclist instead of a school bus (if those were the only choices) is scripting imposed on the program by a human once the objects are recognized. The AI is not actually independently reasoning to arrive at that decision, as many assume.
Determining whether AI techniques are appropriate for a particular problem requires several things to be true, such as:
• Black-box sufficiency – A black-box refers to models or processes that are either not transparent or cannot be explained due to their complexity. When it comes to approval or pricing decisions on U.S. consumer loans, no lender should think of this unless they wish to make things very easy for the Consumer Financial Protection Bureau. Lenders must be able to not only explain how the model works but be able to demonstrate that the model does not have a disparate impact on protected classes. For other problems, such as loan structuring or repossession timing, black-box solutions may be sufficient. Companies must understand that when people use terms like machine “learning” or artificial “intelligence”, it is the black-box that has the learnings or intelligence, not the user. There are a limited number of applications in lending where it doesn’t matter if the company understands the thing they are attempting to manage.
• High signal-to-noise ratio – A substantial amount of time and money has been spent by lenders and industry groups studying the reasons why consumers default on their auto loans. Among the top reasons are financial difficulties that arise out of a job loss, divorce or catastrophic medical issue. Unfortunately, there is nothing on the credit report that will tell you a customer is going to lose their job or get divorced over the life of the loan.
Furthermore, many who run into these financial difficulties successfully pay off their car loan. Credit data has a lot of noise in it, so lenders must rely on factors that, at best, help stack the odds in their favor. In other words, the more paid credit one has, the less likely one is to default. The successful applications of AI have occurred in areas where there is much less volatility in the relationship of the predictive factors and the target event. This is referred to as a high signal-to-noise ratio. Examples of this include speech recognition and load-balancing system resources between software applications on a network.
• Nonlinear data relationships – When modelers correlate one factor to another, such as the number of delinquent accounts to default rate, they are exploring a linear relationship. Non-linear relationships describe the case where multiple factors are combined to form a single node, but the importance (weightings) of those factors change depending on what they are combined with. Neural networks and decision tree models are examples of non-linear models. Using AI techniques on non-linear data with a low signal-to-noise ratio will create a highly volatile black box.
• Massive amounts of data – Conventional wisdom in credit scoring suggests that a lender needs at least 1,500 examples of a default within a narrow time period in order to have enough data to build a single credit scoring model. It would take a lender funding 500 loans per month approximately two years to produce 1,500 defaults, assuming a 25 percent lifetime default rate with half of the defaults occurring in the first two years. Such a lender would likely have a $250 million portfolio balance. Imagine if the model demands (complexity) were multiplied by 1,000. That would require a portfolio 300 times the size of the entire auto finance market to have enough degrees of freedom to produce a robust model. Given that most non-prime lenders close only four to five percent of their applications, no one would have the volume to satisfy the requirements of such a model; however, on the platforms managed by Google, Amazon, Facebook and other big tech companies, such transactional volume exists in abundance.
• Robust state-space – The primary criticism of AI techniques is that they overfit the data. In statistical terms, this means they read too much into the relationships within the sample of data that the models were trained on. In credit data, the relationship of prior delinquency to future auto defaults changes based on where we are in the credit cycle–because credit is, in fact, a moving target. When it comes to other big data problems, such as load balancing network resources, the relationships between present and future states are more reliable. Using a modeling technique that requires stability on unstable data will not produce a good answer, regardless of how sophisticated the technique is.
There are many areas within auto finance where AI techniques have the potential to bring tremendous advantage. These include automated loan structuring, collections optimization and maximizing closure rate. For lenders to gain from advancements in AI, they must be able to distinguish between applications where the techniques can add value, and where they are merely marketing hype, that if taken seriously, would produce a worse answer.
Making AI a practical reality
In late 2017, the Long Island Iced Tea Corp. changed its name to Long Blockchain Corp. The company stated that they were going to focus investment and partnerships on other companies with distributed ledgers. In the first 9 months of 2017, Long Island Iced Tea Corp. lost nearly $12 million on $4 million in sales. After the announcement, the company’s shares rose 289 percent. To date, they have no actual blockchain relationships, and are not certain if they will in the near future.
The rise in the valuation of Bitcoin over the last year has everyone focused on how they can ride the wave of interest in the underlying blockchain technology. This is very similar to what occurred in the late 1990s, when companies that had nothing to do with the internet rebranded themselves as internet companies. Hype sells for a time, but eventually the market figures out which offerings have substance, and which must be shaken out.
There is an obvious corollary as it relates to AI. Two logical fallacies are routinely employed by those selling these solutions. The first is the “Appeal to Novelty”, which says that the solution is better by virtue of the fact that it is new. The second is the “Appeal to Authority”, which says “Look how sophisticated my techniques are. I am smart; therefore, you should just trust me.” Neither of these fallacies translates to a positive bottom line result.
For AI to become a true, practical reality requires that lenders understand when and where these techniques can bring the most value. These are challenges where the problems are complex, data is abundant, and there is no requirement to explain the answer. TruDecision, and other analytics firms, are actively developing practical applications of AI to the business of auto finance. If these solutions are truly effective, they will stand up to the educated lender’s scrutiny. While hype can sell for a time, the market ultimately demands substance. Substantive AI will come from lenders who ask “Will the Real AI Please Stand Up?”