Recent advancements in AI automation are changing lenders’ expectations of vendor performance. Traditional enterprise software deployments require service level agreements (SLAs) that focus on system uptime and response times. These static metrics are insufficient when deploying new, enterprise AI systems, which must continually adapt to changing real-world data. To maintain sustainable, high quality performance, lenders should consider new categories of metrics – automation and accuracy SLAs.
What are automation and accuracy SLAs? In a nutshell, these are metrics that assure the lender that a vendor’s promised automation of services and accuracy of results stay above a certain minimum threshold. These metrics help assure that an AI system’s initial performance, timeliness, and cost-effectiveness is durable.
If system accuracy drifts over time or the automation breaks down, the benefits of these AI systems are greatly compromised. In fact, lenders can face steep financial and opportunity costs trying to implement short-term patches and long-term fixes to an AI system that doesn’t perform as expected.
At the core of this situation is the need to create more accuracy and automation of SLAs for AI lending applications.
Understanding today’s risks for lenders
Lenders across industries are working with AI and data automation providers to implement automated workflows. In auto lending, for example, AI document intelligence providers help lenders to automatically classify documents within an auto loan deal jacket, extracting relevant information, and running complex business rules. These workflows can be compromised if live application data is different than the information the model was trained on, which is always a possibility in any of the following:
New income documents: Bank statements and pay stubs come from thousands of sources. No matter the size of the lender, new documents enter their document flow;
New government documents: State and Federal governments regularly update new documents such as drivers’ licenses or title documents;
New product offerings: There are tens of thousands of ancillary product offerings and more come on-line all the time;
New legal requirements: State and Federal laws evolve, imposing new fee limits, disclosures, and compliance requirements;
New document production technologies: Scanners and other document capture technology continually evolve.
Losses in accuracy and automation are susceptible and can lead to financial losses, reputational damage, and potentially severe regulatory and legal consequences, especially in the highly scrutinized consumer lending sector.
Challenges with performance and accuracy
Performance declines are particularly damaging in the absence of appropriate controls. Document intelligence solutions that are not purpose-built for specific lending applications fail to report on actual performance.
Most solutions rely on the lenders to perform their own analysis to track performance. Lenders run the risk of not catching these declines until they encounter materially adverse results or a borrower files a complaint. Compounding the problem, automation often comes with a reduction in staff with the technical expertise to fix the issue. Recovery is slow, resource-intensive, and painful.
Accuracy metrics can also impact compliance. They provide a continuous, data-rich record to support model governance and internal audits. Without them lenders would have to secure additional budget to build out, monitor, and maintain an audit trail.
However, lenders that believe they can solve these challenges by simply employing off-the-shelf AI and generative AI tools, such as ChatGPT, run the risk of significant and damaging consequences with their SLAs.
Accuracy-based SLAs must be industry-specific and are incredibly challenging to implement. They demand significant investment in a vertical sector and need a statistically significant, labeled dataset, which requires continuous updating.
SLAs redefined through vertically focused AI platforms
Leading lenders overcome this challenge by working with AI and automation-focused data workflow partners who rely on vertically built platforms specific for auto lenders. To implement this next generation of Accuracy SLAs, these partners offer lenders a governance and control system of document intelligence designed to keep AI explainable and responsible.
These platforms are noticeably different from standard AI tools because they combine all the necessary components of an accuracy SLA system by supporting the ongoing monitoring of core accuracy metrics: recall, precision, and concept drift; continuous expert labeling of real-world, representative data; audit-ready insights for regulatory and compliance functions; transparent AI with PII-redacted document viewing, and an easy-to-navigate user interface.
With this level of accuracy, auto lenders are assured they will maintain proper accuracy, automation, and compliance within their SLAs and document workflows in today’s increasingly digital era.
