Why the new US regulatory standards for accuracy, integrity, and reliability in credit scoring models matter — a lot
For several years now, groups of stakeholders large and small and points in between have been working on ethical AI, rules for AI, privacy and explainability in AI, and more. WPF actively participated as a delegate in one of the largest international efforts, that of the OECD to write the OECD Principles on Artificial Intelligence, which have since been published and ratified by OECD member countries as soft law, including the US. The OECD Guidelines on AI are broad principles, which need to be made practical in any number of contexts.
One of those contexts is in credit score modeling, specifically, third party credit score models used in the financial sector for credit and other financial eligibility decisions, like financing a home mortgage. In August, a new regulation went into force in the US regarding credit score model development. The impetus for the new regulation was to standardize how credit score models are to be validated and approved for use by the Federal National Mortgage Association (Fannie Mae) and the Federal Home Loan Mortgage Corporation (Freddie Mac). Section 310 of the Economic Growth, Regulatory Relief, and Consumer Protection Act of 2018 amended an older law (the Safety and Soundness Act, 1992) and set for new requirements for the validation and approval of third -party credit score models.
While the FICO credit score is perhaps the best-known credit score at the moment, there are competitors who also create credit scores. The specifics of how to determine which among the predictive credit score models are acceptable for use, and how to ensure a competitive marketplace is being allowed in credit score model development is what this new regulation addresses with specificity.
The new regulations set a detailed process by which AI models in the credit score area will be scrutinized, analyzed, and benchmarked. To accomplish this, the new rule requires significant technical assessments and also economic assessments regarding the impact of the score model on competition. Of particular note, the new regulation creates three important new standards in the US for credit score modeling in the areas of accuracy, integrity, and reliability. The regulation also sets forth specific assessment standards for analyzing competitiveness / anti-competitivenss in the broader ecosystem. To accomplish these goals, the regulation sets forth a four-part process for model validation.
First, model creators need to complete two rigorous assessments:
- A Credit Score Assessment, which is an evaluation of the credit score model’s accuracy, reliability, and integrity. For example, regarding integrity, the credit score modelers will be required to certify that no characteristic based directly on or directly correlated with a classification prohibited under ECOA (Equal Credit Opportunity Act) FHA (Fair Housing Act) and the SSA (Safety and Soundness Act) was used in the development of the model, or was used as a factor.
- A Comprehensive Enterprise Business Assessment that evaluates all of the potential impacts using each credit score model could have on an the enterprise and the mortgage finance industry. Several factors will have to be considered, including an analysis of automated underwriting systems impacts overall, and on specified risk management requirements.
Second, credit score model developers must meet certain requirements to submit the applications — such as market and competition impacts “related to the ownership structure of the credit score model developer and its relationship to other market participants.”
Third, the final rule states that there is not a requirement in section 310 of the Economic Growth, Regulatory Relief, and Consumer Protection Act of 2018 for an enterprise to use a third party credit score model for any part of its business operations or purchase conditions. However if a business conditions its purchase of mortgages on the provision of a credit score, section 310 “requires that the score must be derived from a model that has been validated and approved in accordance” with the new rule. If adopted by a business, the approved credit score must be used in all of the business purchase related systems and procedures that use a credit score. The regulation has a process for the replacement of one validated credit score model with another validated and approved model.
Fourth, Fannie Mae and Freddie Mac currently require lenders to use the “Classic FICO” credit score model for loans. The new regulations establish specific criteria for an initial credit score assessment that will permit an enterprise to evaluate the Classic FICO on an expedited basis. (The Classic FICO also has to undergo evaluation under the new rules). The rule did not make a predetermination about what credit score model would eventually be used by any enterprise, but did say the following:
“However, FHFA acknowledges that approving a credit score model in use for the past decade would not satisfy the intent of section 310 that the Enterprises consider credit score models developed after Classic FICO. FHFA expects that the Enterprises will also evaluate applications received in response to the initial Credit Score Solicitation and that the Enterprises may submit to FHFA a proposed determination to approve one or more of those credit score models for use, either to replace Classic FICO or in addition to Classic FICO.”
This appears to create conditions for a more competitive marketplace in the credit score model sphere.
More about the Accuracy Standard
The final rule requires an Enterprise to establish a credit score accuracy cutoff as a benchmark for the initial Credit Score Assessment. Applicants’ credit scores must be at least as accurate as the benchmark in order to pass the required Credit Score Assessment. There is an expected transition process, and there is an expectation that benchmarking for the initial Credit Score Assessment will be informed by the accuracy of the credit score in primary use today, which is the Classic FICO. However, there is no longer any guarantee that the Classic FICO will be the only score in the marketplace. This leaves room for competition, and ideally, competition around accuracy levels.
The final rule establishes that future Credit Score Assessments must use the validated and approved credit score models in use “at the time the testing is conducted” as the accuracy standard. Basing the benchmark on the most accurate validated and approved score in use at that time is equivalent to what the rule described as “the champion-challenger approach” where the applicant’s credit score model (the ‘‘challenger’’) must be more accurate than the existing credit score model in use (the ‘’champion’’). Over time, the financial sector should have at its disposal many more accuracy benchmarks to utilize for comparison.
More about the Reliability Standard
The final rule establishes a reliability standard that must be met as part of the Credit Score Assessment process. Under the reliability standard, a credit score model is deemed “reliable” if it produces “credit scores that maintain accuracy through the economic cycle.” The credit score models will need to perform accurately despite economic fluctuations.
More about the Integrity Standard
The new regulation sets an integrity standard, under which a credit score model has integrity if, “when producing a credit score, it uses relevant data that reasonably encompasses the borrower’s credit history and financial performance.” The goal of the standard is to ensure that credit score model developers have utilized data elements that are legally permissible, and relevant. The integrity standard is designed to permit credit score model developers to innovate with data sets, but with guardrails.
To meet the new standard and become a validated model, a credit score model applicant would be required to prove that the model has integrity, based on “appropriate evaluations or requirements identified by the Enterprise (which may address, for example, the level of aggregation of data or observable data that may not be omitted or discounted when constructing a credit score).”
The integrity standard is particularly key to ensuring that prohibited factors under ECOA do not drift in to credit scoring models exploring new data types. This has been and will continue to be an area of concern, but it is a good step forward that the new regulation requires certification that no characteristic based directly on or directly correlated with a classification prohibited under ECOA was used in the development of final iterations of a model.
With little fanfare, the US now has new and practical guidance around the development of credit score models. The standards that have been set for accuracy, integrity, and reliability are important, as is the overall process that sets parameters for ensuring non-discrimination in the use of factors or correlates.
Related Documents
Validation and Approval of Credit Score Models (Federal Housing Finance Agency)
Economic Growth, Regulatory Relief, and Consumer Protection Act of 2018
The Scoring of America, see the Credit Score section, and the History of the Credit Score.