On February 14, 2020, California State Assembly Member Ed Chau introduced the Automated Decision Systems Accountability Act of 2020, which would require any business in California that provides a person with a program or device that uses an “automated decision system” (“ADS”) to establish processes to “continually test for biases during the development and usage of the ADS” and to conduct an impact assessment on that program or device.

ADS is defined broadly as “a computational process, including one derived from machine learning, statistics, or other data processing or artificial intelligence techniques, that makes a decision or facilitates human decision making, that impacts persons.”  The required ADS impact assessments would study the various aspects of the ADS and its development process, “including, but not limited to, the design and training data of the ADS, for impacts on accuracy, fairness, bias, discrimination, privacy, and security.”  At minimum, the assessments must include “[a] detailed description of the ADS, its design, training provided on its use, its data, and its purpose” and “[a]n assessment of the relative benefits and costs of the ADS in light of its purpose,” with certain factors such as data minimization and risk mitigation required in the cost-benefit analysis.

The provider of the ADS also must determine whether the ADS system “has a disproportionate adverse impact on a protected class,” examine whether it serves “reasonable objectives and furthers a legitimate interest,” and consider alternatives or reasonable modifications that could be incorporated “to limit adverse consequences on protected classes.”

By March 1, 2022, businesses will be required to annually submit to the Department of Business Oversight a report that summarizes the results of their ADS impact assessments.  If a change is made during the year, then the results of a new ADS impact assessment must be submitted within 60 days.  To review these reports, the Department will establish an ADS Advisory Task Force consisting of two representatives from each of the following: advocacy organizations representing consumers or protected classes of communities; state or local government agencies; digital or software companies who use or create ADS technology; and universities or research institutions with expertise in ADS technology.

Notably, the bill comes at the heels of a proposal by Senator Kirsten Gillibrand (D-NY) in the U.S. Senate that would create a new federal agency tasked with, among other things, requiring impact assessments of “high-risk data practices,” including “systematic or extensive evaluation[s] of personal data that [are] based on automated processing . . . on which decisions are based that produce legal effects concerning [an] individual.”  The proposed new agency also would regulate “consumer scoring” and other practices that determine consumer eligibility for rights, benefits or privileges in certain contexts (e.g., employment, credit).