AI Governance Tools

The World Privacy Forum has filed detailed comments to the U.S. Office of Management and Budget (OMB) in response to its Request for Information on Privacy Impact Assessments. Specifically, OMB requested information about how the U.S. Federal government should update or adjust its requirements for Privacy Impact Assessments (PIAs) in regards to changes to data ecosystems brought about by Artificial Intelligence (AI). WPF provided substantive recommendations regarding administrative provisions of the Privacy Act, scalable automated AI governance tools for privacy and trustworthy AI, ensuring nimble processes for privacy and AI assessments, and ensuring balanced, skillful socio-legal-technical decisionmaking.

Today the Biden-Harris Administration published a Memorandum that sets forth how U.S. Federal Agencies and Executive Departments will govern their use of Artificial Intelligence. The OMB memorandum provides an extensive and in some ways surprising articulation of emergent guardrails around modern AI. There are many points of interest to discuss, but the most striking includes the thread of biometrics systems guidance throughout the memorandum and continuing on in the White House Fact Sheet and associated materials. Additionally, the articulation of minimum practices for safety -impacting and rights- impacting AI will likely become important touch points in regulatory discussions in the U.S. and elsewhere. The guidance represents a significant policy shift for the U.S. Federal government, particularly around biometrics.

In December 2023, WPF submitted detailed comments to the U.S. Office of Management and Budget regarding its Request for Comments on Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence Memorandum.  OMB published the request in the Federal Register on November 3, 2023. This particular Memorandum is of historic importance, as it articulates the establishment of new agency requirements in the areas of AI governance, innovation, and risk management, and would direct agencies to adopt specific minimum risk management practices for uses of AI that impact the rights and safety of the public.

We are pleased to announce the publication of a new WPF report, “Risky Analysis: Assessing and Improving AI Governance Tools.” This report sets out a definition of AI governance tools, documents why and how these tools are critically important for trustworthy AI, and where these tools are around the world. The report also documents problems in some AI governance tools themselves, and suggests pathways to improve AI governance tools and create an evaluative environment to measure their effectiveness. AI systems should not be deployed without simultaneously evaluating the potential adverse impacts of such systems and mitigating their risks, and most of the world agrees about the need to take precautions against the threats posed. The specific tools and techniques that exist to evaluate and measure AI systems for their inclusiveness, fairness, explainability, privacy, safety and other trustworthiness issues — called in the report collectively AI governance tools – can improve such issues. While some AI governance tools provide reassurance to the public and to regulators, the tools too often lack meaningful oversight and quality assessments. Incomplete or ineffective AI governance tools can create a false sense of confidence, cause unintended problems, and generally undermine the promise of AI systems. The report contains rich background details, use cases, potential solutions to the problems discussed in the report, and a global index of AI Governance Tools.