Refining Technical Robustness and Safety Questions

  • Shahar Avin profile
    Shahar Avin
    7 November 2019
    Total votes: 4

As a researcher specialising in AI risks, I was encouraged to see the section of the Trustworthy AI Assessment List that focuses on Technical Robustness and Safety. It provides a good start, but I believe it can be futher refined to reflect current research on AI safety and security. In particular, the following questions reflect findings from the report on the Malicious Use of AI, which I helped co-author:

  1. Matching responsibility to capability and risk: how powerful is your AI system relative to other systems currently available? Is your system intended to be deployed in a high-risk setting, e.g. a safety-critical setting, or interacting with vulnerable populations? The higher the risk, the more thought and resources should be allocated to addressing potential risks and building resilience. Do you have a plan to ramp up safety and security if/when your products becomes more powerful or is deployed in more and higher-stakes settings?
  2. What process do you have to manage and mitigate malicious use, misuse and inappropriate use of your AI system? What risk-assessment process is in place to evaluate proposed research and development projects? Is there a responsiblity champion that can advocate for shutting down / restricting projects with high risk of misuse? Are similar risk-assessment processes and decision junctures exist during product development, pre-release, and post-release? Novel information about misuse risks can arise at all stages of the development process. Does the team have sufficient expertise to assess risks of malicious use, misuse and inappropriate use? Collaboration with domain experts, e.g. in information security, law enforecement, journalism, psychology or others (depending on the AI system's domain of application) may be called for.
  3. Does the team have sufficient support in terms of security best practices? Is there a CISO? Are they given sufficient resources to catch up with the rapidly developing field of AI safety and security? Are they connected to the expert community working on these risks? If the team is small, are there external advisors the team can call on for security advice and support?
  4. Is there a process to solicit feedback on potential risks from a wide range of stakeholders? Are intended users included in the design, research and development processes? Who else might be imacted, and are they consulted as well? Different stakeholder communities will often have insights into potential risks and unintended consequesnes that the team may not be aware of.