Assessing Trustworthy AI

The Ethics Guidelines for Trustworthy Artificial Intelligence (AI) is a document prepared by the High-Level Expert Group on Artificial Intelligence (AI HLEG). This independent expert group was set up by the European Commission in June 2018, as part of the AI strategy announced earlier that year.

The following Chapter sets out a non-exhaustive Trustworthy AI assessment list (pilot version) to operationalise Trustworthy AI. It particularly applies to AI systems that directly interact with users, and is primarily addressed to developers and deployers of AI systems (whether self-developed or acquired from third parties). This assessment list does not address the operationalisation of the first component of Trustworthy AI (lawful AI). Compliance with this assessment list is not evidence of legal compliance, nor is it intended as guidance to ensure compliance with applicable law. Given the application-specificity of AI systems, the assessment list will need to be tailored to the specific use case and context in which the system operates. In addition, this chapter offers a general recommendation on how to implement the assessment list for Trustworthy AI though a governance structure embracing both operational and management level.
The assessment list and governance structure will be developed in close collaboration with stakeholders across the public and private sector. The process will be driven as a piloting process, allowing for extensive feedback from two parallel processes:

a) a qualitative process, ensuring representability, where a small selection of companies, organisations and institutions (from different sectors and of different sizes) will sign up to pilot the assessment list and the governance structure in practice and to provide in-depth feedback;
b) a quantitative process where all interested stakeholders can sign up to pilot the assessment list and provide feedback through an open consultation.

After the piloting phase, we will integrate the results from the feedback process into the assessment list and prepare a revised version in early 2020. The aim is to achieve a framework that can be horizontally used across all applications and hence offer a foundation for ensuring Trustworthy AI in all domains. Once such foundation has been established, a sectorial or application-specific framework could be developed.

Governance

Stakeholders may wish to consider how the Trustworthy AI assessment list can be implemented in their organisation. This can be done by incorporating the assessment process into existing governance mechanisms, or by implementing new processes. This choice will depend on the internal structure of the organisation as well as its size and available resources.

Research demonstrates that management attention at the highest level is essential to achieve change. It also demonstrates that involving all stakeholders in a company, organisation or institution fosters the acceptance and the relevance of the introduction of any new process (whether or not technological). Therefore, we recommend implementing a process that embraces both the involvement of operational level as well as top management level.

Level

Relevant roles (depending on the organisation)

Management and Board

Top management discusses and evaluates the AI systems’ development, deployment or procurement and serves as an escalation board for evaluating all AI innovations and uses, when critical concerns are detected. It involves those impacted by the possible introduction of AI systems (e.g. workers) and their representatives throughout the process via information, consultation and participation procedures.

Compliance/Legal department/Corporate responsibility department

The responsibility department monitors the use of the Assessment List and its necessary evolution to meet the technological or regulatory changes. It updates the standards or internal policies on AI systems and ensures that the use of such system complies with the current legal and regulatory framework and to the values of the organisation.

Product and Service Development or equivalent

The Product and Service Development department uses the Assessment List to evaluate AI-based products and services and logs all the results. These results are discussed at management level, which ultimately approves the new or revised AI-based applications.

Quality Assurance

The Quality Assurance department (or equivalent) ensures and checks the results of the Assessment List and takes action to escalate an issue higher up if the result is not satisfactory or if unforeseen results are detected.

HR

The HR department ensures the right mix of competences and diversity of profiles for developers of AI systems. It ensures that the appropriate level of training is delivered on Trustworthy AI inside the organisation.

Procurement

The procurement department ensures that the process to procure AI-based products or services includes a check of Trustworthy AI.

Day-to-day Operations

Developers and project managers include the Assessment List in their daily work and document the results and outcomes of the assessment.

 

 

 

Using the Trustworthy AI Assessment List

When using the Assessment List in practice, we recommend paying attention not only to the areas of concern but also to the questions that cannot be (easily) answered. One potential problem might be the lack of diversity of skills and competences in the team developing and testing the AI system, and therefore it might be necessary to involve other stakeholders inside or outside the organisation. It is strongly recommended to log all results both in technical terms and in management terms ensuring that the problem solving can be understood at all levels in the governance structure.

This Assessment List is meant to guide AI practitioners to develop, deploy and use Trustworthy AI. The assessment should be tailored to the specific use case in a proportionate way. During the piloting phase, specific sensitive areas might be revealed and the need for further specifications in such cases will be evaluated in the next step. While this Assessment List does not provide concrete answers to address the raised questions, it encourages reflection on the steps that can help ensuring AI systems’ trustworthiness and on potential steps should be taken in this regard.

Relation to existing law and processes

It is also important for those involved in AI development, deployment and use to recognise that there are various existing laws mandating particular processes and the prohibition of particular outcomes, which may overlap and coincide with some of the measures listed in the Assessment List. For example, data protection law sets out a series of legal requirements that must be met by those engaged in the collection and processing of personal data. Yet, because Trustworthy AI also requires the ethical handling of data, internal procedures and policies aimed at securing compliance with data protection laws might also help to facilitate the ethical data handling and can hence complement existing legal processes. Compliance with this Assessment List is not, however, evidence of legal compliance, nor is it intended as guidance to ensure compliance with applicable laws. Rather, the aim of this Assessment List is to offer a set of specific questions to addressees in seeking to ensure that their approach to AI development, deployment is oriented towards, and seeks to secure, Trustworthy AI.

Similarly, many AI practitioners already have existing assessment tools and software development processes in place to ensure compliance also with non-legal standards. The below assessment should not necessarily be carried out as a stand-alone exercise, but can be incorporated into such existing practices.

Trustworthy AI Assessment List (Pilot Version)

1. Transparency

Traceability:

Did you put measures in place that can ensure traceability? This could entail the documentation of:

Methods used for designing and developing the algorithmic system:

in case of a rule-based AI system, the method of programming or how the model was built should be documented;

in case of a learning-based AI system, the method of training the algorithm, including which input data was gathered and selected, and how this occurred, should be documented.

Methods used to test and validate the algorithmic system:

in case of a rule-based AI system, the scenarios or cases used in order to test and validate should be documented;

in case of a learning-based model, information about the data used to test and validate should be documented.

Outcomes of the algorithmic system:

The outcomes of or decisions taken by the algorithm, as well as potential other decisions that would result from different cases (e.g. for other subgroups of users) should be documented.

Explainability:

Did you assess the extent to which the decisions and hence the outcome made by the AI system can be understood?

Did you ensure that an explanation as to why a system took a certain choice resulting in the a certain outcome can be made understandable to all users that may desire an explanation?

Did you assess to what degree the system’s decision influence the organisation’s decision-making processes?

Did you assess why this particular system was deployed in this specific area?

Did you assess the business model concerning this system (e.g. how does it create value for the organisation)?

Did you design the AI system with interpretability in mind from the start?

Did you research and try to use the simplest and most interpretable model possible for the application in question?

Did you assess whether you can analyse your training and testing data? Can you change and update this over time?

Did you assess whether you have any options after the model’s training and development to examine interpretability, or whether you have access to the internal workflow of the model?

Communication:

Did you communicate to (end)-users – through a disclaimer or any other means – that they are interacting with an AI system and not with another human? Did you label your AI system as such?

Did you put in place mechanisms to inform users on the reasons and criteria behind the AI system’s outcomes?

Is this clearly and intelligibly communicated to the intended users?

Did you establish processes that take into account users’ feedback and use this to adapt the system?

Did you also communicate around potential or perceived risks, such as bias?

Depending on the use case, did you also consider communication and transparency towards other audiences, third parties or the general public?

Did you make clear what the purpose of the AI system is and who or what may benefit from the product/service?

Have the usage scenarios for the product been specified and clearly communicated, considering also alternative forms of communication to ensure that it is understandable and appropriate for the addressed user?

Depending on the use case, did you think about human psychology and potential limitations, such as risk of confusion, confirmation bias or cognitive fatigue?

Did you clearly communicate characteristics, limitations and potential shortcomings of the AI system:

in case of development: to whoever is deploying it into a product or service?

in case of deployment: to the end-user or consumer?

2. Technical robustness and safety

Resilience to attack and security:

Did you assess potential forms of attack to which the AI system could be vulnerable?

In particular, did you consider different types and natures of vulnerabilities, such as data pollution, physical infrastructure, cyber-attacks?

Did you put measures or systems in place to ensure the integrity and resilience of the AI system against potential attacks?

Did you assess how your system behaves in unexpected situations and environments?

Did you consider whether or not, and to what degree your system could be dual-use? If so, did you take suitable preventative measures against this case (including for instance not publishing the research or deploying the system)?

Fall-back plan and general safety:

Did you ensure that your system has a sufficient fallback plan should it encounter adversarial attacks or other unexpected situations (e.g. technical switching procedures or asking for a human operator before proceeding)?

Did you consider the level of risk raised by the AI system in this specific use case?

Did you put any process in place to measure and assess risks and safety?

Did you provide the necessary information in case of a risk for human physical integrity? 

Did you consider an insurance policy to deal with potential damage from the AI system?

Did you identify the potential safety risks of (other) foreseeable uses of the technology, including accidental or malicious misuse thereof? Is there a plan to mitigate or manage these risks?

Did you assess whether there is a probable chance that the AI system may cause damage or harm to users or third parties? If so, did you assess the likelihood, potential damage, impacted audience and severity?

In case there is a risk of the AI system causing damage, did you consider liability and consumer protection rules, and how did you take these into account?

Did you consider the potential impact or safety risk to the environment or to animals?

Did your risk analysis consider whether security or network problems (for example cybersecurity hazards) pose safety risks or damage due to unintentional behaviour of the AI system?

Did you estimate the likely impact of a failure of your AI system that leads to providing wrong results, that leads to your system being unavailable, or to your system providing societally unacceptable results (e.g. discriminatory practices)?

Did you define thresholds and governance for the above scenarios to trigger alternative/fall-back plans?

Did you defined and test fallback plans?

Accuracy

Did you assess what level and definition of accuracy would be required in the context of the AI system and use case?

Did you assess how accuracy is measured and assured?

Did you put in place measures to ensure that the data used is comprehensive and up to date?

Did you put in place measures in place to assess whether there is a need for additional data, for example to improve accuracy or to eliminate bias?

Did you assess the harm that would be caused if the AI system makes inaccurate predictions?

Did you put in place ways to measure whether your system is making an unacceptable number of inaccurate predictions?

If inaccurate predictions are being made, did you put in place a series of steps to resolve the issue?

Reliability and reproducibility:

Did you put in place a strategy to monitor and test that the AI system meets the goals, purposes and intended applications?

Did you test whether any specific contexts or particular conditions need to be taken into account to ensure reproducibility?

Did you put in place processes or methods for verification to measure and ensure different aspects of reliability and reproducibility?

Did you put in place processes for describing when an AI system fails in some types of settings?

Did you clearly document and operationalise these processes for the testing and verification of the reliability of AI systems?

Did you put in place a mechanisms or communication to assure (end-)users of the reliability of the AI system?

3. Privacy and data governance

Respect for privacy and data Protection:

Depending on the use case, did you establish a mechanisms that allows others to flag issues related to privacy or data protection issues concerning the AI system’s processes of data collection (for training as well as operation) and data processing?

Did you assess the type and scope of data in your data sets (e.g. whether they contain personal data)?

Did you consider ways to develop the AI system or train the model without or with minimal use of potentially sensitive or personal data?

Did you build in mechanisms for notice and control over personal data depending on the use case (such as valid consent and possibility to revoke, when applicable)?

Did you take measures to enhance privacy, such as via encryption, anonymisation and aggregation?

Where a Data Privacy Officer (DPO) exists, did you involve this person at an early stage in the process?

Quality and integrity of data:

Did you align your system with potential relevant standards (e.g. ISO, IEEE) or widely adopted protocols for your daily data management and governance?

Did you establish oversight mechanisms for data collection, storage, processing and use?

Did you assess the extent to which you are in control of the quality of the external data sources used?

Did you put in place processes to ensure the quality and integrity of your data? Did you consider other processes? How are you verifying that your data sets have not been compromised or hacked?

Access to data:

What protocols, processes and procedures were followed to manage and ensure proper data governance?

Did you assess who can access users’ data, and under what circumstances?

Did you ensure that these persons are qualified and required to access the data, and that they have the necessary competences to understand the details of data protection policy?

Did you ensure an oversight mechanism to log when, where, how, by whom and for what purpose data was accessed?

4. Diversity, non-discrimination and fairness

Unfair bias avoidance:

Did you ensure a strategy or a set of procedures to avoid creating or reinforcing unfair bias in the AI system, both regarding the use of input data as well as for the algorithm design?

Did you assess and acknowledge the possible limitations stemming from the composition of the used data sets?

Did you consider diversity and representativeness of users in the data? Did you test for specific populations or problematic use cases?

Did you research and use available technical tools to improve your understanding of the data, model and performance?

Did you put in place processes to test and monitor for potential biases during the development, deployment and use phase of the system?

Depending on the use case, did you ensure a mechanism that allows others to flag issues related to bias, discrimination or poor performance of the AI system?

Did you consider clear steps and ways of communication regarding how and to whom such issues can be raised?

Did you consider not only the (end)-users but also others potentially indirectly affected by the AI system?

Did you assess whether there is any possible decision variability that can occur under the same conditions?

If so, did you consider what would be the possible causes of this?

In of variability, did you establish a measurement or assessment mechanism of the potential impact of such variability on fundamental rights?

Did you ensure an adequate working definition of “fairness” that you apply in designing AI systems?

Is your definition commonly used? Have you considered other definitions before choosing this one?

Did you ensure a quantitative analysis or metrics to measure and test the applied definition of fairness?

Did you establish mechanisms to ensure fairness in your AI systems? Did you consider other potential mechanisms?

Accessibility and universal design:

Did you ensure that the AI system accommodates a wide range of individual preferences and abilities?

Did you assess whether the AI system usable by those with special needs or disabilities or those at risk of exclusion? How was this designed into the system and how is it verified?

Did you ensure that information about the AI system is accessible also to users of assistive technologies?

Did you involve or consult this community during the development phase of the AI system?

Did you take into account the impact of your AI system on the potential user audience?

Is the team involved in building the AI system representative of your target user audience? Is it representative of the wider population, considering also of other groups who might tangentially be impacted?

Did you assess whether there may be persons or groups who might be disproportionately affected by negative implications?

Did you get feedback from other teams or groups that represent different backgrounds and experiences?

Stakeholder participation:

Did you consider a mechanism to include the participation of different stakeholders in the AI system’s development and use?

Did you pave the way for the introduction of the AI system in your organisation by informing and involving impacted workers and their representatives in advance?

5. Human agency and oversight

Fundamental rights:

In those use cases where there can potentially be a negative impact on fundamental rights, did you carry out a fundamental rights impact assessment? Did you identify and document potential trade-offs made between the different principles and rights?

Does the AI system interact with decision-making by human end users (e.g. recommended actions or decisions to take, presenting of options)?

In those cases, is there a risk that the AI system affects human autonomy by interfering with the end user’s decision-making process in an unintended way?

Did you consider whether the AI system should communicate to users that a decision, content, advice or outcome is the result of an algorithmic decision?

In case the AI system features a chat bot or conversational system, are the human end users made aware of the fact that they are interacting with a non-human agent?

Human agency:

In case the AI system is implemented in work and labour process, did you consider the task allocation between the AI system and human workers for meaningful interactions and appropriate human oversight and control?

Does the AI System enhance or augment human capabilities?

Did you take safeguards to prevent overconfidence or overreliance in the AI system in work processes?

Human oversight:

Did you consider what would be the appropriate level of human control for the particular AI system and use case?

Can you describe the level of human control or involvement, if applicable? Who is the “human in control” and what are the moments or tools for human intervention?

Did you put in place mechanisms and measures to ensure such potential human control or oversight, or to ensure that decisions are taken under the overall responsibility of human beings?

Did you take any measures to enable audit and to remedy issues related to governing AI autonomy?

In case there is a self-learning or autonomous AI system or use case, did you put in place more specific mechanisms of control and oversight?

What kind of detection and response mechanisms did you establish to assess whether the something could go wrong?

Did you ensure a "stop button" or procedure to safely abort an operation where needed? Does this procedure abort the process entirely, in part or delegate control to a human?

6. Societal and environmental well-being

Sustainable and environmentally friendly AI:

Did you put in place mechanisms to measure the environmental impact of the AI system’s development, deployment and use (e.g. energy used by data centre, type of energy used by the data centres, etc.)?

Did you ensure measures to reduce the environmental impact of your AI system’s life cycle?

Social impact:

In case the AI system directly interacts with humans:

Did you assess whether the AI system encourages humans to develop attachment and empathy towards the system?

Did you ensure that the AI system clearly signals that its social interaction is simulated and that it has no capacities of “understanding” and “feeling”?

Did you ensure that the social impacts of the AI system are well understood? For example, did you assess whether there is a risk of job loss or de-skilling of the workforce? What steps have been taken to counteract such risks?

Society and democracy:

Did you assess the broader societal impact of the AI system’s use beyond the individual (end-)user, such as potentially indirectly affected stakeholders?

7. Accountability

Auditability:

Did you put in place mechanisms that facilitate the system’s auditability by internal and/or independent actors, such as ensuring traceability and logging of the AI system’s processes and outcomes?

Minimising and reporting negative Impact:

Did you carry out a risk or impact assessment of the AI system which takes into account different stakeholders that are directly and indirectly affected?

Did you put in place training and education frameworks to develop accountability practices?

Which workers or branches of the team are involved? Does it go beyond the development phase?

Do these trainings also teach the potential legal framework applicable to the AI system?

Did you consider establishing an ‘ethical AI review board’ or a similar mechanism to discuss overall accountability and ethics practices, including potentially unclear grey areas?

In addition to internal initiatives or frameworks to oversee ethics and accountability, is there any kind of external guidance or were auditing processes set up as well?

Are there any processes in place for third parties (e.g. suppliers, consumers, distributors/vendors) or workers to report potential vulnerabilities, risks or biases in the AI system/application?

Documenting trade-offs:

Did you establish a mechanism to identify relevant interests and values implicated by the AI system and potential trade-offs between them?

What process do you use to do decide on such trade-offs? Did you ensure that the trade-off decision was documented?

Ability to redress:

Did you establish an adequate set of mechanisms that allows for redress in case of the occurrence of any harm or adverse impact?

Did you put mechanisms in place both to provide information to (end-)users/third parties about opportunities for redress?

 

 

We invite all stakeholders to pilot this Assessment List in practice and to provide feedback on its implementability, completeness, relevance for the specific AI application or domain, as well as overlap or complementarity with existing compliance or assessment processes. Based on this feedback, a revised version of the Trustworthy AI assessment list will be proposed to the Commission in early 2020.