Our Process
Stanford Health Care requires that proposed uses of AI tools undergo a formal review process called a Fair, Useful, and Reliable Model (FURM) assessment. HEAL-AI’s ethical assessments are one component of the FURM process.
Our Objectives
The goal of the ethical assessment is to identify potential ethical issues with new uses of AI tools and make recommendations that can help avoid harm. The HEAL-AI lab conducts stakeholder interviews and writes up findings, focusing especially on issues where the values and expectations of AI tool developers, implementers, clinician users, and patients don’t line up. Each report includes recommendations for how the implementation and monitoring plan could be strengthened. Using a coaching model, we provide direct feedback to proposer teams to help them make the strongest possible case that their proposed implementation will help deliver care of the highest quality while treating the workforce fairly and minimizing risks.
We deliver our report and recommendations to a decision-making body at Stanford Health Care, the Data Science Executive Committee, which makes the decision about whether the proposed use of AI will go forward. Where a proposed use case proceeds to deployment, our report helps the health system ensure that the implementation and monitoring plans are robust. Sometimes, we also help the organization say no. By identifying use cases where the risks seem to outweigh the benefits, or where the benefits are too modest to justify the investment, our assessments help decision makers make wise choices navigating a deluge of potential uses of AI.
Our Process
The ethical assessment process is designed to be completed within 30 days by a small team without a high degree of technical mastery of machine learning methods. We work with data scientists at Stanford Health Care to ensure that we learn about key aspects of AI tools, and we work with clinical quality leaders to ensure that we understand how the tools will be integrated into hospital workflows.
The ethics assessment begins during an intake interview, in which our team seeks to clarify how the tool works and will be integrated into clinical workflow and begins to identify ethical issues.
Next, the team conducts semi-structured interviews with key informants from 3 stakeholder groups: proposers (AI tool developers and clinicians leading the implementation planning), users (prospective or, if a pilot is underway, actual), and patients (current or former Stanford Health Care patient volunteers). We speak with 8-10 individuals for each AI tool, a number that balances breadth of stakeholder views with our need to complete assessments on a rapid cycle. Interviews explore perceptions about the potential benefits, risks, and burdens of the AI tool, privacy, informed consent, transparency, cost, bias, human-computer interaction, and fairness.
A critical aspect of our work is engaging patients in the AI governance process. We hold focus groups of members of a learning community of patient representatives called the Patient Partner Panel. We provide this standing group of patient partners with training on AI fundamentals and continuing opportunities to learn about AI tools. In engaging with the Patient Partner Panel, we seek to surface issues that matter to patients, and to ask–rather than assume–what patients want on matters such as patient consent and how to allocate predictive tools’ risk of false positives and false negatives.
To create our written report, we analyze interview and focus group transcripts using thematic content analysis. After the ethics team discusses the draft report, it is submitted for review by the Patient Partner Panel as well as a panel of AI ethics experts. The role of the experts is to provide a final check on the completeness of the issues discussed and the strength of our recommendations. The final report is then delivered to the proposer team and the health system.
We are developing open-source materials providing more details and guidance on the ethical assessment process. Interview guides, templates for tracking assessments, and sample reports from completed ethics evaluations are available, and new materials will be added regularly as our work continues.