AI Accountability for Healthcare Equity

Healthcare disparities in the US and and how to ensure AI tools enable more equitable healthcare outcomes

Bobbie Dousa
January 23, 2020
February 10, 2020

AI or machine learning systems for oncology are hailing an array of new insights for the study and treatment of cancer. These systems have demonstrated heightened abilities to predict clinical outcomes, determine changes in drug treatment protocols, and identify cancers by analyzing available and existing health data such as health insurance data, medical images, medical literature, electronic medical records and prescription records. Utilizing machine learning techniques, AI systems have also aided in advancing our understanding of the evolution and biophysics of cancer. Yet amidst the excitement for what other medical insights machine learning might enable stands an increasing and emphatic chorus of experts urging both the developers of these systems and health specialists to ensure that these systems work to mitigate rather than entrench existing healthcare inequities. In this post, I will offer a sense of modes by which health inequities manifest based on my and others’ original research and discuss the kinds of remediation scholars of technology and medicine are currently proposing.

Technology experts and critical algorithm studies scholars implore that we evaluate how these AI models — which increasingly manage and organize our lives — are far from neutral or objective tools. Rather, as mathematician Cathy O’Neil asserts, we must soberly weigh how these instruments are demonstrably encoded with human prejudice, misunderstanding, and bias. One reason for this lies in the fact that these systems and the insights they generate are fundamentally reliant on training data sets composed of existing reference data. Conveying the fallibility of the data-driven paradigm, in 2018 Amazon reported that the company was forced to discontinue its AI hiring and recruitment system because it discriminated against women applicants. Amazon’s recruiting tool relied on resumes submitted to the company over the previous ten years — the majority of which came from men. Accordingly, this reference data organized the algorithm to give preference to male applicants and to screen out women applicants vis-as-vis subtle cues in their resumes such as experience in a women’s organization or education at a women’s college.

In another example, in 2016, investigative journalists uncovered how predictive criminal risk assessment algorithms — software used by US courts to predict how likely a person is to commit a crime in the future and relay a recommendation for sentencing to a presiding judge — are prejudiced toward people of color as they consistently recommend stronger sentencing for Black and Latinx people. Scholars, among them Ruha Benjamin and Safiya Noble, and investigative journalists such as Julia Angwin continue to scrutinize the ramifications of integrating AI systems across a multitude of disparate realms among them: housing, finance, news media, welfare eligibility, social media platforms, popular search engines, and healthcare. Their research confirms that AI systems possess the capacity to exacerbate existing social inequities.

“We must soberly weigh how these instruments are demonstrably encoded with human prejudice, misunderstanding, and bias.”

As the preponderance of data-driven solutions becomes the norm for healthcare specifically, experts demand that we address how these tools can compound existing disparities in healthcare outcomes. One step toward this remediation, researchers assert, involves educating healthcare providers and healthtech developers to ensure they sufficiently comprehend how systemic inequities affect individual health.Having a robust understanding of the causes, consequences, and modes in which health inequities exist not only affords medical specialists and health tech developers a sense of what research and technological solution need to be prioritized to address injustices, but it can also coincide with a self-reflexive method of medical engagement. In other words, knowing how, why, and what health inequities exist, can allow one to approach health interventions with a heightened awareness of the imbrications and the potentially far-reaching implications of their actions and mediations. It would allow one a crucial frame of reference to question how their instruments and actions might be a catalyst for social good or social harm. Fundamentally, they argue that this knowledge is a necessary initial step toward remediating health injustice.

Tina K. Sacks, a medical sociologist who investigates how race and gender impact health outcomes and a proponent of this kind of knowledge building, advocates for a structural approach to understanding health inequities. Sacks asserts:

“Although the dominant paradigm in the United States emphasizes individual choice and responsibility, the empirical evidence indicates that our neighborhoods, schools, jobs, and other factors of day-to-day life shape individual and population health.”

The work of medical historian John Hoberman fleshes out Sacks’ claim. Hoberman analyzes how the historical legacy of racialized thinking is reflected in the contemporary US medical establishment by focusing on how physician racism contributes to health disparities. Hoberman’s research suggests that medical providers rely on false beliefs rooted in racial essentialism — such as the pernicious myth of so-called Black “hardiness” — to determine diagnosis and treatment for Black patients. In addition to racial and gendered oppression, in the past several decades, scholars have also demonstrated that health and well-being strongly correlate with socioeconomic status. Sacks summarizes: “One of the most important systemic inequalities is unequal access to income and wealth, which may lead to poor health behaviors, chronic conditions, and disease.”

In interviews I have conducted with them over the past several months, cancer patients and cancer patient advocates have shared personal accounts that further enumerate how structural social inequities inform and contour healthcare encounters. To illustrate, I conducted an interview with a patient advocate and community health worker in the San Francisco Bay Area with nearly three decades of experience. This patient advocate who I will refer to with the pseudonym, Shelly, primarily serves low-income, women of color diagnosed with cancer. The accounts and perceptions she shared during our discussion offered an acute and poignant sense of how race, gender, and class oppressions intersect within the context of cancer care. For instance, when asked how the patients she serves are impacted by the US health insurance system, she explained:

“When a person has private insurance, they can go to UCSF, they can go to Stanford [hospital] for treatment. They can go to an oncologist who is outside of the system versus people who are receiving emergency MediCal through a breast and cervical cancer treatment fund. Most of [my clients] will go to a public safety hospital… will the doctor know the latest information about clinical trials? Whereas in the case of someone who is at UCSF or Stanford, there’s cutting edge treatment going on. So there’s a difference in the treatment options. That difference impacts longevity, it impacts survival, it impacts recurrence. And the other difference is [related to] how is a person at UCSF or Stanford is going to be treated because they have insurance, whereas a person at a county hospital is going to be waiting even if they have an appointment.”

She elaborated on this point by relating the experience of one of her clients. This patient, a Black woman, was low-income, uninsured, was previously treated for cancer, and was being treated once again at a public hospital for a lingering ailment. When the doctor informed her that her cancer had relapsed, the patient became emotional and visibly upset. Because the doctor had been informed via her available medical records that the patient had had a mental health condition and had previously had a substance abuse issue, the doctor grew afraid and called the hospital security on the patient. After describing this scenario, Shelly raised an additional point: “I find that interesting because I have worked with other women who are not of color [like one individual who] I have seen to be very active and very demanding and never was the security called on her. She was seen as ‘Oh, she’s knowledgeable. She wants to know. She is advocating for herself.’ So when I went back to the hospital with this particular patient I said to the doctor, ‘You know I just want to say that last time, you mentioned some very difficult news...she had a belief about her cancer that it was gone…did you ever think about how devastating that was for her?’ And [the doctor] really couldn’t say anything. But it was because she had already created a picture of this woman by looking at her chart and knowing that she had had a substance problem previously and a mental health condition. So [the doctor] wasn’t trying to get to know [the patient] and maybe felt like, ‘She’s on MediCal, is she really going to complain about me?’ Whereas most of the time I’ve seen women of the dominant culture — they are going to write a letter, they are going to feel empowered. They are going to feel entitled because that’s how they’ve been raised — you have a right to have a voice. Many of my clients that I see that aren’t [of the dominant culture]?…They don’t feel empowered. So something [like this] happens and they just say, ‘You know? This is how the system treats us, they don’t care.”

Shelly’s account of this patient’s disturbing ordeal, coupled with her perceptions of her clients’ general experience, exemplifies the findings of the Institute of Medicine’s (now known as the National Academy of Medicine) seminal study of the causes and ramifications of pervasive healthcare disparities in the US. This study and the volume of research it prompted, found physician bias, whether conscious or unconscious, to be a crucial factor in the production of disproportionate healthcare outcomes. Subsequent empirical studies suggest that people of color and ethnic minorities, women, and other people who occupy vulnerable social positions are most susceptible to the noxious consequences of bias and stereotyping. Dr. Tina K. Sacks further flags that “numerous studies have documented that healthcare providers are unconsciously or unintentionally biased against members of marginalized groups, which ultimately leads to difference in treatment across multiple domains (i.e., specialty care, pain management, mental health services, etc.

Sacks’ most recent book, Invisible Visits: Black Middle Class Women in the American Healthcare System (2019) offers additional insight into how socio-medical inequities shape healthcare encounters and outcomes in the contemporary US. Through a sociological study utilizing qualitative methods in original research, Sacks analyzes how the perception of bias and stereotyping affects middle class Black women who are not poor but remain socially and economically vulnerable. Sacks explains that “the economic foundation of Blacks is substantially more precarious than whites. On average, Black people do not have the savings to weather economic downturns, to take vacations to reduce their stress, or to carry them through an unexpected job loss.” Sacks argues that these women, cognizant of their tenuous economic and social position in society, anticipate racism and discrimination to follow them into the clinic. Because of this, Black middle class women adopt compensatory strategies (e.g., emphasizing their careers and education) in an effort to ensure quality care from medical providers. However, Sacks’ research and additional empirical studies suggests that these strategies have deleterious emotional, psychological, and physiological effects. For instance, anticipating prejudice and discrimination has been shown to activate biological stress response systems more than other types of stress. Social researchers have also documented how longterm exposure to prejudice and discrimination can affect wellbeing and longevity — linking it to premature aging, chronic stress, higher allostatic loads, higher disease burdens as well as higher incidence of debilitating chronic conditions such as hypertension and diabetes.

“Empirical studies suggest that people of color and ethnic minorities, women, and other people who occupy vulnerable social positions are most susceptible to the noxious consequences of bias and stereotyping.”

Myriad experts assert that it is imperative that we are cognizant and considerate of how social inequities are embedded into the health data upon which A.I. systems are built. Due to design and optimization constraints, training data sets primarily utilize the health data profiles of those who can afford and have access to long-term, continuous healthcare as opposed to those who have limited access to care, discontinuous care, or fragmented records. Moreover, data gathered via clinical trials has long been known to be unrepresentative of the US population. Clinical trials routinely fail to recruit people of color and other marginalized people. Recently, investigative journalists at ProPublica reported that Black Americans, Native Americans, and other Americans of color are steeply under-represented in clinical trials for cancer drugs — even when the type of cancer disproportionately affects them. Similar to how researchers at UCSF found albuterol, the commonly used asthma medication, to be less effective for children of African American and Puerto Rican descent, this could, and as these journalists report, has translated to cancer treatments that are least effective for the population most afflicted by the disease. Critically, people of color continue to have disproportionately higher incidence and mortality rates for kidney, breast, prostate and other cancers.

Likewise, AI tools designed to detect skin cancer have proven less adept at diagnosing skin cancer in Black and brown patients than white patients. While people with fair skin have the highest incidence rates for skin cancer — the most prevalent human malignancy — , the mortality rate for people with darker skin such as African Americans is considerably higher. Eric Topol, clinician researcher at the Scripps Institute, contends that this is especially noteworthy for genomic studies driven by machine learning techniques. Topol explains:

“First, people of European Ancestry compose most or all of the subjects in large cohort studies, which means that, second, they are of limited value to most people, as so much of genomics of disease and health is ancestry specific.”

Underlying Topol’s assertion is the suggestion that prioritizing health equity would not only result in more robust scientific and medical knowledge, but would also constitute a step toward engendering quality healthcare for all.

Increasingly, health researchers and medical sociologists such as Sacks and Jonathan Metzl, propose efforts toward remediating health inequities that center on structural competency and knowledge building. Preeminently, they advocate for robust and well-researched efforts at the institutional level that aim to address the enduring effects of historical oppression. For example, Sacks explains that structural competency involves moving beyond obfuscating framings of racism as a troubled American past or simply an individual failing of “bad” or “uneducated” people. Instead, structural competency demands that we analyze how racism —reflected in the racist stereotyping Sacks researches— are structural phenomenon embedded and reproduced in US institutions such as medical schools and healthcare settings.

Kinjal Dave, a researcher of socio-technical systems, presents another opportunity for cultivating structural competency. In a recent essay, Dave critiques the language of “algorithmic bias” by pointing to the historical lineage of the term “bias” as a theory of psychology. Dave explicates: “Because both ‘stereotype’ and ‘bias’ are theories of individual perception, our discussions do not adequately prioritize naming and locating the systemic harms of the technologies we build. When we stop overusing the word “bias,” we can begin to use language that has been designed to theorize at the level of structural oppression, both in terms of identifying the scope of the harm and who experiences it…Today, the social and cognitive psychology literature describes bias as something that is implicit and inevitable in our thought process as we categorize the world around us. The stereotype, therefore, is a foundational component of social psychology’s attempt to theorize why people engage in the kinds of behaviors that allow social hierarchies to persist. Given this history, when we say ‘an algorithm is biased,’ we, in some ways, are treating an algorithm as if it were a flawed individual, rather than an institutional force.” Just as Dave critiques the use of language that places the onus on individual-level problems and solutions, sociologist of technology, Ruha Benjamin, similarly demands that we amend simplistic calls for tech industry diversity as a panacea for the potential for AI systems to reproduce inequities. Benjamin reasons:

“Unless all those diverse people are empowered to challenge discriminatory design processes, diversity is a ruse. We need a complete overhaul of the larger accountability structures that shape tech development, and we definitely can’t wait for Silicon Valley to become more diverse before implementing much stronger regulation and accountability.”

Furthermore, when asked what they recommend: health tech developers and medical researchers prioritize health equity, patient advocates I have interviewed stressed the importance of three domains: ensuring access for all, research involvement, and patient-healthworker education. Firstly, patient advocates urge health tech developers to build and create tech with a fundamental commitment to universal access. In one advocate’s words:

“Your tech needs to include and account for everyone or you will create more barriers to quality care. It’s about making sure you don’t leave certain patients in the ‘Dark Age’ and giving all patients the right treatments for the strongest chance at survival.”

In regard to making tech inclusive rather than exclusionary, patient advocates advise developers and medical researchers to seek out and collaborate with communities of color and other social marginalized groups. They encourage conducting research and creating tech that focuses on and addresses the needs of vulnerable groups. A crucial aspect of such a venture, they assert, involves: building relationships and collaborative problem solving with these interlocutors to ensure that needs of these groups (such as basic access to standard treatment options) as well as the analyst’s research goals are met. Patient advocates stress that those willing to be pioneering in this regard will be hailed as vanguards.

“Prioritizing health equity does not mean creating “neutral” instruments. Instead it calls for both the defiance and honesty to create, for instance, explicitly feminist and antiracist tech.”

This framework, moreover, aligns with critical algorithm studies scholars’ call for researchers to conceive of data-sharing as a mode of gift exchange. Indeed, the word “data” derives from Latin word for “something given.” As a patient I interviewed explained: prioritizing health equity does not mean creating “neutral” instruments. Instead it calls for both the defiance and honesty to create, for instance, explicitly feminist and antiracist tech.

Furthermore, patients and patient advocates recommend cultivating patient and health practitioner education in relation to developments in technology and healthcare as a significant step toward getting patients the right treatment involves informing them of their treatment options and of any potential consequences and side-effects. This mandates that medical care providers be sufficiently educated to guide patients and that education materials are deliberately designed to be accessible and easily comprehendible (e.g., offering treatment pamphlets in several languages rather than solely in the dominant language). For patient advocates, these three recommendations are critically imbricated in one another. One patient advocate succinctly questioned: “How am I supposed to educate a patient about a new treatment or drug they won’t have access to it?” Experts across the realms of healthcare and technology declare that prioritizing health equity necessitates that we create systems of accountability; educate ourselves on the causes and implications of health inequity; and set our aim ultimately at structural interventions.

  • Written by Roberta Dousa, Patient Experience Researcher at
  • Edited by Belle Taylor, Strategic Communications and Partnerships Manager at
  • Thanks to the people who granted interviews and to Dominic Magirr and Yasmeen Kussad for valuable discussions

References consulted:

This is some text inside of a div block.