Building AI trust: The

key role of explainability

AI systems are powerful but often operate like “black boxes,” shrouded
in mystery. Here’s how companies can shed some light and drive
adoption of AI solutions that users trust and understand.
by Carlo Giovine and Roger Roberts
with Mara Pometti and Medha Bankhwal

November 2024
Artificial intelligence has the potential to Why invest in this capability:
deliver massive gains in economic productivity Getting ROI from XAI
and enable positive social change around the As with any investment in an uncertain environment,
world. So it’s little surprise that the number of organizations seeking to enhance AI explainability
companies adopting AI-powered software, tools, must consider the benefits and costs to decide
and platforms, including generative AI (gen AI), has how and when to act in the absence of perfect
surged throughout 2024. But that enthusiasm has information on the potential upside and risks
been accompanied by a fair amount of trepidation: involved. Today’s AI landscape is fraught with
in McKinsey research, 91 percent of respondents uncertainty, and in this context, leading AI labs like
doubt their organizations are “very prepared” Anthropic are making bets that investments in XAI
to implement and scale the technology safely will pay off as a path to differentiation in a crowded
and responsibly.1 Such doubt is understandable. field of foundation model builders (see sidebar
Along with its potential to boost productivity “The evolution of XAI and today’s challenges”).
and innovation, gen AI in particular poses novel Meanwhile, enterprises are seeking to meet the
risks—for example, hallucinations and inaccurate or expectations of their stakeholders and regulators.
biased outputs—which threaten to undermine trust
in the technology. One thing is certain: demand for XAI is rising. As
global AI regulations begin to take shape, the need
To capture the full potential value of AI, for explainability and interpretation is increasing,
organizations need to build trust. Trust, in fact, with more organizations seeking guidelines on
is the foundation for adoption of AI-powered how to determine what level of explainability to
products and services. After all, if customers adopt and how much information to release about
or employees lack trust in the outputs of AI their models. The EU AI Act, for example, imposes
systems, they won’t use them. Trust in AI comes specific transparency requirements for different
via understanding the outputs of AI-powered AI use cases classified according to its risk-based
software and how—at least at a high level—they framework. For example, in the case of high-risk
are created. Organizations increasingly recognize AI systems—such as systems used in recruitment,
this. In a McKinsey survey of the state of AI in 2024, like résumé-ranking software—organizations are
40 percent of respondents identified explainability required to provide information about the system’s
as a key risk in adopting gen AI. Yet at the same capabilities, limitations, data lineage, and the logic
time, only 17 percent said they were currently behind the decisions it makes.
working to mitigate it.2
Imagine driving a car. Setting a speed limit of
This conundrum has raised the need for enhanced 45 miles per hour is useless if your vehicle lacks
AI explainability (XAI)—an emerging approach to a speedometer that indicates where you are
building AI systems designed to help organizations relative to the standard. Similarly, to respond
understand the inner workings of those systems to AI regulations, organizations need methods
and monitor the objectivity and accuracy of their that provide visibility into how AI models are
outputs. By shedding some light on the complexity built and how they can be tested before release.
of so-called black-box AI algorithms, XAI can Organizations also need observability solutions
increase trust and engagement among those that provide sufficient insight into their AI models
who use AI tools. This is an essential step as to ensure they comply with regulations and their
AI initiatives make the difficult journey from own values and standards. This raises crucial
early use case deployments to scaled, enterprise- questions: Are organizations prepared to deliver this
wide adoption. level of transparency? Do they have the necessary
capabilities and technologies in place? Have

1
“Implementing generative AI with speed and safety,” McKinsey Quarterly, March 13, 2024.
2
“The state of AI in early 2024: Gen AI adoption spikes and starts to generate value,” McKinsey, May 30, 2024.

Building AI trust: The key role of explainability 2

The evolution of XAI and today’s challenges

The field of AI explainability has evolved example, a ProPublica investigation into for example, has provided significant
significantly in recent years. Early AI the COMPAS algorithm, used by US courts improvements to techniques for LLM
tools, employing rule-based systems to assess the likelihood of a defendant explainability and interpretability. Tools
and decision trees, were relatively simple reoffending, revealed systematic bias to interpret the behavior of language
and transparent by design. However, as against African American defendants. models, including OpenAI’s transformer
machine learning models have grown Unfortunately, addressing these concerns debugger, are new and only beginning to be
more complex, it has become more is no simple matter. One major issue is the understood and implemented. An example
difficult to trace the reasons underpinning increasing complexity of advanced large of how tech companies are incorporating
their decision-making processes. The language models (LLMs), which rely on explainability tools into their platforms
early 2000s saw the development of deep neural networks and often operate is Google’s Vertex Explainable AI, which
methods like local interpretable model- as black boxes, with opaque decision- enhances understanding of generative
agnostic explanations (LIME) and making processes. And the lack of access AI and LLMs through feature-based and
Shapley additive explanations (SHAP), to the architecture of proprietary models example-based explanations that give
which provided insights into individual makes it difficult to understand how they users insights into model predictions by
predictions of complex models. Google operate. Previously, teams could control identifying influential features in complex
introduced its What-If Tool, enhancing fairness by curating training data and generative models like transformer-based
model transparency through interactive applying calibration techniques. However, LLMs. In addition, recent community-driven
visualizations; IBM released the AI today’s LLMs resist such control, making research, like work on behavior analysis
Explainability 360 tool kit; and DARPA explainability difficult. Finally, organizations at the head level of LLM architectures,
produced an Explainable AI (XAI) program, increasingly face a trade-off between reflects growing momentum toward
which further advanced the field by model accuracy and interpretability: more unpacking model behaviors. The scale and
developing comprehensive tool kits and complex models and LLMs often achieve complexity of more mature techniques for
techniques to interpret AI models. higher accuracy but at the cost of being unpacking these intricate systems present
less interpretable and harder to explain. unprecedented challenges, but even if
In the meantime, several highly publicized
much work remains, we anticipate progress
missteps have highlighted the growing Innovators are focused on these issues, and
in the coming years.
need for AI explainability. In 2016, for some strides have been made. Anthropic,

platforms and innovators created reliable methods into the SDLC right from the conception of new
of measurement? AI-powered offerings. These experts can form an
XAI center of excellence (COE) to provide expertise
XAI is best thought of as a set of tools and practices and training across teams, reshaping the software
designed to help humans understand why an AI development life cycle and assuring coordinated
model makes a certain prediction or generates enterprise-wide investments in tools and training.
a specific piece of content. The ultimate goal is The COE also can address the need for additional
to ensure that these outputs are of high quality, compute power and cloud consumption to deliver
untainted by bias, inaccuracy, or hallucination. the additional training, post-training, and production
This requires several kinds of investment—in tools, monitoring essential to enhancing explainability.
people, and processes. Building more explainable AI
and machine learning solutions requires deployment How can we ensure a return on these investments
of new technology in the software delivery life cycle in technologies that are often in their early stages?
(SDLC) from the start, when models are trained While XAI is still emerging from research-focused
and tested or as pretrained models are fine-tuned, efforts in academia and R&D labs into real-world
ending when code moves into production and applications, its benefits are far more tangible than
ongoing monitoring and observability are needed. commonly thought. We see five areas where XAI
Expertise in XAI techniques must be built via hiring can deliver a return that drives positive ROI:
and/or training, and the experts must be integrated

Building AI trust: The key role of explainability 3

1. Operational-risk mitigation. By revealing 5. User adoption. XAI helps organizations monitor
how AI models process data and produce the alignment between a model’s outputs
results, XAI enables early identification and and users’ expectations. Greater alignment,
mitigation of potential issues, such as bias or in turn, increases adoption, satisfaction, and
inaccuracy, reducing the risk of operational ultimately top-line growth through innovation
failures and reputational damage. For example, and change management.
many financial services companies use AI in
fraud detection yet often struggle to control or XAI is not and cannot be an afterthought.
understand why their AI systems behave the way Rather, by integrating explainability into the
they do—a potential problem, considering how design, development, and governance of AI
damaging false positives for fraud can be to both systems, organizations can unlock tangible value
the company and the customer. Explainability by facilitating adoption, improving AI model
can increase organizations’ understanding of performance, and boosting user confidence. Nor is
why models flag certain transactions, allowing XAI simply a compliance issue or requirement. It’s a
them to fine-tune their systems or introduce strategic enabler of adoption, trust, and ultimately
greater human oversight. business success—a crucial tool for maximizing the
value of AI technologies across the organization.
2. Regulatory compliance and safety. XAI
ensures that AI systems operate within
industry, ethical, and regulatory frameworks, XAI is a catalyst for a human-
minimizing the risk of noncompliance penalties centered approach to AI
and protecting brand integrity. In human As organizations consider investing to capture a
resources, for example, many recruiters use return from XAI, they first must understand the
AI tools to help screen and select candidates. diverse needs of the different constituencies
Explainability ensures that hiring decisions are involved and align their explainability efforts to
fair and based on relevant criteria, avoiding those needs. Varied stakeholders, situations,
bias and discrimination. and consequences call for different types of
explanations and formats. For instance, the level
3. Continuous improvement. XAI supports the of explainability required for an AI-driven loan
ongoing refinement of AI systems by providing approval system differs from what is needed to
insight into the way the systems function, understand how an autonomous vehicle stops
fostering targeted debugging and iterative at an intersection. A high-risk scenario, such
improvements that help developers keep as a cancer diagnosis, could demand a precise
AI systems aligned with user and business explanation provided rapidly, while the rationale
expectations. Many online retailers, for for a restaurant recommendation can be handled
example, use explainability to improve their with less urgency. Whatever the circumstances,
recommendation engines so they better match the type of explanation needed, which informs the
recommendations with customer preferences. XAI technique required, should be derived with
a human-centered approach—one rooted in the
4. Stakeholder confidence in AI. By attempting needs of the people seeking explanations of an AI’s
to make AI systems understandable, XAI shifts outputs (see sidebar “A human-centered approach
the focus from the technical functioning of to AI explainability”).
models to the users of those models, fostering
a human-centric approach that empowers It’s helpful to think of AI explainability as a bridge
users by boosting their understanding of how AI across a chasm. On one side are the engineers and
outputs are generated. In the healthcare sector, researchers who study and design explainability
for example, AI systems increasingly are used to techniques in academia and research labs, while
identify potential illness. XAI can help doctors on the other side are the end users, who may lack
better understand why these models behave the technical skills but still require AI understanding.
way they do, driving confidence and adoption. In the middle, bridging two extremes, are AI-savvy
humanists, who seek to translate AI explanations

Building AI trust: The key role of explainability 4

A human-centered approach to AI explainability

One of the advantages of XAI is that make AI’s complex logic more accessible— understanding through a shared language
it places humans at the center of AI and therefore truly useful.” of curiosity and exploration,” she says.
efforts. “Data storytelling plays a crucial “It’s not just about demystifying AI; it’s
According to Lupi, explainability efforts
role in bridging the gap between human about finding poetry in how we learn to
are fundamentally about humanizing the
understanding and AI,” says Giorgia Lupi, work alongside it. Think of AI like another
machine’s inner workings and framing AI’s
a partner at the design firm Pentagram and member of the team that is from another
data as stories that reveal its logic. “When
creator of the Data-Humanism Manifesto. culture. We, as humans, need to be open
we embrace storytelling to articulate how
“By translating the machine’s thought to learning how AI reasons, even if it’s not
AI ‘thinks,’ we invite people to connect
processes into narratives that resonate immediately intuitive for us.”
with this new kind of teammate, fostering
with our natural ways of learning, we can

developed by researchers and engineers to respond — Developers need explanations of the models’
to the needs and questions of a diverse group of functioning so they can improve and debug
stakeholders and users. This emerging talent will be these nondeterministic systems, add post-
the key to designing XAI that works for all. training enhancements, and ensure the AI
models deliver expected outcomes.
Stakeholder needs can be broken down into
six personas, each benefitting from different Beyond these different stakeholders, varying
techniques and explanations: contexts and risk scenarios influence the format of
the explanations provided. Explanations can take the
— Executive decision makers require enough form of data visualizations or text reports and will vary
understanding and information about models in technical detail. Understanding the specific needs
to be accountable for their actions with respect of each stakeholder at a particular time is essential
to customers and employees—specifically, to to providing effective and meaningful AI explanations
ensure that models behave in alignment with that meet their unique needs.
the organization’s strategies, brand ethos,
and values.
How does XAI work, and what
— AI governance leaders constitute a cross- techniques are available?
functional group—drawn from functions like To meet these diverse needs, the XAI community
legal, risk, information security, engineering, continues to create new explainability techniques,
and product—that is responsible for shaping AI which involve algorithms to make the decision-
systems in accordance with policies, standards, making processes of AI models more transparent.
and regulations. These can be grouped based on stakeholders’
intents and goals. In general, techniques can
— Affected users need explanations about the be categorized along two dimensions: when the
outcomes they get from AI models. explanation is produced (before or after the model
is trained) and the scope of the explanation (global
— Business users require insights to enhance or local).
everyday decision making, improve processes,
and optimize operational efficiency. The first macro category of XAI techniques
comprises “post-hoc methods,” which involve
— Regulators/auditors require explanations analyzing models after they have been trained,
and interpretability from models to make in contrast to “ante-hoc methods,” which refer
sure they are safe and compliant as rules and to intrinsically explainable models, like decision
regulations evolve. trees. For example, when an HR department

Building AI trust: The key role of explainability 5

seeks to predict which employees may be more Beyond these two macro categories, AI explainability
likely to leave an organization, a decision tree techniques also can be mapped to the needs of
can transparently show why certain employees different personas according to the use cases
are identified as turnover risks based on factors and their context—say, to help developers debug
like job satisfaction, tenure, and workload. In this systems to boost accuracy or strengthen bias
case, an ante-hoc explanation is inherent in the detection or assist product leaders in improving
AI model and its functioning. By contrast, an AI personalization efforts. Much academic and research
model that uses neural networks to predict the work in AI labs is ongoing to enhance and improve
risk of a condition like diabetes or heart disease the range of capabilities available to meet the rising
in a healthcare setting would need to provide demand for XAI and, when paired effectively with
an explanation post hoc, or after the results are user-centered design, to meet the needs of the six
generated. Most often, it does this by applying personas described earlier in this article.
techniques (like SHAP or LIME) to identify which
factors (for example, age, lifestyle, or genetics)
contribute most to the risk score and determine How to start with XAI
whether the risk score is accurate and unbiased. Given that the appropriate techniques used to
get explanations on AI models are informed by
The second dimension differentiates between the personas that need explanations in different
global and local explanations. Global explanations contexts, organizations should consider several
help us understand how an AI model makes steps for embedding explainability methods into
decisions across all cases. Imagine a bank that their AI development.
uses an AI model to assess loan applications. By
using a global explanation tool (such as Boolean Build the right XAI team
rule column generation), the bank can see which Organizations should create truly cross-functional
factors—such as income, debt, and credit score— teams, comprising data scientists, AI engineers,
generally influence its loan approval decisions domain experts, compliance leaders, regulatory
across all customer segments. The global view experts, and user experience (UX) designers. This
reveals patterns or rules that the model follows diverse group ensures that the explainability efforts
across the entire customer base, allowing the bank address technical, legal, and user-centric questions.
to confirm that the model aligns with fair-lending Data scientists and AI engineers will focus on
regulations and treats all customers equitably. the technical aspects, while domain experts and
Deploying XAI algorithms atop a loan application designers provide context-specific insights and
model provides loan officers with a rich base of shape the content and format of the explanations.
information and statistical insights to understand
the factors driving the system’s decisions, allowing Establish the right mindset
them to confidently explain approval patterns to The XAI team should consist of builders, not judges.
customers and regulators. It should focus on accelerating innovation while
assuring the right insights are wrapped around the
Local explanations, in contrast, focus on specific products or services being built. To do this, the team
decisions. Consider a healthcare setting, in which needs to engage while ideas are being shaped into
a doctor uses AI to help diagnose patients. By buildable concepts, not at some later stage. Early
applying a local explanation tool (such as SHAP), involvement helps establish a human-centered
the doctor can see precisely why the model engineering culture in AI while avoiding downstream
predicted a certain condition for that specific conflicts between “engineers” and “explainers.”
patient—showing, for instance, that a patient’s age,
medical history, and recent test results influenced Define clear objectives
the model’s prediction. This level of detail can Set clear goals for AI explainability for each
help doctors understand the model’s reasoning stakeholder persona. Determine what needs to
for individual cases, so they have more trust in its be explained, to whom, and why. This involves
recommendations and can provide more informed, interviewing key stakeholders and end users and
personalized care. understanding their specific needs. Establishing

Building AI trust: The key role of explainability 6

 clear goals helps with selecting the right techniques Monitor and iterate
Find more content like this on the
and tools and integrating them into a build plan. Continuously monitor the effectiveness of the
McKinsey Insights App
explainability efforts and gather feedback from
Develop an action plan stakeholders. Use this feedback to iterate and
Create a strategy to embed explainability improve the explainability processes. Regularly
practices, from the design of AI solutions to the update the models and explanations to reflect
way explanations will be communicated to different changes in the data and business environment.
stakeholders. The former ensures the adoption
of explainability tools across the entire AI life By following this path, organizations can
cycle. The latter involves deciding on the format successfully embed explainability into their AI
Scan • Download • Personalize
(visualizations, textual descriptions, interactive development practices. Then AI explainability will
dashboards) and level of technical detail (high- not only enhance transparency and trust but also
level summaries for executives versus detailed ensure that AI systems are aligned with ethical
technical reports for developers). Ensure that the standards and regulatory requirements and deliver
explanations are clear, concise, and tailored to the the levels of adoption that create real outcomes
audience’s understanding. and value.

Measure metrics and benchmarks

AI explainability also demands a strong push for
industry-wide transparency and standardized As enterprises increasingly rely on AI-driven
benchmarks that not only help users understand decision making, the need for transparency and
AI systems better but also align with regulatory understanding becomes paramount across all levels
expectations. For instance, Hugging Face’s of the organization. Those that fail to build trust will
benchmarking efforts, in which it measures and miss the opportunity to deliver on AI’s full potential
tracks compliance with the EU AI Act, and the for their customers and employees and will fall
COMPL-AI initiative’s focus on assessing and behind their competitors.
measuring model transparency are important
steps toward greater accountability. As these Ultimately, trust will be a key to responsible
frameworks mature, they will be crucial for adoption of artificial intelligence and bridging the
fostering trust and advancing responsible AI gap between a transformative technology and
practices across the industry. its human users. However, trust cannot stand
alone. As a bridge, it must be supported by strong
Select or build appropriate tools pillars. For AI trust, those pillars are explainability,
Adopt and integrate explainability tools that governance, information security, and human-
align with the organization’s needs and technical centricity. Together, these pillars will enable AI and
stack. Some widely used tools include open- its human users to interact harmoniously, making AI
source algorithms such as LIME, SHAP, IBM’s AI work for people and not the other way around—and
Explainability 360 tool kit, Google’s What-If Tool, providing a foundation to ensure that AI systems
and Microsoft’s InterpretM. Ensure that the XAI deliver tangible value to users while preserving
core team keeps an eye on the rapid innovation in respect for human autonomy and dignity.
this domain.

Carlo Giovine is a partner in McKinsey’s London office, where Mara Pometti is a consultant; Roger Roberts is a partner in the
Bay Area office, where Medha Bankhwal is an associate partner.

This article was edited by Larry Kanter, a senior editor in the New York office.

Designed by McKinsey Global Publishing

Copyright © 2024 McKinsey & Company. All rights reserved.

Building AI trust: The key role of explainability 7