Healthcare AI bias may start with design, not just data

Health systems evaluating AI must look beyond datasets and algorithms to understand the assumptions and priorities embedded within the applications themselves.

Artificial Intelligence By Nathan Eddy | March 11, 2026 | 10:00 PM From left, Gil Bashe of FINN Partners, Hal Wolf, president and CEO of HIMSS and Dr. Isaac Kohane of Harvard Medical School Photo: Nathan Eddy/HFN HIMSS Las Vegas – Artificial intelligence (AI) applications in healthcare are often evaluated based on accuracy, data quality and clinical performance. But another factor may play an equally important role: the values embedded in how those tools are designed.During a session at the HIMSS Global Health Conference & Exhibition, Hal Wolf, president and CEO of HIMSS, Isaac Kohane of Harvard Medical School, and Ran Balicer, chief innovation officer at Clalit Health Services, who joined via video link, discussed how the “value proposition” built into AI systems can shape the way those tools operate in real-world healthcare settings.The discussion, moderated by Gil Bashe of FINN Partners, highlighted why health systems evaluating AI must look beyond datasets and algorithms to understand the assumptions and priorities embedded within the applications themselves.All noted AI is rapidly moving from experimentation into the operational core of healthcare, shaping how data flows, how clinicians access information and how decisions are made across health systems.However, one of the biggest risks in healthcare AI may not lie in the datasets or algorithms themselves—it may lie in the values embedded in how these tools are designed.Bashe opened the discussion by noting that enthusiasm for AI in healthcare is growing quickly, but the real question facing leaders is how those tools will ultimately reshape care delivery.What began as exploratory technology is now embedded in clinical systems and operational workflows, influencing how information moves and how decisions are made.Major investments are arriving at a moment when health systems face an aging population, rising clinical complexity and workforce shortages. With healthcare accounting for roughly one-fifth of the U.S. economy, the stakes are enormous.“AI introduces new questions about governance and responsibility for outcomes,” Bashe said, emphasizing that these are leadership decisions rather than purely technical ones.Kohane warned healthcare organizations face a paradox when it comes to AI adoption: they risk moving both too slowly and too quickly at the same time.On the one hand, health systems have spent considerable time debating safety and governance, and many organizations have been cautious about deploying AI beyond areas such as revenue cycle management. That slow pace has created an opening for clinicians to experiment independently with external tools.“The disruptive elements are going in without evaluation and oversight,” Kohane said. “The worst outcome will be if the worst practices in healthcare get poured over with concrete by AI.”Wolf said the situation resembles earlier moments of technology adoption in healthcare, including the introduction of Wi-fi connectivity inside hospitals.At the time, clinicians often brought their own routers into facilities to access data more easily, creating security vulnerabilities that organizations later had to address.“It took a long time for the infrastructure to catch up,” Wolf said. “The same thing is happening with AI.”The challenge, he added, is that demand for AI tools is rising faster than governance structures can keep up.“When you have a gap and a demand, and you don’t have structured governance behind it, there’s a risk,” Wolf said. “We have to go faster because of the criticality, and we have to educate all layers within our healthcare community on how to use it.”Balicer said the deeper concern is the possibility of misaligned care at scale if AI systems are adopted without carefully evaluating the assumptions built into them.“Algorithms are just opinions embedded in code,” he said.Even when organizations develop their own tools, he noted, the underlying models often contain embedded values that are difficult to detect.“There’s no such thing as AI neutrality,” Balicer said. “It has an embedded set of values, and they’re hidden from us.”Those embedded priorities can lead to outcomes that conflict with an organization’s clinical or operational goals. For example, even if a health system aligns leadership priorities between clinical and financial teams, an external AI model may still optimize for different objectives.“When we use an LLM out of the box, we get a whole set of values that could be completely misaligned with our intentions,” Balicer said.For that reason, panelists emphasized that evaluating AI tools must go beyond testing accuracy or performance metrics.This means healthcare organizations must also examine the assumptions embedded in the systems themselves and invest in governance frameworks capable of identifying those risks early.Without that oversight, they warned, AI could amplify existing problems across healthcare systems rather than solving them. Topic: Artificial Intelligence, Operations, Quality and Safety