AI in healthcare needs patient-centred regulation to avoid discrimination – new commentary
New commentary published in the Journal of the Royal Society of Medicine warns that current risk-based regulatory approaches to Artificial Intelligence (AI) in healthcare fall short in protecting patients, potentially leading to over- and undertreatment as well as discrimination against patient groups.
The authors found that while AI and machine learning systems can enhance clinical accuracy, concerns remain over their inherent inaccuracy, opacity, and potential for bias which are not adequately addressed by the current regulatory efforts introduced by the European Union’s AI Act.
Passed in 2025, the AI Act categorises medical AI as "high risk" and introduces strict controls on providers and deployers. But the authors argue this risk-based framework overlooks three critical issues: individual patient preferences, systemic and long-term effects of AI implementation, and the disempowerment of patients in regulatory processes.
“Patients have different values when it comes to accuracy, bias, or the role AI plays in their care,” said lead author Thomas Ploug, Professor of Data and AI Ethics at Aalborg University, Denmark. “Regulation must move beyond system-level safety and account for individual rights and participation.”
The authors call for the introduction of patient rights relating to AI-generated diagnosis or treatment planning, including the right to:
- request an explanation;
- give or withdraw consent;
- seek a second opinion; and
- refuse diagnosis or screening based on publicly available data without consent.
They warn that without urgent engagement from healthcare stakeholders - including clinicians, regulators, and patient groups - these rights risk being left behind in the rapid evolution of AI in healthcare.
“AI is transforming healthcare, but it must not do so at the expense of patient autonomy and trust,” said Professor Ploug. “It is time to define the rights that will protect and empower patients in an AI-driven health system.”
Journal
Journal of the Royal Society of Medicine
Article Title
The need for patient rights in AI-driven healthcare – risk-based regulation is not enough
Technology and data will save lives: Analytics can help deliver improved healthcare - new book
Carnegie Mellon University
Lives could be saved and treatment times cut, with data-driven decision-making, according to the book, written by an international team. Analytics Edge in Healthcare, was written for health professionals, policymakers and decision-makers, by Holly Wiberg from Carnegie Mellon University’s Heinz College of Information Systems and Public Policy, Agni Orfanoudaki from the University of Oxford and their former academic supervisor, Dimitris Bertsimas, Vice Provost for Open Learning and Associate Dean of Business Analytics at MIT Sloan School Of Management.
"We have taught analytics in different executive environments and have seen first-hand the power of equipping industry domain experts with these tools to solve real-world problems. We want to bring this education into healthcare. And given the unique opportunities and challenges in the clinical setting, we saw a need to develop a new resource that introduces these methods in context, with tailored approaches and examples," said Wiberg.
"Our goal in writing this book was to bridge that gap. Every case study we present stems from our own research and collaborations in the field. These examples show how analytics can drive meaningful change in healthcare settings—and this book is our way of helping bring that vision to life," Orfanoudaki continued.
While many concerns have been voiced about the perceived detrimental impact of AI, on jobs and on consumers’ experience, the team argues there is much to gain, for patients and clinicians, through the use of AI in healthcare management.
Not only could technology manage more effectively hospital bed and appointment allocation, potentially easing squeeze points in the system, but the use of data would result in improved treatment and better health outcomes.
The book includes case studies spanning clinical and operational applications with demonstrated practical impact. "Take transplant care in the U.S.—data-driven techniques helped the national transplant agency improve the fairness and efficiency of organ allocation. That translated into lives saved each year," said Orfanoudaki.
The team explained how these methods can also be used for care efficiency - with managing bed allocation or capacity managing.
"Optimisation and machine learning are being used to solve these classic operations problems that are not directly clinically oriented, but have far reaching implications for how smoothly a hospital runs, which impacts a patient’s experience and outcomes," Wiberg said.
The book is packed with examples of how analytics have helped healthcare management, largely drawn from the author team’s own work with collaborators across various health systems. The authors attribute their successful track record to these clinical partnerships, and they hope their book will enable similar opportunities for other teams.
Using data and AI to create better health care systems
Weill Cornell Medicine
image:
Dr. Peter Steel
view moreCredit: Weill Cornell Medicine
Academic medical centers could transform patient care by adopting principles from learning health systems principles, according to researchers from Weill Cornell Medicine and the University of California, San Diego. In this approach, information from electronic health records, clinical trials and day-to-day hospital operations is analyzed in real-time to uncover insights that continuously improve patient care.
The perspective, published June 17 in npj Health Systems, reasons that a smarter, more efficient and more equitable model of care can be created by harnessing existing data to support system-wide learning. Yet, adoption of this model remains limited.
“Integrating diverse databases is part of creating a dynamic health care system,” said lead author Dr. Peter Steel, associate professor of clinical emergency medicine at Weill Cornell Medicine and an emergency medicine physician at NewYork-Presbyterian/Weill Cornell Medical Center. “Practitioners will be able to more easily and quickly see what’s working and what’s not; and what’s driving up unnecessary costs.”
Also, contributing to the perspective are Dr. Robert Harrington, the Stephen and Suzanne Weiss Dean of Weill Cornell Medicine, and Dr. Christopher Longhurst and Dr. Gabriel Wardi, both from the University of California, San Diego.
The authors say establishing a learning health system is especially important now as academic institutions are facing financial strain caused by rising research costs, declining margins and growing patient expectations. The perspective is a call to action for academic health centers to make systemic changes by rethinking how they generate and apply knowledge.
Barriers to Implementation
The idea behind this approach is not new—medical researchers first envisioned learning health systems when hospitals transitioned from paper to electronic health records. However, electronic health records were designed primarily for the convenience of clinicians and patients, rather than for researchers and quality improvement initiatives. Data siloes further complicate establishing learning health systems. Information—patient histories, lab results, imaging or billing records—is stored in separate, disconnected systems that don’t communicate with each other.
Consequently, it can often take years to gather and analyze data needed to improve patient care, the authors noted. A functioning learning health system could shrink this time frame to weeks while maintaining ethical, patient-centric research and using strong security systems to ensure patient privacy. Those insights can then be used to revise treatment guidelines, enhance patient safety and spur innovations.
Part of the issue, the authors argue, is insufficient integration between the people focused on clinical care, research and education. Ideally, future doctors could be taught how to use data to efficiently ask and answer clinical questions that will bring together different stakeholders to collaborate.
“A learning health system, powered by AI, has the potential to elevate clinical care and outcomes,” said Dr. Harrington. “When we enable future clinicians to learn from every clinical encounter, we can improve quality and effectiveness in ways we couldn't before.”
Beyond data organization and analysis, the cost of implementing a learning health system may reach tens of millions of dollars. But the long-term return may be strategic: Years after implementation, health care organizations that successfully utilize this approach could become significantly more competitive than those that do not, the authors said.
Artificial Intelligence is Driving Change
Despite the challenges, the recent advances in artificial intelligence make learning health systems adoption more critical. Patients are starting to expect doctors to leverage AI to deliver personalized, proactive care, but AI depends on clean, well-structured, real-world data. “AI can only fulfill its promise if it’s built on a foundation of learning infrastructure,” Dr. Steel said.
AI tools can analyze huge volumes of medical data quickly, helping doctors spot early warning signs of illness, streamline operations and make faster, more individualized decisions. A learning health system enables essential quality control, ensuring AI tools are continuously monitored for safety, bias and effectiveness.
“Academic medical centers face a rapidly changing funding landscape, even as the costs of technological transformation and administration in health care continue to rise,” Dr. Steel said. “Implementing the learning health system is no longer a theoretical goal, but a strategic imperative.”
Journal
npj Health Systems
Article Title
Learning health system strategies in the AI era
Article Publication Date
17-Jun-2025
