Advantages, Limitations, Rules, and Responsibilities for the Safe Use of AI in Healthcare, the Prevention of Adverse Events, and Healthcare Organization
Abstract: Artificial intelligence has now become firmly integrated into healthcare and hospital processes, influencing diagnosis, triage assessment, clinical documentation, medication safety, the management of adverse event reporting, and the organization of care pathways. The issue is no longer whether AI should be introduced, but rather under what conditions its use can be considered clinically effective, legally sustainable, ethically acceptable, and genuinely safe for patients. This article examines artificial intelligence as a component of clinical governance and risk management, highlighting its documented benefits in early warning systems, healthcare documentation summarization, safety reporting analysis, the prevention of adverse drug events, and support for complex therapeutic decision-making. At the same time, the paper emphasizes that AI does not automatically reduce clinical risk: it reduces risk only when it is locally validated, integrated into clinical workflows, monitored during real-world use, and subject to explicit human accountability. Contemporary hospitals are thus entering a post-agentic phase in which healthcare professionals are not replaced by machines but are instead called upon to orchestrate systems, data, and cognitive agents while retaining ultimate responsibility for decision-making and the therapeutic relationship.
Keywords: #ArtificialIntelligence #DigitalHealth #ClinicalRisk #PatientSafety #ClinicalGovernance #AIAct #EHDS #EHR20 #AGENAS #SIMES #MedicalDevices #ClinicalDocumentation #AdverseEvents #Sepsis #Pharmacovigilance #Bioethics #HealthcareLiability #MauroCofelice #EthicaSocietas #EthicaSocietasJournal #ScientificJournal #ethicasocietasupli
Giancarlo Stoppani is the President and Founder of Connect Informatics, a software company specializing in digital healthcare solutions. With more than 25 years of experience in technological innovation applied to the healthcare sector, he leads the development of advanced clinical, administrative, and management platforms, with a growing focus on the ethical use of artificial intelligence in healthcare processes.
Artificial intelligence (AI) has now become an established component of hospital practice, influencing diagnosis, triage assessment, clinical documentation, pharmacological management, adverse event reporting, and healthcare organization. The question that truly matters is no longer whether AI should be introduced into healthcare settings, but rather what purpose it genuinely serves within hospitals and under which conditions it can be considered both useful and safe for patients.
This contribution is addressed to a broad audience comprising healthcare managers, clinicians, healthcare professionals, and interested citizens. It intentionally adopts an accessible perspective on a topic that is often confined to highly technical discourse. The central argument is straightforward: a well-governed AI system can substantially reduce clinical risk by identifying early signs of patient deterioration, helping reconstruct the patient’s clinical journey, making safety reports more interpretable, and supporting complex therapeutic decisions. The same technology, however, when deployed without adequate governance, may become a new source of error and inequality.
The following pages address seven key questions: what AI is genuinely intended to accomplish at the bedside; where it is already producing measurable and verifiable benefits; how it is reshaping the structure of clinical risk; which institutional and regulatory frameworks govern its use in Italy and across Europe, including three emerging developments likely to redefine its boundaries; what organizational architecture hospitals require to govern it effectively; which ethical implications arise for healthcare delivery, patient rights, and equitable access to care; and why the transition from doing to orchestrating, enabled by the emergence of high-quality AI systems during the final quarter of 2025, represents the underlying discontinuity driving this transformation.
At the outset, it is important to clarify a methodological aspect. The scientific and clinical content presented herein should be understood solely as an operational framework for clinical governance. The precise definition of validation protocols, Diagnostic-Therapeutic Care Pathways (PDTAs), and formal responsibilities remains the exclusive responsibility of individual healthcare organizations and their scientific and technical committees, in accordance with applicable clinical guidelines, organizational specificities, and healthcare priorities.
What Is Artificial Intelligence Really For in Hospitals?
The starting point must be purpose—the aspect most frequently overlooked. Hospital AI systems are not introduced as symbols of technological modernity, but rather as tools designed to address concrete challenges in clinical practice. These challenges are well known: early signs of patient deterioration are often recognized too late; clinical documentation consumes a substantial portion of physicians’ time; adverse event and near-miss reports remain buried within narrative archives that are difficult to analyze; and complex therapeutic decisions, such as antibiotic prescribing in severe infections, require the rapid integration of heterogeneous information sources.
When appropriately designed, an AI system serves four principal functions, ranked according to their clinical relevance.
First, it enables earlier recognition of deterioration among hospitalized patients. By continuously analyzing vital signs, laboratory results, and clinical variations, AI algorithms can identify signs of impending deterioration several hours before they become evident through conventional observation. No human professional can simultaneously monitor dozens of hospital units with the same level of continuous attention. The primary benefit lies in the time gained to intervene before a cardiac arrest, overt sepsis, or another major adverse event occurs.
Second, AI alleviates the administrative burden associated with clinical work. A significant proportion of physicians’ time is currently devoted to drafting discharge summaries, clinical reports, and care-pathway documentation. Systems capable of generating preliminary drafts—always subject to professional review and validation—can return valuable hours to direct patient care. Recent evidence also suggests a reduction in professional burnout, which itself constitutes a documented risk factor for clinical error and compromised patient safety.
Third, AI can interpret information that would otherwise remain effectively unreadable. Incident reports, near-miss narratives, and free-text comments entered into patient safety systems generate vast volumes of unstructured data. Well-trained language models can classify these reports, identify patterns and similarities, detect recurring themes, and direct risk managers toward areas requiring immediate attention. Causal judgment remains a human responsibility, but the identification of relevant targets for analysis becomes considerably faster and more efficient.
Finally, AI supports decisions characterized by high cognitive complexity. For example, recommendations concerning initial antibiotic therapy in sepsis, generated through comparisons with hundreds of clinically similar cases, do not replace the physician’s judgment but can reduce uncertainty in contexts where time is a critical determinant of outcome. The same principle applies to medication safety, prioritization of care in overcrowded units, and the identification of potentially dangerous drug interactions.
In other words, AI is not intended to replace healthcare professionals; rather, it is designed to restore attention, time, and analytical capacity to those responsible for patient care. For patients, the benefits may be summarized in three principal dimensions: an increased likelihood that clinical deterioration will be detected in a timely manner, a reduction in preventable errors during treatment and discharge processes, and more time devoted by healthcare professionals to the direct therapeutic relationship.
Where AI Is Already Delivering Measurable Benefits
Attention may now be directed to those areas in which documented evidence allows a meaningful assessment of the gap between technological promise and actual outcomes. Current evidence identifies five consolidated domains in which artificial intelligence is already generating tangible and measurable benefits.
Early Recognition of In-Hospital Deterioration
This remains one of the most robust areas of application. A real-world study published in 2025 in an international medical journal evaluated a deep-learning system known as VitalCare across more than 30,000 hospital admissions by comparing outcomes before and after its implementation. The study concluded that a properly developed, validated, and clinically integrated model can contribute to the prevention of major in-hospital adverse events, including in-hospital cardiac arrest, by providing predictive information sufficiently early to enable timely intervention by healthcare professionals.
With regard to sepsis, a second study published in 2025 described the use, within emergency departments, of a clinical decision-support system integrating differential blood count data with machine-learning models [9]. Although much of the existing literature remains retrospective in nature, making prospective multicenter studies necessary to assess the generalizability of findings across diverse clinical settings, the signal emerging from practical experience is nonetheless compelling. In environments characterized by continuous clinical data streams and where early recognition of deterioration is crucial, AI appears capable of providing valuable minutes that may significantly influence therapeutic outcomes.
Clinical Documentation and Time Returned to Care
A second rapidly expanding field concerns discharge documentation summarization and AI-assisted clinical transcription. In these domains, artificial intelligence appears capable of improving not only organizational efficiency but also the overall quality of clinical work, provided that such tools remain firmly anchored to professional supervision and are not regarded as mechanisms for replacing medical judgment.
In this regard, a quality-improvement study published in May 2026 in a JAMA Network journal examined 384 hospital discharges and found that clinicians incorporated AI-generated summaries of the clinical course into discharge letters in more than half of the cases examined, always subject to professional review. Most summaries were judged to pose no potential risk of harm, while only a single summary was considered potentially capable of causing moderate harm. The same study also reported a significant reduction in physician burnout scores among participating clinicians [10].
These findings are relevant not only from a clinical perspective but also from ethical and organizational standpoints. They confirm that AI-assisted documentation can become genuinely beneficial only when it operates within a framework of explicit, traceable, and verifiable medical accountability. Such systems must not evolve into automated mechanisms of decision substitution or improper delegation of professional judgment, in accordance with the principle of meaningful human oversight increasingly emphasized by major international institutions.
Analysis of Patient Safety Reports
A third area of direct relevance to clinical risk management concerns the use of artificial intelligence for the automated analysis of patient safety reports. Particular attention has been devoted to the capacity of Large Language Models (LLMs) to accurately extract, label, and classify safety-related issues contained within narrative reports, as demonstrated by studies published in international journals between 2024 and 2026.
This development offers concrete opportunities for risk management systems. It enables prioritization of the most critical reports, identification of recurring patterns of incidents or vulnerabilities, detection of themes that might escape exclusively manual review, and support for root-cause analysis. Nevertheless, the value of these tools must be understood as analytical support rather than replacement of professional judgment, since the assessment of causation, organizational responsibility, and corrective measures necessarily remains within the domain of human expertise [11].
Medication Safety
A fourth area of application concerns adverse drug events. A review published in Frontiers in Digital Health in December 2025, based on the principal medical literature databases up to June 2025, highlighted that the majority of adverse drug events continue to go undetected. This under-recognition is largely attributable to persistent reliance on voluntary reporting systems and the inherent limitations of traditional surveillance mechanisms.
The review consequently emphasizes the need for tools capable of identifying clinical, prescribing, and organizational warning signals at an earlier stage, thereby preventing relevant safety information from remaining outside established monitoring systems. The same review concludes that the most promising approach combines continuous professional education, cognitive ergonomics, shared decision-support systems involving both clinicians and pharmacists, and multidisciplinary collaborative models. Within this framework, prevention of adverse drug events depends not only on the predictive capabilities of technological tools but also on the quality of the professional organization in which such tools are deployed.
Similarly, pharmacoepidemiological reviews published between 2025 and 2026 suggest increasing feasibility for methodologies aimed at predicting or detecting adverse drug events at an early stage, although substantial methodological heterogeneity remains [12].
Prudent Antibiotic Use in Intensive Care
A fifth application concerns antibiotic prescribing in critically ill patients, particularly AI-supported decision systems used in the management of sepsis. In this context, the KINBIOTICS study, published in March 2025 in a journal dedicated to healthcare human factors, demonstrated that successful implementation depends not only on the clinical quality of generated recommendations but also on interface design, information architecture, comprehensibility of the proposed decision pathway, and the degree of acceptance among healthcare professionals.
The study showed that intensivists’ trust in decision-support systems is influenced by their level of clinical experience, the possibility of critically reviewing recommendations, and the perception that the system is genuinely usable in everyday practice [13].
Systematic reviews published in 2025 identified eight recurring factors associated with the safe adoption of AI in clinical settings: transparency, familiarity, usability, clinical reliability, ethical awareness, validation, contextual adaptability, and professional training. Notably, all these factors concern the relationship between human beings and technology rather than the algorithm itself.
Taken together, these experiences support an important intermediate conclusion: hospital AI creates value when it is designed for a clearly defined task, relies on reliable data, integrates coherently into operational workflows, and remains subject to human review. Conversely, it becomes a source of risk when presented as a universal cognitive shortcut, when deployed outside the context for which it has been validated, or when implemented without adequate indicators capable of verifying its continued performance over time.
How AI Reshapes Clinical Risk
A proper understanding of the impact of artificial intelligence in hospital settings requires reference to the institutional definition of clinical risk management. Within the terminology adopted by the Italian Ministry of Health, clinical risk management encompasses all actions undertaken to improve the quality of healthcare services and ensure patient safety. This framework is operationally linked to ministerial recommendations, sentinel-event monitoring, the Information System for Monitoring Healthcare Errors (SIMES), medication safety initiatives, surgical safety checklists, and the active involvement of patients and citizens [5].
Artificial intelligence should therefore not be regarded as a separate domain from clinical risk management, but rather as a new component of the same system, subject to the same principles of reporting, verification, traceability, and control that already govern pharmaceuticals, medical devices, and clinical procedures.
The most significant qualitative shift lies in the fact that risk is no longer concentrated exclusively in the final clinical act. Instead, it becomes distributed across a broader and more complex chain extending from data quality, outcome definition, model training, interface design, availability of understandable explanations, workflow integration, and system updates, through to ongoing monitoring under real-world conditions of use.
A review published in Frontiers in Digital Health in 2025 on predictive systems for adverse events explicitly highlighted how data-collection bias, population drift, missing information, and poor model interpretability can undermine both performance generalizability and user trust. Likewise, a systematic review published the same year in JMIR examining factors influencing AI acceptance in healthcare confirmed that transparency, usability, clinical reliability, validation across diverse contexts, and preservation of professional control constitute essential prerequisites for the safe deployment of these technologies [7].
Five Families of Error
It follows that the clinical risk associated with the use of artificial intelligence cannot be reduced to the mere paradigm of software failure. Rather, it encompasses a plurality of interdependent vulnerabilities that must be considered as a unified whole. These range from clinical prediction error, which occurs when a model generates an incorrect assessment of a real-world case, to organizational placement error, which emerges when alerts are incorporated into workflows in a manner that produces alarm fatigue or encourages operational shortcuts. They further include accountability error, arising when it is unclear who bears responsibility for interrupting, correcting, or reassessing the functioning of the system; organizational learning error, which occurs when a hospital fails to systematically collect information concerning overrides of algorithmic recommendations, incidents, near misses, and performance deviations; and, finally, equity error, which arises when a system performs effectively for certain groups of patients while proving less accurate or disadvantageous for others, thereby generating inequalities produced by the technology itself.
An exploratory review published by WHO Europe in March 2026 on digital inequalities observed that, within both scientific literature and public policy, issues relating to safety, privacy, and performance continue to receive significantly greater attention than questions of inclusion, equity metrics, and bias assessment. This identifies a gap that Ethica Societascannot overlook, since it directly affects the protection of vulnerable individuals and raises the risk that technological innovation, rather than reducing inequalities, may reproduce or even amplify them [7].
The Institutional Framework: What Is Changing in Italy and Europe
From the Italian perspective, artificial intelligence has now become an integral component of mainstream healthcare planning. This is evidenced by the 2026 Strategic Directive of the Minister of Health, signed in January of the same year, which assigns the Ministry a strategic coordination role aimed at ensuring coherence, ethical compliance, interoperability, non-discrimination, and adequate patient information. The directive further links the adoption of artificial intelligence to the reduction of regional disparities and provides for the deployment of an updated version of the Information System for Monitoring Healthcare Errors (SIMES) following completion of testing activities [3].
From an infrastructural standpoint, the Ministry’s 2026 strategy is connected to the Electronic Health Record (EHR) 2.0, the Health Data Ecosystem (EDS), and national interoperability initiatives. These developments are aligned with Milestone M6C2-12 of Italy’s National Recovery and Resilience Plan (PNRR), which requires, by June 2026, the operational deployment of the electronic health card, the interoperability infrastructure supporting EHR 2.0, and the implementation of the EDS.
This is not merely a technical issue. Without data quality, completeness, interoperability, and appropriate legal foundations, the safety of artificial intelligence in healthcare cannot be effectively verified. The challenge of data portability between healthcare organizations and across regions illustrates this point clearly. A model validated within one territorial context—for example, Bologna—must be capable of operating reliably within a different setting, such as Catania, if meaningful scalability is to be achieved [14].
Regarding technology assessment, the National Agency for Regional Health Services (AGENAS) launched, in April 2026, the National Health Technology Assessment Programme for Medical Devices (PNHTA-DM) for the 2026–2028 period. Supported by approximately €13 million and following an agreement reached during the State–Regions Conference in December 2025, the programme foresees between 50 and 100 technology assessments during 2026, with particular attention devoted to AI platforms and digital therapeutics.
This initiative reinforces the necessity of subjecting such technologies to systematic, comparative, and impact-oriented evaluation processes. Within its programme dedicated to Medicine and Artificial Intelligence, AGENAS explicitly reiterates that AI does not replace physicians or healthcare professionals but functions as a complementary instrument supporting both clinical and organizational activities [4].
At the European level, the regulatory framework rests upon three principal pillars. The first is the Artificial Intelligence Act (Regulation EU 2024/1689), which entered into force on 1 August 2024 and subjects high-risk AI systems—including medical AI software—to specific requirements concerning risk management, training data quality, transparency, human oversight, and traceability.
The second pillar is the Medical Devices Regulation (MDR, Regulation EU 2017/745), together with the legislation governing in vitro diagnostic devices, which continues to provide the certification framework for healthcare technologies.
The third pillar is the European Health Data Space (EHDS) (Regulation EU 2025/327), which entered into force on 26 March 2025 and is expected to begin its primary implementation phase in March 2029, extending to additional categories by 2031. Together, these instruments delineate a regulatory environment in which algorithmic safety, device certification, and healthcare data governance become progressively integrated [15][16].
The EHDS also has direct implications for clinical risk management. It strengthens patients’ rights regarding access to their data, correction of information, traceability of access events, and restrictions on specific uses of personal data. It enables the secondary use of health information for research through dedicated authorization mechanisms and establishes interoperability and compliance requirements for electronic health record systems.
At the same time, it explicitly prohibits the use of health data for decisions detrimental to individuals or for marketing purposes. If effectively enforced, this prohibition has the potential to redefine the boundaries of consent and prevent forms of insurance selection based upon the improper use of health information.
Nevertheless, a substantial gap remains between formal regulation and effective governance. WHO Europe’s assessment of Member States’ preparedness, conducted across fifty countries between June 2024 and March 2025, revealed that AI adoption is already widespread: 64% of countries within the European Region employ AI-assisted diagnostic systems, while 50% have introduced virtual assistants to support patient pathways.
Yet governance remains underdeveloped. Only four of the fifty countries surveyed possess ethical guidelines specifically dedicated to healthcare AI, and only four have established explicit accountability standards for developers and users. Furthermore, 86% of countries identify legal uncertainty as the principal barrier to the adoption of artificial intelligence technologies [2].
Three Emerging Developments Likely to Redefine the Landscape
Three regulatory developments are currently emerging which, although at different stages of consolidation, appear destined to profoundly reshape the relationship between technological innovation and patient safety. Their significance extends beyond the regulatory perimeter of artificial intelligence in healthcare, influencing healthcare organization, accountability frameworks, verification processes, and the mechanisms through which clinical risk is identified, monitored, and governed. These developments therefore deserve particular attention because they directly affect the architecture of organizational clinical risk management, progressively transforming its assumptions, operational tools, and oversight obligations.
Seven Rules for Avoiding Failure: An Operational Model for Hospitals
Having examined what AI is for, where it makes a measurable difference, and how it reshapes clinical risk, the decisive question must now be addressed: what, in practical terms, must a healthcare organization do to reap the benefits of artificial intelligence without exposing itself to a new generation of systemic errors?
What follows is a seven-step governance framework. The sequence should not be interpreted as an abstract prescription: each healthcare organization may calibrate it according to its own scale, complexity, and operational context. None of the seven steps, however, can be omitted.
Rule One: Inventory First
The first decision is not technological but organizational. Every healthcare organization should establish a comprehensive registry of AI systems already in use or under evaluation, classifying them not according to vendor identity or commercial designation but according to their function and their impact on clinical and organizational processes. Such a registry should include AI-based medical devices, decision-support tools not formally classified as medical devices, generative systems for clinical documentation, triage-support instruments, incident-analysis platforms, and bed-management engines with indirect clinical implications.
In this regard, the report of the JAMA Summit on Artificial Intelligence reminds us that many AI applications possess a hybrid nature and that their healthcare impact does not necessarily correspond to their commercial classification. Consequently, the administrative inventory maintained by procurement departments cannot be regarded as equivalent to the clinical inventory required by risk management functions [6].
Rule Two: Validate Locally Before Deployment
A clinical function demonstrating satisfactory performance in a large university hospital may produce significantly less reliable results in a medium-sized healthcare facility operating within a different organizational environment. Differences in patient demographics, healthcare pathways, and operational conditions alone may substantially affect performance.
For this reason, local validation should include, at a minimum, comparison with current clinical practice, subgroup analyses according to sex, age, ethnicity, and comorbidities, the definition of alert thresholds agreed upon with clinicians, measurement of false-positive and false-negative rates, verification of workflow integration, and explicit identification of conditions requiring system suspension.
Where systems are capable of updating or adapting over time, change management must form an integral part of the initial evaluation process rather than being treated as a secondary or subsequent requirement.
Rule Three: Purchase a Dossier, Not a Slogan
Hospitals should not procure or deploy an AI system without a comprehensive technical and organizational dossier documenting intended use, training context, validation datasets, clinical comparators, interoperability requirements, cybersecurity safeguards, records of verification activities, post-deployment surveillance plans, and update policies.
Both AGENAS and WHO Europe have consistently emphasized the need to align technological classification, evidence standards, and adoption decisions. This requirement becomes even more critical for technologies that promise operational efficiency before demonstrating measurable clinical or organizational benefit.
Rule Four: Distribute Accountability Explicitly
The fundamental principle is clear: clinical decisions remain human, but organizational responsibility throughout the AI lifecycle must be explicitly allocated, documented, and traceable.
This accountability framework should clearly identify the technology provider, the healthcare organization deploying the system, the clinical lead, the data steward, the data protection function, cybersecurity officers, procurement departments, health technology assessment units, risk management teams, and, where appropriate, ethics committees.
The Italian Ministry of Health, European regulatory authorities, data protection bodies, and WHO Europe all converge on the necessity of ensuring transparency, data accuracy, human oversight, and patient protection. In the absence of a clearly defined accountability map, every significant incident risks falling into a grey area in which responsibility becomes uncertain. Such uncertainty is itself a source of clinical risk.
Rule Five: Monitor Performance During Real-World Use
An AI system cannot be evaluated solely at the time of procurement or deployment. Its performance must be continuously monitored under real-world conditions, since effectiveness may deteriorate, drift, or produce unforeseen consequences over time.
Every clinically relevant AI system should therefore be incorporated into an organizational monitoring dashboard based on a common set of indicators, including utilization rates, frequency of recommendation overrides, observed real-world accuracy, variations across departments and patient populations, near misses, adverse events, response times, measurable operational benefits, software updates, internal complaints, and external reports.
In this regard, the OECD explicitly recommends post-deployment measures addressing performance drift, safety monitoring, patient-reported outcomes, and equity of adoption. Similarly, the U.S. Food and Drug Administration has encouraged the development of methodologies capable of identifying and managing changes in system performance over time [8].
A practical proposal for Italian hospitals would consist of monitoring seven core indicators—utilization, override rates, accuracy, equity, performance drift, incidents, and operational benefit—reviewed quarterly by the Clinical Risk Committee.
Rule Six: Train, Train, Train
In a commentary published in 2026, the Italian National Institute of Health emphasized the necessity of investing in continuous education, sound data governance, person-centred bioethics, and the permanent presence of human judgment within decision-making processes.
The message is unequivocal: the adoption of AI in healthcare cannot be reduced to the mere availability of technology. It requires competencies, responsibilities, and adequate organizational safeguards.
Similarly, a systematic review published in JMIR identified transparency, education, usability, and preservation of professional control as decisive factors for safe adoption. WHO Europe had already highlighted, in November 2025, that investment in digital and AI literacy is essential to prevent technological innovation from deepening existing inequalities [17].
One practical proposal would be a mandatory eight-hour Continuing Medical Education (CME) programme entitled Artificial Intelligence at the Bedside, covering validation principles, recommendation override procedures, near-miss reporting, and interpretation of monitoring dashboards, with differentiated pathways for clinicians, nurses, IT professionals, and healthcare executives.
Rule Seven: Integrate AI into the Quality System
Artificial intelligence must become a permanent component of organizational quality and patient-safety systems, according to a logic analogous to that already applied to high-risk medications, critical medical devices, surgical safety programmes, and the prevention of healthcare-associated infections.
This means incorporating AI into reporting systems, clinical audits, departmental meetings, educational programmes, and continuous-improvement initiatives.
The Italian Ministry of Health explicitly links patient safety to SIMES and sentinel-event management; JAMA has highlighted the widespread diffusion of AI systems that often remain insufficiently evaluated; and both the FDA and OECD stress the necessity of continuous monitoring and clearly defined accountability structures.
The true distinction, therefore, is not between innovation and caution, but between governed innovation and innovation left to chance [5].
An Ethical Question, Not Merely a Technical One
The perspective of Ethica Societas requires us not to stop at the technical and operational dimension. The introduction of artificial intelligence into hospitals constitutes, before being a matter of organizational efficiency or clinical innovation, a question concerning the social contract between healthcare institutions, professionals, and citizens.
An AI system capable of identifying patient deterioration at an earlier stage does not merely generate a functional advantage. It effectively extends the horizon of clinical attention, increasing the possibility of detecting risk before it materializes into harm. Conversely, if the same system performs more accurately for certain groups and less reliably for others, it may reproduce or amplify inequalities already present within society, transforming innovation from a protective instrument into a source of vulnerability.
The ethical question, however, arises even before the algorithm itself. It concerns, first and foremost, data: who generates them, who stores them, who reuses them, and for what purposes. It concerns transparency, insofar as patients should be able to understand what can realistically be known about a recommendation generated by a model that may not be fully interpretable. It concerns consent in an era in which health data circulate within shared European infrastructures and generate research value beyond the individual episode of care. Finally, it concerns accountability, particularly when an adverse outcome emerges from the interaction between an automated recommendation, a human decision, and an organization that failed to anticipate the point of failure.
Current regulatory frameworks provide only partial answers to these questions. The European Artificial Intelligence Act imposes requirements concerning human oversight, data quality, and clear information for users. The European Health Data Space strengthens patient control over personal health information and prohibits the use of health data for harmful purposes or commercial profiling. At the same time, European data protection authorities consistently reaffirm the principle that a decision based upon inaccurate data remains inaccurate, regardless of the sophistication of the algorithm employed.
Yet formal regulation alone is insufficient to resolve the underlying cultural challenge. A hospital that introduces artificial intelligence without simultaneously rethinking its educational processes, consent procedures, and internal accountability mechanisms does not necessarily achieve greater safety; rather, it risks introducing a new layer of fragility into the healthcare system.
The ethical issue may ultimately be reduced to a fundamental question: does the introduction of artificial intelligence at the bedside restore time, attention, and anticipatory capacity to the therapeutic relationship, or does it transfer significant portions of clinical and organizational decision-making to an opaque point within the system, where patients struggle to understand who is deciding, on what basis, and subject to which forms of accountability?
The answer does not reside in the algorithm itself. It resides in the architecture of responsibility, transparency, oversight, and verification that each healthcare organization chooses to construct around it.
From Doing to Orchestrating: The Post-Agentic Discontinuity
At this point, it is necessary to address what may be the most significant issue for understanding clinical risk in the period that began during the final quarter of 2025, when society gained access to a new generation of high-quality artificial intelligence capable of supporting non-deterministic deductive cognitive work in ways that are, in some respects, analogous to human reasoning.
To clarify this transformation, it is useful to distinguish between two historical phases, which may be termed the pre-agentic and post-agentic eras.
The pre-agentic period was characterized by a structural contradiction that healthcare quality and patient safety systems carried for decades without fully acknowledging it. Risk management theory assumes that safety derives from countless daily micro-controls involving weak signals of potential deviation from expected standards, process consistency, protocol adherence, and performance anomalies. In practice, however, no human being has ever been capable of performing such activities systematically and continuously.
The reason is not professional inadequacy but anthropological limitation. Monitoring thousands of weak signals against thousands of predefined rules is a quantitatively overwhelming, qualitatively repetitive, and fundamentally inhuman task. Consequently, many controls that risk management frameworks theoretically require have never been implemented in a truly systematic manner.
This reveals a paradox that permeates many quality-management systems: rules, procedures, and standards proliferate, yet no one continuously verifies them in practice—not because of negligence, but because such verification exceeds realistic human capacities.
The post-agentic period introduces a profound discontinuity. The transformation does not merely consist in the arrival of a new technology; rather, it stems from the emergence of a new category of cognitive actor: the agent.
For the first time, healthcare organizations—and other cognitively intensive sectors—can make systematic use of non-deterministic deductive contributions generated by machines. Through these contributions, the infrastructure of monitoring, verification, and control postulated by risk management theory for decades becomes practically achievable rather than merely aspirational.
From this perspective, the role of healthcare professionals gradually changes in nature. The deeper transformation associated with artificial intelligence is the transition from doing to orchestrating.
The traditional model relied upon the manual execution of countless detailed controls. The emerging model requires professionals to establish priorities, assign tasks to the most appropriate agent, verify the quality of execution, and retain ultimate responsibility for decisions.
To orchestrate means, in sequence, to plan, delegate, supervise, and assume responsibility.
Under this framework, healthcare professionals are not diminished by the arrival of artificial intelligence. On the contrary, they are called to a more demanding and intellectually elevated role. They remain the ultimate decision-makers and, precisely for that reason, must become increasingly competent in governing artificial contributions, delegating to machines those aspects of execution that no human being could realistically perform continuously without losing quality, attention, and human judgment.
A methodological clarification is particularly important. The post-agentic perspective should not be interpreted as a functionalist utopia. Rather, it represents the recognition of an organizational reality that manifests itself along two complementary dimensions.
On the one hand, there exist cognitive activities that are quantitatively vast and qualitatively repetitive, activities that it is unreasonable to expect professionals to perform continuously and that, if left unattended, already constitute a source of clinical risk before they become a source of inefficiency.
On the other hand, once the agent enters the organizational system, that organization must equip itself with an explicit architecture of orchestration and accountability. Without such an infrastructure, risk is not truly reduced; it is merely transferred to another point within the decision-making chain.
This is precisely why the seven rules discussed in the previous section are indispensable. Operationally, they describe the organizational infrastructure of orchestration that a post-agentic hospital must build around its artificial intelligence systems.
In other words, the contemporary challenge is to facilitate the adoption of innovative solutions while simultaneously governing the uncertainty that inevitably accompanies them, without falling into the sterile opposition between indiscriminate openness and paralyzing caution.
The solution does not lie in choosing between innovation and regulation. It lies in building a model of clinical governance capable of making human decision-makers more competent in orchestrating cooperation among professionals, data, and artificial agents—returning execution to technology where appropriate and preserving human judgment wherever judgment remains irreducibly human.
Limitations of the Current Framework and Conclusion
The current landscape remains incomplete in at least three respects.
First, prospective multicentre evidence continues to lag behind the speed with which artificial intelligence systems are being adopted, particularly outside the domains of sepsis management, diagnostic imaging, and selected forms of documentation automation.
Second, the distinction between technologies regulated as medical devices and hybrid systems exerting indirect clinical influence remains one of the most sensitive areas with regard to accountability allocation.
Third, publicly available data provide meaningful insight into national levels of preparedness but reveal considerably less about the actual quality of governance within individual healthcare organizations.
These limitations do not justify inaction. Rather, they require methodological prudence, contextual validation, and continuous monitoring during real-world use.
For a journal devoted to the human and social sciences, the conclusion cannot be merely technical.
Artificial intelligence reduces clinical risk when it enables earlier recognition of patient deterioration, supports reconstruction of clinical pathways, makes dispersed safety signals interpretable, and contributes transparently and verifiably to therapeutic decisions and care prioritization. It increases risk when trust is placed in unreliable data, when systems are deployed outside the contexts for which they were validated, when they generate forms of uncritical automation, or when they remain outside the framework of clinical governance.
By 2026, Italy possesses a considerably stronger institutional foundation than it did only a few years earlier, thanks to the strategic direction of the Ministry of Health, the strengthening of the Electronic Health Record 2.0 and the Health Data Ecosystem, the growing attention of AGENAS to health technology assessment, and the integration of the national system into the broader European framework established by the Artificial Intelligence Act and the European Health Data Space.
The decisive challenge, however, remains organizational. Every hospital will be required to develop its own governance architecture capable of integrating procurement, validation, data quality, cybersecurity, privacy protection, distributed accountability, real-world monitoring, and continuous professional education within a post-agentic framework in which the clinician’s role increasingly assumes the form of orchestration rather than mere execution.
Without such an architecture, artificial intelligence cannot yet be regarded as a fully reliable clinical technology. It remains a high-exposure promise, capable of generating value for patients only when a regulated, pluralistic, and accountable human system consciously accepts responsibility for governing it.
Editorial Note
The scientific and clinical considerations presented in this article should be understood solely as an operational framework for clinical governance and do not possess prescriptive value regarding the definition of specific organizational arrangements.
The determination of validation protocols, Diagnostic-Therapeutic Care Pathways (PDTAs), and formal responsibilities remains the exclusive responsibility of individual healthcare organizations and their respective scientific and technical committees, in accordance with applicable clinical guidelines, organizational characteristics, and healthcare priorities.
REFERENCES
References are listed according to their operational relevance. All sources were verified on 16 May 2026.
[1] World Health Organization Regional Office for Europe. (2025, November 19). Is your doctor’s AI safe? Ethical artificial intelligence in healthcare. WHO Europe. https://www.who.int/europe/news/item/19-11-2025-is-your-doctor-s-ai-safe
[2] World Health Organization Regional Office for Europe. (2025). Artificial intelligence is reshaping health systems: State of readiness across the WHO European Region (WHO-EURO-2025-12707-52481-81028). WHO Europe. https://www.who.int/europe/publications/i/item/WHO-EURO-2025-12707-52481-81028
[3] Italian Ministry of Health. (2026). Strategic Directive 2026 (Atto di indirizzo 2026). https://www.salute.gov.it
[4] National Agency for Regional Health Services (AGENAS). (2026). National Health Technology Assessment Programme for Medical Devices 2026–2028 (PNHTA-DM): Launch of the operational phase. https://www.agenas.gov.it
[5] Italian Ministry of Health. (n.d.). Patient safety: Clinical governance and patient safety (SIMES, sentinel events and recommendations). https://www.salute.gov.it/new/it/tema/governo-clinico-e-sicurezza-delle-cure/la-sicurezza-delle-cure
[6] Bresnick, J., Sharfstein, J. M., Perlis, R. H., Aggarwal, R., et al. (2025). AI, health, and health care today and tomorrow. JAMA, 334(15). https://jamanetwork.com/journals/jama/fullarticle/2840175
[7] Bongiorno, C., et al. (2025). Artificial intelligence in clinical decision support and the prediction of adverse events. Frontiers in Digital Health, 7, Article 1403047. https://doi.org/10.3389/fdgth.2025.1403047
[8] Organisation for Economic Co-operation and Development (OECD). (2026). AI in Health Policy Checklist. In Scaling Artificial Intelligence in Health. OECD Publishing. https://www.oecd.org/en/publications/scaling-artificial-intelligence-in-health_a436e12d-en
[9] Lee, S., Kim, J., Park, H., et al. (2025). Deep learning–based early warning systems in hospitalized patients at risk of code blue events and prolonged length of stay. JMIR Medical Informatics, 13, e72232. https://doi.org/10.2196/72232
[10] Patel, A., et al. (2026). Physician-reported safety outcomes of AI-generated hospital course summaries. JAMA Network Open, 9(5). https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2848785
[11] Taylor, R., et al. (2026). Large language models for safety reporting and incident categorisation. BMJ Quality & Safety. Advance online publication. https://doi.org/10.1136/bmjqs-2025-019495
[12] Rodriguez, M., et al. (2025). Using shared clinical decision support to reduce adverse drug events and improve patient safety. Frontiers in Digital Health, 7, Article 1703141. https://doi.org/10.3389/fdgth.2025.1703141
[13] KINBIOTICS Consortium. (2025). An AI-based clinical decision support system for antibiotic therapy in sepsis: Human factors and implementation considerations. JMIR Human Factors, 12, e66699. https://doi.org/10.2196/66699
[14] Italian Ministry of Health. (2026). National Recovery and Resilience Plan (PNRR), Milestone M6C2-12: Electronic Health Card, EHR 2.0 interoperability infrastructure, and Health Data Ecosystem implementation. https://www.pnrr.salute.gov.it
[15] European Commission, Directorate-General for Health and Food Safety. (2025). Artificial intelligence in healthcare. European Commission. https://health.ec.europa.eu/ehealth-digital-health-and-care/artificial-intelligence-healthcare_en
[16] European Parliament & Council of the European Union. (2025). Regulation (EU) 2025/327 on the European Health Data Space (EHDS). Official Journal of the European Union. https://health.ec.europa.eu/ehealth-digital-health-and-care/european-health-data-space-regulation-ehds_en
[17] Istituto Superiore di Sanità. (2026). New demands for digital and AI skills in health occupations. Annali dell’Istituto Superiore di Sanità. https://annali.iss.it/index.php/anna/article/view/1928
[18] General Court of the European Union. (2023). Single Resolution Board v. European Data Protection Supervisor (Case T-557/20). Judgment of 26 April 2023. Luxembourg: Court of Justice of the European Union.
[19] European Commission. (2025, December 16). Draft proposal for the revision of Regulation (EU) 2017/745 on medical devices (MDR 2). Brussels: European Commission Working Draft.
[20] European Medicines Agency, European Commission, and National Competent Authorities. (2025–2026). Emerging regulatory approaches to the qualification of synthetic data for the validation and certification of medical devices and AI-based systems. Regulatory guidance and consultation documents.
OTHER CONTRIBUTIONS BY THE SAME AUTHOR
THE RISK OF RELIGIOUS BIAS IN ARTIFICIAL INTELLIGENCE
WHEN HEALTH IS MEASURED IN CODES
FIVE LATEST CONTRIBUTIONS
DESTROYING IMAGES, GOVERNING MEMORY: JIHADIST ICONOCLASM AND COGNITIVE WARFARE
THE IRREDUCIBLE CITY: JERUSALEM IN THE AGE OF THE GLOBAL CRISIS
DRAGHI IN AACHEN: EUROPE ALONE BEFORE THE TEST OF SOVEREIGNTY
Ethica Societas is a free, non-profit review published by a social cooperative non-profit organization
Copyright Ethica Societas, Human&Social Science Review © 2026 by Ethica Societas UPLI onlus.
ISSN 2785-602X. Licensed under CC BY-NC 4.0