Development and evaluation of a clinical decision support system for the reduction of medication errors

Medication errors are preventable events that can lead to inappropriate drug use and patient harm. Clinical decision support systems (CDSSs) integrated into electronic health records (EHR) have the potential to significantly reduce medication errors. A major problem with CDSSs is alert fatigue, where clinicians ignore or override frequent alerts. Improving the specificity and relevance of CDSSs is critical to reduce alert fatigue. The Cantonal hospital of Aarau AG (KSA) has developed and implemented a CDSS (KPharm) with algorithms that focus on specificity by taking patient relevant data and medication chronology into account. To manage alerts more effectively, some are sent directly to prescribers, while others are initially assessed by clinical pharmacists.
The aim of the thesis was to scientifically guide the evaluation of existing algorithms and the development of new algorithms. The overarching question was whether the algorithms meet the requirements high specificity and other favorable performance parameters in practice, and how algorithms need to be designed to be specific.
Part 1: Evaluation of existing algorithms
In the first study, the performance of the CDSS was evaluated in terms of acceptance rate and alert burden. All alerts generated by the CDSS between January and December 2021 were included in a retrospective quantitative evaluation. Of 10,556 alerts generated, 619 triggered a direct notification to the physician and 2,231 notifications were send to the physician after evaluation by a clinical pharmacist. The acceptance rate was 89.8% for direct alerts and 68.4% for alerts pre-assessed by clinical pharmacists, which resulted in an overall acceptance rate of 72.4%. Clinical pharmacists handled an average of 17.2 alerts daily, while all hospital physicians combined received an average of 7.8 notifications daily. Moreover, a web-based survey was conducted amongst physicians of our hospital to assess their satisfaction with the CDSS. In the survey, 94.5% of physicians reported being satisfied or very satisfied. Algorithms addressing potential medication errors of anticoagulants received the highest usefulness ratings.
The second study investigated the performance of a specific algorithm in more detail, namely the triple whammy algorithm. The term triple whammy refers to the concomitant use of non-steroidal anti-inflammatory drugs, diuretics, and angiotensin system inhibitors; this combination significantly increases the risk of acute kidney injury. We identified all patients at the KSA who received a triple whammy prescription, had a triple whammy alert, or developed acute kidney injury during triple whammy therapy over the course of a year. Algorithm performance was evaluated by calculating the sensitivity and specificity as a primary endpoint and determining the acceptance rate among clinical pharmacists and physicians as a secondary endpoint. Out of 21,332 hospitalized patients, 290 patients had a triple whammy prescription, of which 12 patients experienced acute kidney injury. Overall, 216 patients were flagged by the algorithm, including 11 of 12 patients with acute kidney injury. The algorithm had a sensitivity of 88.3% and a specificity of 99.7%. Physician acceptance was high (77.7%), but clinical pharmacists were reluctant to forward the alerts to prescribers in some cases.
The third study focused on an algorithm to prevent anticoagulant duplications. This algorithm was developed in two phases, starting with an external algorithm that was run only once a day on weekdays and integrated into the hospital's EHR in the second phase. A retrospective analysis of three phases, one without CDSS and the two implementation phases, was performed to assess the impact of the CDSSs on the incidence and duration of anticoagulant duplications. The incidence of anticoagulant duplications in patients receiving two or more anticoagulants during the study period dropped from 91 anticoagulant duplications in 1581 patients in phase I, to 70 anticoagulant duplications in 1692 patients in phase II and 57 anticoagulant duplications in 1575 patients in phase III. The durations of anticoagulant duplications were also reduced over time, with anticoagulant duplications lasting a mean of 1.8, 1.4, and 1.1 calendar days during phases I, II, and III, respectively. Compared to the baseline in phase I, there was a 42% relative risk reduction for anticoagulant duplications in Phase III (relative risk (RR): 0.58, confidence interval (CI): 0.42 – 0.81). Phase III was analyzed in more detail regarding the acceptance rate, sensitivity, and specificity of the alerts. The acceptance rate was high at 97%; and the sensitivity and specificity of the algorithm was 87.4% and 87.9%, respectively.
The fourth study compared the performance of the final anticoagulant duplication algorithm described in study 3 to the performance of a CDSS with a different design in another cantonal hospital in Switzerland. While the CDSS at the KSA uses delayed expert alerts, the CDSS at the other hospital uses interruptive pop-up alerts during prescribing. To compare the performances of both CDSSs, we repeated the retrospective analysis in the other hospital one year before and after the implementation of its CDSS. The incidence of anticoagulant duplications was decreased in both hospitals after CDSS implementation, however, to a different extent. While the KSA had a relative risk reduction of 0.42 (RR = 0.58, CI = 0.42, 0.81), the relative risk reduction at the other hospital was less profound at 0.28 (RR = 0.72, CI = 0.52, 0.99). Moreover, the median duration of anticoagulant duplications was significantly reduced at the KSA, but not in the other hospital.
Part 2: Development of new algorithms
Three new algorithms were developed as part of this thesis. During the development phase, care was taken to ensure that not only the medical and technical aspects of the algorithms were adequately considered, but also the process level.
The Drug Duplication algorithm detects duplicate prescription both for same drugs and for therapeutic drug duplications. For this purpose, the principle of conventional CDSSs for recognizing drug duplications, which are often based on Anatomical Therapeutic Chemical codes, was expanded and refined. The algorithm has been in operation since May 2022 and has been well received by clinical pharmacists. In the first year, 366 interventions in patient medication were made on the basis of this algorithm, with an implementation rate of 75.7% by physicians. For certain drugs and in some departments, we observed a high false positive rate, which can be attributed to procedural peculiarities.
The HIT algorithm is used to recognize risk situations for heparin-induced thrombocytopenia. It recognizes patients who have a drop in platelets in temporal relation to heparin, patients who have received argatroban in the past (as indicator for previous HIT), and patients whose platelet levels are not being adequately monitored. The technical challenge with this algorithm was to correctly capture the chronology of HIT. The algorithm was integrated into the EHR in July 2023. A comprehensive evaluation of its performance has not yet been conducted. Given that the detection of an adverse drug reaction is a more challenging task than that of a medication error, this algorithm will likely exhibit a lower level of specificity than other KPharm algorithms.
The phenprocoumon algorithm was developed in response to a critical incident case, where a negative outcome was observed. The underlying problem was the declining use and consequently the decreasing medical experience with phenprocoumon. The objective of the algorithm is to compensate for this by improving the dosing and monitoring of phenprocoumon therapy in critical situations. The algorithm was integrated into the EHR in March 2024 and has already identified patients who have benefited from a pharmaceutical medication reconciliation.
Overall, the KPharm CDSS enhances patient safety at the KSA by demonstrating satisfactory performance across key parameters such as acceptance rates, sensitivity, specificity, and physician satisfaction. Furthermore, the introduction of the CDSS was accompanied by a risk reduction of critical medication errors. This confirms that KPharm meets its intended goals. The analyses also offer insights into the impact of different CDSS designs on CDSS effectiveness.
The development of new algorithms has enabled the detection and remediation of further potentially critical medication errors. A comprehensive evaluation of these algorithms regarding performance is yet to be conducted. The key findings from the development of context- and time-dependent algorithms, which emphasize the importance of considering medical, technical, and process-level factors, can contribute to the development and improvement of other complex CDSSs in different settings. Ultimately, the implementation of carefully designed context-based algorithms, tailored to hospital workflows, will help to reduce alert fatigue experienced by healthcare professionals when using CDSSs, thus increasing the effectiveness of CDSSs in preventing medication errors.