How to Use an Article Evaluating the Clinical Impact of a Computer-Based Clinical Decision Support System
Adrienne G. Randolph, R. Brian Haynes, Jeremy C. Wyatt, Deborah J. Cook, Gordon H. Guyatt, and the Evidence Based Medicine Working Group
Based on the Users Guides to Evidence-based Medicine and reproduced with permission from JAMA. (1999; 282(1):67-74). Copyright 1999, American Medical Association.
- Clinical Scenario
- The Search
- Introduction
- I. Are the results of the study valid?
- II. What are the results?
- III. Can you apply the CDSS in your Clinical Setting?
- Resolution of the Scenario
- References
Clinical Scenario
It is 7 a.m. and medical rounds are starting on University Hospital Ward 3B. In the past 24 hours of your residency, you have transferred two critically ill patients to the intensive care unit, accepted 11 patients to your medical service, examined and revised medication orders on 22 patients, placed 9 intravascular catheters, written 35 notes, and reviewed, categorized and acted upon over 300 new pieces of laboratory and radiology data. You were planning to ask the infectious disease specialist about a patient but he looks really busy and the broad spectrum antibiotic regimen you prescribed should cover everything. You were just told that you ordered total parenteral nutrition for the wrong patient. While deciding which patient the order belongs to, you realize that the calculations for the amino acid concentration are erroneous. Five minutes into your first patient presentation, the senior physician asks you details from the past medical history. You wish you could refer to your admission note but you couldn’t access it before rounds because a utilization review clerk had the chart.
The Chair of Medicine keeps promising to install computers to help manage all of this information but she is feeling the budget squeeze. She needs proof that computerization will improve patient care to justify such a major expense. She asks you to help. You remember reading, in the many journals piled up at home, about how computers can be used to provide decision support leading to improved patient outcomes. If you can show that computers improve patient care, maybe the hospital administration will see the expense as an investment that could reduce costs.
The Search
When you get home that night, you connect to the internet and decide to search the medical literature using Internet Grateful Med from the U.S. National Library of Medicine. You type http://igm.nlm.nih.gov into your browser. You quickly realize that you don’t know what search terms to use. You enter “decision” then click the button for “Find MeSH/Meta Terms”. From the 31 MeSH terms offered you choose “Decision Making, Computer-Assisted”, “Therapy, Computer Assisted”, “Diagnosis, Computer-Assisted”, “Drug Therapy, Computer-Assisted” specifying that they are the major topic of the article. You limit your search to randomized controlled trials in English from the years 1995-1998. Browsing through the 45 abstracts from the search, you choose “A randomized trial of corollary orders to prevent errors of omission”. The abstract of this article concludes that “physician work stations, linked to a comprehensive electronic medical record, can be an efficient means for decreasing errors of omissions and improving adherence to practice guidelines” [1].
You order the full article over the internet from Loansome Doc. In this study [1], conducted on the inpatient general medical wards of an inner-city public hospital, 6 independent services (Red service, Green service, etc.) care for the inpatients. Each service includes a faculty internist, a senior resident, and 2 interns. A different physician team rotates onto each service every 6 weeks, and during a year, 8 different teams work on each service. At the beginning of the study, the investigators randomly allocated 3 of the 6 services to the intervention group which had access to a computer-based clinical decision support systems (CDSS) and the other 3 served as controls which did not. Teams were randomly assigned to the intervention and control services. The CDSS responded to trigger orders by suggesting corollary orders needed to detect or ameliorate adverse reactions, and allowed physicians to accept or reject these suggestions. Examples of corollary orders would be the orders to monitor electrolyte, magnesium and creatinine levels in patients receiving Amphotericin B (the trigger order). Table 1 shows more examples of these corollary orders and their trigger order.
Table 1: Example Trigger and Corollary Orders
|
Clinicians managing the care of patients are dependent upon computers. Laboratory data management software, pharmacy information management systems, applications for tracking patient location through admission and discharge, mechanical ventilators, and oxygen saturation measurement devices are among the many types of computerized systems that have become an integral part of the modern hospital. These devices and systems capture, transform, display, or analyze data for use in clinical decision-making. Using computers to search the medical literature or to improve the legibility, display and accessibility of information in the patient chart may produce benefits that can sometimes be related to the care of an individual patient. However, medical literature databases and ordinary patient charting systems do not filter and abstract information from detailed clinical data. We use the term CDSS to describe software designed to directly aid in clinical decision making about individual patients. Specifically, detailed individual patient data are input into a computer program where they are sorted and matched to programs or algorithms in a computerized knowledge base resulting in the generation of patient-specific assessments or recommendations for clinicians [2]. Table 2 shows categories of decision support systems developed for the following medical purposes: alerting, reminding, critiquing, interpreting, predicting, diagnosing, and suggesting [3].
Table 2. Functions of Computer-Based Clinical Decision Support Systems
|
Many alerting, reminding and critiquing systems are based upon simple “IF - THEN” rules which tell the computer what to do when a certain event occurs. Alerting systems monitor a continuous signal or stream of data and generate a message (an alert) in response to items or patterns that might require action on the part of the care provider [4]. A simple example of an alert is the starred (*) or highlighted items (with H or L marking or with BOLD or changed colors on the screen) that alert the clinician to values that are ‘out of range’ on computerized laboratory printouts and display screens. Alerting systems draw attention to events as they occur. Reminder systems notify clinicians of important tasks that need to be done before an event occurs. An outpatient clinic reminder system may generate a list of immunizations that each patient on the daily schedule requires. Although the rules behind alerts and reminders are often simple, alerting the right person in a timely fashion is quite complex.
When the clinician has made a decision and the computer evaluates that decision and generates an appropriateness rating or alternative suggestion, the decision support approach is called critiquing. The distinction between assisting and critiquing decision support programs is that assisting programs help formulate the clinical decision where critiquing programs have no part in suggesting the order or plan, but evaluate the plan, after it is entered, against an algorithm in the computer [3]. Critiquing systems are commonly applied to physician order entry. For example, a clinician entering an order for a blood transfusion may receive a message stating that the patient’s hemoglobin is above the transfusion threshold and the clinician must justify the order by stating an indication such as active bleeding [5]. Getting the attention of the person who can take action is one of the most difficult aspects of making alerting, reminding and critiquing systems effective.
The automated interpretations of electrocardiogram readings [6] and the outcome predictions generated by severity of illness scoring systems [7] are examples of decision support systems used for interpreting and predicting, respectively. These systems filter and abstract detailed clinical data and generate a report characterizing the meaning of the data such as “anterior myocardial infarction” [6].
Computer aided diagnostic systems assist the clinician with the process of differential diagnosis [8]. When the ECG is not definitive, computer systems that try to distinguish between myocardial infarction and other sources of chest pain can sometimes outperform a clinician [9]. These types of systems require pertinent patient information such as signs, symptoms, past medical history, laboratory values and demographic characteristics. The program starts generating hypotheses, often prompts the user for more information, and ultimately provides a diagnosis or a list of possible diagnoses ranked probabilistically.
Computerized patient management systems are complex programs that make suggestions about the optimal decision based upon the information currently known by the system. These types of systems are often integrated into the physician ordering process. After collecting information on specific patient variables, the assistant program tailors the order to the patient based upon prior information in the database regarding appropriate dosages, or by implementing specified protocols. The Antibiotic Assistant [10] is a CDSS which implements guidelines to assist physicians with ordering antibiotics. This system recommends the most cost-effective antibiotic regimen taking into account: the patient’s renal function, drug allergies, the site of infection, the epidemiology of organisms in patients with this infection at this hospital over many years, the efficacy of the antibiotic regimen, and the cost of therapy. A system which instructs caregivers on how to manage the ventilation of patients with adult respiratory distress syndrome [11] provides another example.
The primary reason to invest in computer support is to improve quality of care. If a computer system purports to aid clinical decisions, enhance patient care and improve outcomes, then it should be subject to the same rules of testing as for any other health care intervention with similar claims. In this article, we describe how to use articles that evaluate the clinical impact of a CDSS. While the focus of a CDSS may be restricted to diagnosis or prognosis, we will limit our discussion to the situation in which the CDSS is designed to change clinician behavior and patient outcome. Many iterative steps are involved in developing, evaluating and improving a CDSS before it can move beyond the laboratory environment and pilot testing phase to be allowed to have a wider impact across physicians and patients. These evaluations involve social science methods for evaluating human behavior and computer science methods for evaluating technological safety and robustness [4]. We limit our discussion to mature systems that have surpassed initial evaluation and are being implemented to change physician behavior and patient outcome.
The User’s Guides to the Medical Literature [12] is a series of articles that describe how a reader can evaluate the information presented in articles addressing different types of clinical problems. In this addition to the series, we follow the outline used in the other articles and discuss: Are the results valid?, What are the results?, and Can you apply the CDSS in your clinical setting? We will continue to refer back to the article by Overhage et. al. [1] evaluating the impact of computerized reminders of corollary orders to prevent errors of omission during the ordering process.
I. Are the results of the study valid?
When clinicians examine the effect of a CDSS on patient management or outcome they should use the same criteria appropriate for any other intervention whether a drug, a rehabilitation program, or an approach to diagnosis or screening [13]. In our Users' Guide to prevention and therapy [14], the importance of random assignment, blinding of patients and outcome assessors, and complete follow-up were explained. The purpose of our discussion in this article is to highlight issues of particular importance in the evaluation of a CDSS.
1. Was the Method of Participant Allocation Appropriate?
The validity of the observational study designs often used to evaluate a CDSS is limited. The most common observational design is the before-after study design in which investigators compare outcomes before a technology is implemented (historic control group) to those after the system is implemented. The validity of this approach is threatened by the risk that changes over time (called “secular trends”) in patient mix, or in aspects of health care delivery, may be responsible for changes in behavior that appear to be attributable to the CDSS. Consider a CDSS assisting physicians with antibiotic ordering [10] that was implemented in the late 1980's and was associated with improvements in the cost-effectiveness of antibiotic ordering over the next 5 years. Changes in the health care system, including the advent of managed care, were occurring simultaneously during that time period. To control for secular trends, the computerized antibiotic practice guideline study investigators [10] compared antibiotic prescribing practices to those of other nonfederal U.S. acute care hospitals for the time period of the study.
One type of time-series design in which the intervention is turned on and off multiple times has been used to control for potential secular trends. Although this provides some protection against bias, random allocation of patients to a concurrent control group remains the strongest study design for evaluating therapeutic or preventive interventions [14]. Use of historical controls may lead to a higher tendency to see positive results. A comparison of the 2 types of studies used to evaluate the same antihypertensive drugs revealed that 80% of historically controlled studies suggested that the new drugs were effective where only 20% of randomized controlled trials confirmed this result [15]. Randomized controlled trials have been successfully used to evaluate over 70 CDSSs [2] [16] [17] [18].
A special issue for CDSS evaluation is the unit of allocation. Usually, investigators in clinical trials randomize patients. When evaluating the effect of a CDSS in patient care, the intervention is usually aimed at changing the decision making of the clinician. So investigators may randomize individual clinicians or clinician clusters such as health care teams, hospital wards or outpatient practices [19]. A common mistake made by investigators is to analyze their data as if they had randomized patients rather than clinicians. This is called the “unit of analysis error” [20].
To highlight the problem, we will use an extreme example. Investigators randomize study participants to ensure that treatment and control groups are balanced with respect to important predictors of outcome. Randomization often fails to balance groups if sample size is small. Consider a study in which an investigator randomizes one team of clinicians to a CDSS, and another to standard practice. During the course of the study, each team sees 10,000 patients. If the investigator analyzes the data as if patients were individually randomized, the sample size appears huge (the “unit of analysis error” [20]). However, it is very plausible, perhaps even likely, that the two teams' performance differed at the start, and this difference persisted through the study independent of the CDSS. Because the base sample size in this study is only 2 (2 teams), the likelihood of imbalance despite randomization is very large.
Obtaining a sample of sufficient size can be difficult when randomizing physicians and health care teams. If only a few health care teams are available, stratification of these teams according to important prognostic factors can reduce potential imbalances on these factors. If there are many known risk factors, investigators can pair health care teams according to their similarities on numerous factors and randomly allocate the intervention within each matched pair [21]. In addition, investigators can use statistical methods developed specifically for analyzing studies using cluster randomization [22].
There is one other issue regarding randomization to which clinicians should attend. Consider: if some clinicians assigned to CDSS fail to receive the intervention, should these clinicians be included in the analysis? The answer, counterintuitive to some, is “Yes”. Randomization can accomplish the goal of balancing groups with respect to both known and unknown determinants of outcome only if patients (or clinicians) are analyzed in the groups to which they are randomized. Deleting or moving patients after randomization compromises or destroys the balance randomization is designed to achieve. The technical term for an analysis in which patients are included in the groups to which they were randomized, whether or not they received the intervention, is “intention to treat” [14].
In the study by Overhage et al. [1], over the course of a year there were 36 teams randomly assigned to 18 CDSS and 18 control services. Housestaff were required to write all orders and were used as the unit of analysis. Each service admitted patients in sequence so that all 6 services received equal numbers of patients. A total of 86 housestaff physicians who received more than 5 corollary orders during the study cared for 2,181 different patients during 2,955 different admissions.
Random assignment of teams to CDSS and non-CDSS services increases our belief that the results are valid. However, although investigators did not randomly assign housestaff to services, they conducted their analysis at the individual housestaff level comparing 45 intervention physicians and 41 control physicians. They took no steps to ensure that the characteristics of housestaff on the intervention and control teams were similar, leaving the study open to biases from baseline or intrinsic differences in housestaff performance. Moreover, the use of individual housestaff instead of the team as the unit of analysis may have lead to false precision in estimating the impact of the intervention (as described above, a falsely inflated sample size).
In this study [1], the investigators excluded 6 physicians from the intervention group because they received fewer than 5 suggestions about corollary orders. This decision violates the intention to treat principle, and risks introducing bias (after all, similar physicians on the control side would be included). Fortunately, the small number of excluded physicians were mostly off-service physicians covering night calls for 1 or 2 nights and not actually service team members, so the contribution of such physicians to the comparison of CDSS and control is small.
2. Was the Control Group Uninfluenced by the CDSS?
One problem with performing a controlled trial randomizing a CDSS across patients is the difficulty in controlling for contamination of the control group by the intervention. Strickland et al. [23] randomly allocated patients to have changes in their level of mechanical ventilator support directed by a computer protocol and implemented through a physician or directed by the physician independently. Because the same physicians and respiratory therapists using the computer protocol were managing the care of patients not assigned to the protocol, it is possible clinicians could remember and apply protocol algorithms in control patients. When the control group is influenced by the intervention, the effect of the CDSS may be diluted. Contamination may spuriously decrease, or even eliminate, a true intervention effect.
One method of preventing exposure of the control group to the CDSS is to assign individual clinicians to use or not use the CDSS. This is often problematic because of cross-coverage of patients. Comparing the performance of wards or hospitals that do or do not use the CDSS is another possibility. Unfortunately, it is usually not feasible to enroll a sufficient number of hospitals to avoid the problem that we described earlier: when sample size is small, randomization may fail to ensure prognostically similar groups.
In the Overhage study [1], physicians whose team was assigned to a control service had the CDSS guidelines available on paper but did not receive assistance when ordering. To control for the risk that cross-coverage of patients could expose the control group to the CDSS, the investigators had the Chief Medical Resident construct the residents' evening call schedule to separate coverage for patients based upon their study status. If switches in the schedule were made, control physicians provided call coverage only for non-CDSS patients and intervention physicians covered only CDSS patients. Further, to avoid contamination that could occur if intervention physicians cared for control patients, the computer suggested orders only when the patient had been assigned to a physician in the CDSS group and corollary order suggestions were suppressed if the patient was assigned to the control group. If physicians returned for a second rotation and changed study status, the investigators excluded data from their second rotation. All of these efforts were to prevent contamination of the control group by the CDSS.
3. Aside from the CDSS, Were the Groups Treated Equally?
The results of studies evaluating interventions aimed at therapy or prevention are more believable if patients, their caregivers, and study personnel are blind to the treatment [14]. Unblinded study personnel who are measuring outcomes may provide different interpretations of marginal findings or differential encouragement during performance tests [24]. Blinding also diminishes the placebo effect [14], which in the case of CDSS may be the tendency of patients to ascribe positive attributes to use of a computer workstation [4]. Although blinding the clinicians, patients and study personnel to the presence of the computer-based CDSS may prevent this type of bias, blinding is sometimes not possible.
Interventions other than the treatment under study that can influence the outcome of the study are called "co-interventions". Co-interventions frequently exist because most patients receive multiple therapies aimed at improving their outcome. A problem arises when co-interventions are differentially applied to the treatment and control groups. This situation is more likely to arise in unblinded studies, particularly if the use of very effective non-study treatments is permitted at the physicians' discretion [14]. Clinicians’ concerns regarding lack of blinding are ameliorated if investigators describe permissible co-interventions and their differential use and/or standardize co-interventions [25] to ensure that their application was similar in both treatment and control groups.
It is also important to ensure that the evaluation of the outcome for each group is not biased. In some studies the computer system may be used as a data collection tool to evaluate the outcome in the CDSS group. The “data completeness bias” can occur when the information system is used to log episodes in the treatment group and a manual system is used to log episodes in the non-CDSS group [4]. Because the computer may log more episodes than the manual system, it may appear that the CDSS group had more events which could bias the outcome in favour of or against the CDSS group. To prevent this bias, outcomes should be logged similarly in both groups.
In the study by Overhage et al [1] although faculty were proscribed from writing orders except during emergencies, physicians practice within teams and the faculty influenced the residents through their teaching. Further complicating this situation, faculty could rotate with different housestaff on different rotations during the study. To allow for this clustering of physicians within teams, the investigators used a statistical method (generalized estimating equations) to control for this potential co-intervention.
II. What are the Results?
1. What Is the Effect of the CDSS?
A CDSS is an intervention often aimed at preventing adverse events or health outcomes or at improving compliance with a treatment regimen. Therefore, we refer you to the discussion in our Users’ Guide for prevention or therapy [14] for a discussion of relative risk and relative risk reductions, risk differences and absolute risk reductions, and confidence intervals. In the Overhage study [1], intervention physicians ordered the corollary orders suggested by the CDSS much more frequently than control physicians spontaneously ordered them. This was true when measured by immediate compliance (46.3% versus 21.9%, relative increase 2.11, p < 0.0001), 24-hour compliance (50.4% versus 29.0%, relative increase 1.74, p < 0.0001), or hospital stay compliance (55.9% versus 37.1%, relative increase 1.51, p < 0.0001). Because the numerators and denominators are not reported for the total numbers of corollary orders complied with and not complied with for each group, we cannot calculate the confidence intervals around the risk difference for the increase in compliance. However, because the p-values are very small, we know that the lower boundary of the confidence interval is a long way from 1, and the confidence interval is therefore relatively narrow.
Length of stay and hospital charges did not differ significantly. Pharmacists made 105 interventions with the CDSS group of physicians and 156 with control physicians (two tailed p = 0.003) for errors considered to be life threatening, severe or significant.
III. Can you apply the CDSS in your Clinical Setting?
1. What Elements of the CDSS are Required?
It is important to specify what intervention is being evaluated. Two of the major elements comprising a CDSS, the logic and the computer interface used to present the logic, could each be evaluated as a separate intervention. However, sometimes it is not possible to separate these 2 elements and achieve the same impact. For example, we mentioned a randomized controlled trial comparing a computerized protocol for managing patients with Adult Respiratory Distress Syndrome which was compared to standard clinical care with Extracorporeal CO2 removal used as rescue therapy [11]. The computerized protocol group without rescue therapy did as well as the rescue therapy group. Was this due to the logic in the protocol, the use of the computer, or both interacting together? To test whether the computer is needed requires that one group apply the protocol logic as written on paper and the other group use the same logic implemented in the computer. Sometimes the logic is so complex that use of a computer may be required for implementation.
The CDSS may have a positive impact for unintended reasons. The impact of structured data collection forms and performance evaluations (respectively called the Checklist Effect and the Feedback Effect [4]) on decision making can equal that of computer-generated advice [26]. The CDSS intervention itself may be administered by research personnel or paid clinical staff receiving scant mention in the published report but without whom the impact of the system is seriously undermined.
The CDSS in the Overhage study of corollary orders [1] and in the ARDS study [11] had 3 components, 1.) a knowledge base defining which corollary orders were required for each trigger order, 2.) a database that stored the trigger orders, and 3.) an inference engine that compared the database to the knowledge base when a trigger order was received and sent a list of suggested corollary orders to the computer terminal for display.
2. Is the CDSS Exportable to a New Site?
For a CDSS to be exported to a new site, it has to be able to integrate with existing software, users at the new site must be able to maintain the system, and users must accept the system. Double-charting occurs when systems require staff (usually nurses) to enter the data once into the computer and again on a flow sheet. Systems that require double-charting increase staff time devoted to documentation, frustrate users, and divert time that could be devoted to patient care. In general, such systems fail in clinical use.
Therefore, it is important to assess how the information necessary to run the decision support gets into the system. In general, successful systems are ones with automatic electronic interfaces to existing data producing systems. Unfortunately, building interfaces to diverse computer systems is often extremely challenging and sometimes impossible.
The program described in the Overhage study [1] was implemented using the Regenstrief Medical Record System developed at Indiana University School of Medicine. This system provides an electronic medical record system and a physician order entry system. While it may be possible to take the knowledge built into the system and use it in a care environment where the patient population is similar, the inference engine used to compare the rules against the order entered into the database is not easily exported to other locations. If, after critically appraising the article you are convinced that a CDSS for implementing guidelines would be useful, you would need sufficient resources to rebuild the system at your own site.
3. Is the CDSS Likely to be Accepted by Clinicians in Your Setting?
One reason a CDSS may not be accepted is if the clinicians differ in important ways from those who participated in the study. The choice of evaluative group may limit the generalizability of the conclusions if recruitment is based upon enthusiasm, demographics or a zest for new technology. Clinicians in a new setting may be surprised when their colleagues do not use a CDSS with the same avidity as the original participants.
The user interface is an important component of the effectiveness of a CDSS. The CDSS interface should be developed on the basis of potential users’ capabilities and limitations, the users’ tasks, and the environment in which those tasks are performed [27]. One of the main difficulties with alerting systems is getting the information that there is an abnormal laboratory value, or other potential problem, as rapidly as possible to the individual with decision making capability. A group of investigators tried a number of different alerting methods, from a highlighted icon on the computer screen to a flashing yellow light placed on the top of the computer [28]. These investigators later gave the nurses pagers to alert them about abnormal laboratory values [29]. The nurses could then decide how to act upon the information and when to alert the physician.
To ensure user acceptance, users must feel that they can count on the system to be available whenever they need it. The amount of down-time needed for data back-up, troubleshooting, and upgrading should be minimal. The response time must be fast, data integrity must be maintained and data redundancy minimized. If systems have been functioning at other sites for a period of time, major problems or software bugs may have been eradicated, decreasing down time and improving acceptance. It is also important to assess the amount of training required for users to feel comfortable with the system. If users become frustrated with the system, system performance will be suboptimal.
Many computer programs may function well at the site where the program was developed. Unfortunately, the staff at your own institution may have objections to the approaches taken elsewhere. For example, an expert system for managing ventilator patients who have adult respiratory distress system may use continuous positive airway pressure trials to wean patients off of the ventilator, while clinicians at your institution may prefer pressure support weaning. Syntax, laboratory coding and phrasing of diagnoses and therapeutic interventions can vary markedly across institutions. Customizing the application to the environment may not be feasible and additional expense may be invoked when mapping vocabulary to synonyms unless a mechanism to do so is already programmed in. To ensure user acceptance, the needs and concerns of users should be considered and users should be included in decision making and implementation stages.
The logic in the Regenstrief Order Entry system [1] was mainly based upon the expertise of a hospital committee of staff physicians and pharmacists. Although reference texts were used, the degree to which the investigators applied an evidence-based approach is not clear. Use of solid evidence [30] from the literature could enhance clinician acceptance by convincing physicians that the rules positively impact patient outcomes. However, gaining consensus even with “evidence-based” practices can be difficult and a method for gaining consensus must be integrated into the local processes and “culture” of care. Further, physicians will need some time to become acquainted with any new system, especially an order entry system.
When the Overhage study began, all physicians on the medical wards had been entering all inpatient orders directly into physician work-stations for 12 months. Because the order entry program was developed over time and refined by user input, it was tailored to the needs of the clinicians at that hospital. Whether this system would be easily accepted in a new environment by clinicians who had nothing to do with its development is open to question.
4. Do the Benefits of the CDSS Justify the Risks and Costs?
Does the report reveal the “behind the scenes” costs? The real cost of the CDSS is usually much higher than the initial hardware, software, interface, training, maintenance fees, and upgrade costs (which may not be in the report). Often the CDSS is designed and maintained by staff whose actions are critical to the success of the intervention. Your institution might not want to pay for the time of such people in addition to the cost of the computer software and hardware. Indeed, it can be very difficult to estimate the costs of purchasing or building and implementing an integrated CDSS.
Are CDSSs beneficial? Human performance may improve when participants are aware that their behavior is being observed (the Hawthorne Effect [31]). The same behavior may not be exhibited when the monitoring of outcomes has stopped. Taking into account the influence of a study environment, a published systematic review of studies assessing CDSSs used in inpatient and outpatient clinical settings by health care providers [2] that was recently updated [18] showed that the majority of CDSSs studied were beneficial. The outcomes assessed were patient related outcomes (e.g. mortality, length of stay, decrease in infections) or health-care-process measures (e.g. compliance with reminders or with evidence-based processes-of-care). A total of 68 prospective trials using a concurrent control group have reported the effects of using CDSSs on drug dosing, diagnosis, preventive care, and active medical care. Sixty-six percent of studies (43/65) showed that CDSSs improved physician performance. These included 9/15 studies on drug dosing systems, 1/5 studies on diagnostic aids, 14/19 preventive care systems, and 19/26 studies evaluating CDSSs for active medical care. Forty-three percent of studies (6/14) showed that CDSSs improved patient outcomes, 3 studies showed no benefit, and the remaining studies lacked sufficient power to detect a clinically important effect.
Health care processesare more often evaluated than patient health outcomes because process events occur more frequently than major health outcomes. For example, a trial designed to show a 25% improvement (from 50% to 62.5%) in the proportion of patients who are compliant with a certain medication regimen would need to enroll 246 patients per group. A trial designed to show that this medication reduces mortality by 25% (from 5% to 3.75%) would need to enroll 4,177 patients per group. Furthermore, long follow-up periods are required to show that preventive interventions improve patient health outcomes.
Fortunately, evaluation of health-care-processes will be adequate to infer benefit if the care processes being monitored are already known to improve outcomes [32]. We could conclude that a CDSS that increased the frequency with which aspirin, beta blockers and ACE inhibitors were administered to appropriate patients after myocardial infarction was beneficial. The reason is that large, well-designed randomized trials have demonstrated the benefit of these three interventions. Unfortunately, the link between processes and outcomes is often unknown or weak.
The study by Overhage et al [1] was able to demonstrate that physician work stations, when linked to an order entry system able to run a series of rules, was an efficient means for decreasing errors of omissions and improving adherence to practice guidelines. It is unclear how many of the rules in the system were based upon solid evidence, and thus how likely compliance with rules is to improve outcomes. Therefore, it is unclear whether the benefits are worth the cost of purchasing, configuring, installing and maintaining the CDSS.
Resolution of the Scenario
A computer-based CDSS evaluation involves the interplay between three complex elements: 1. one or more human intermediaries, 2. an integrated computerized system and its interface and 3. the knowledge in the decision support. This makes evaluation of a computer-based CDSS a complex undertaking. Systematic reviews [33] of the impact of a CDSS on provider behavior and patient outcome have shown evidence of benefit [2] [16] [17] [18]. Because the evaluation process was not standard, it is difficult to compare the results of these reviews.
In this article, we described a process of evaluating articles that aim to measure the impact of a computer-based CDSS on provider decisions or patient outcomes. Despite the complexity in evaluation, clinicians can use basic principles of evidence-based care to evaluate a CDSS. A study evaluating a CDSS is more believable if there is a concurrent control group with random allocation of subjects. Randomization of clinicians by clusters can prevent the cross-contamination of the control group by the intervention that could mask the effect of the CDSS. When using multi-level designs (the physician or physician group and their respective patients) it is important to consider the physician or group to be the unit of analysis and not the patients. Because most studies evaluating a CDSS are not blinded, the importance of controlling for co-interventions that could bias the outcome was stressed.
Even if the study is valid, and a positive effect is shown, CDSSs have special applicability issues that must be considered. Is the computer essential to deployment of the knowledge in the CDSS? Can the CDSS be exported to a new site? Will clinicians at your site accept the CDSS? And finally, is it possible to accurately evaluate the cost of the CDSS when assessing risks and benefits.
References
1. Overhage J, Tierney W, Zhou X-H, McDonald C. A randomized trial of "corollary orders" to prevent errors of omission. JAMIA. 1997;4:364-375.
2. Johnston M, Langton K, Haynes R, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome: a critical appraisal of research. Ann Intern Med. 1994;120:135-142.
3. Pryor A. Development of decision support systems. Int J Clin Monit Comput. 1990;7:137-146.
4. Friedman CP, Wyatt JC. Evaluation Methods in Medical Informatics. Springer-Verlag, 1997, New York, New York.
5. Lepage E, Gardner R, Laub R, Golubjatnikov O. Improving blood transfusion practice: role of a computerized hospital information system. Transfusion. 1992;32:253-259.
6. Weinfurt PT. Electrocardiographic monitoring: an overview. J Clin Monit. 1990; 6: 132-138.
7. Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, et al. The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest. 1991;100:1619-1636.
8. Berner E, Webster G, Shugerman A, et al. Performance of four computer-based diagnostic systems. N Engl J Med. 1994;330:1792-1796.
9. Kennedy R, Harrison R, Burton A, et al. An artificial neural network system for diagnosis of acute myocardial infarction (AMI) in the accident and emergency department: evaluation and comparison with serum myoglobin measurements. Comput Meth Prog Biomed. 1997;52:93-103.
10. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med. 1998; 38:232-238.
11. Morris AH, Wallace CJ, Menlove RL, Clemmer TP, et al. Randomized clinical trial of pressure-controlled inverse ratio ventilation and extracorporeal CO2 removal for adult respiratory distress syndrome. Am J Respir Crit Care Med. 1994;149:295-305.
12. Oxman A, Sackett D, Guyatt G, for the Evidence-Based Medicine Working Group. Users' guides to the medical literature: how to get started. JAMA. 1993;270:2093-2095.
13. Spiegelhalter DJ. Evaluation of clinical decision-aids, with an application to a system for dyspepsia. Stat Med. 1983;2:207-216.
14. Guyatt G, Sackett D, Cook D, for the Evidence-Based Medicine Working Group. Users' guides to the medical literature, II: how to use an article about therapy or prevention, A: are the results of the study valid? JAMA. 1993;270:2598-2601.
15. Sacks H, Chalmers TC, Smith H. Randomized vs. historical controls for clinical trials. Am J Med. 1982;72:233-240.
16. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. JAMIA. 1996;3:399-409.
17. Balas E, Austin S, Mitchell J, Ewigman B, Bopp K, Brown G. The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch Fam Med. 1996;5:271-278.
18. Hunt D, Haynes R, Hanna S. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280:1339-1346.
19. Cornfield J. Randomization by group: a formal analysis. Am J Epidemiol. 1978:108;100-102.
20. Whiting-O'Keefe Q, Henke C, Simborg D. Choosing the correct unit of analysis in medical care experiments. Medical Care. 1984;22:1101-1114.
21. Klar N, Donner A. The merits of matching in community intervention trials: a cautionary tale. Stat Med 1997;16:1753-1764.
22. Thompson SG, Pyke SD, Hardy RJ. The design and analysis of paired cluster randomised trials: an application of meta-analysis techniques. Stat Med 1997;16:2063-2079.
23. Strickland JH, Hasson JH. A computer-controlled ventilator weaning system. Chest. 1993;103:1220-1226.
24. Guyatt GH, Pugsley SO, Sullivan MJ, et. al. Effect of encouragement on walking test performance. Thorax. 1984;39:818-822.
25. Morris A, East T, Wallace C, et al. Standardization of clinical decision making for the conduct of credible clinical research in complicated medical environments. Proceedings/ AMIA Annual Fall Symposium. 1996:418-422.
26. Adams ID, Chan M, Clifford PC, et al. Computer aided diagnosis of acute abdominal pain: a multicentre study. BMJ. 1986;293:800-804.
27. Salvemini AV. Improving the human-computer interface: a human factors engineering approach. MD Comput. 1998;15:311-315.
28. Bradshaw K, Gardner R, Pryor T. Development of a computerized laboratory alerting system. Comput Biomed Res. 1989;22:575-587.
29. Tate K, Gardner R, Scherting K. Nurses, pagers, and patient-specific criteria: three keys to improved critical value reporting . Annual Symposium on Computer Applications in Medical Care . Washington D.C.: American Medical Informatics Association; 1995:164-168.
30. Guyatt G, Sackett D, Sinclair J, et al. Users' guides to the medical literature. IX. A method for grading health care recommendations. JAMA. 1995;274:1800-1804.
31. Roethligsburger FJ, Dickson WJ: Management and the Worker. Cambridge, MA: Harvard University Press, 1939.
32. Mant J, Hicks N. Detecting differences in quality of care: the sensitivity of measures of process and outcome in treating acute myocardial infarction. BMJ. 1995;23:793-796.
33. Oxman A, Cook D, Guyatt G, for the Evidence-Based Medicine Working Group. Users' guides to the medical literature. VI. How to use an overview. JAMA. 1994;272:1367-1371.
© 2001 Evidence-Based Medicine Informatics Project
