Centre for Health Evidence: Home » Users' Guides to EBP
|
Centre for Health Evidence: Home » Users' Guides to EBP |
C. David Naylor, Gordon H. Guyatt, for the Evidence-Based Medicine Working Group
Based on the Users Guides to Evidence-based Medicine and reproduced with permission from JAMA. (1996;275(18):1435-9). Copyright 1996, American Medical Association.
You are a general internist and newly-appointed chief of staff at a large community hospital. At a meeting of the Medical Advisory Committee, a senior administrator announces that the hospital's utilization profile for percutaneous transluminal coronary angioplasty (PTCA) diverges markedly from two other hospitals in the region, and alleges unnecessary PTCAs are being performed. He provides raw numbers showing that the number of PTCAs per annum is high relative to similar-sized centres with similar numbers of invasive cardiologists. Clinicians present are infuriated by the allegations, and the meeting quickly degenerates into a shouting match. After the hospital CEO brings the meeting back to order, you volunteer to research the matter and report back in one week.
Each hospital has its own special catchment area and referral patterns. Thus, raw utilization data are insufficient to assess whether cardiologists at your hospital are using PTCA inappropriately. You decide to review their practice in light of criteria for deciding whether each application of PTCA was likely, given a balance of risks and benefits, to be in the patient's best interest. Using Medline on CD-ROM, you search from 1990 to June 1994. As a MeSH Subject Heading, "Angioplasty, Transluminal, Percutaneous Coronary" yields 2203 citations even after the search is limited to "human" and "english language". You then try "Guideline" or "practice guideline" as key words. The guideline references look useful for informing a practitioner's decisions, but you cannot readily see how to translate them into criteria for auditing individual charts.
Finally you try PTCA with "utilization review" as a MeSH heading, and three references turn up. One article looks perfect. This study by researchers with the RAND Corporation, used explicit criteria to assess the appropriateness of use of PTCA in New York State, drawing on randomly selected records for 1306 patients from 15 randomly-selected hospitals [1]. Fifty-eight percent of PTCAs were rated appropriate; 38%, uncertain; and 4%, inappropriate. The inappropriate rate varied by hospital from 1% to 9% (P = .12), while the uncertain rate ranged from 26% to 50% (P = .02). Judging from this article, your hospital would have a defensible profile if its rate of apparently inappropriate PTCA were under 10%. But are the criteria developed by the RAND investigators reasonable, or easily applied?
Physicians' practices may change in response to original reports on randomized trials of therapeutic interventions or new diagnostic tests. But with so many studies published each month, it is often more efficient to rely on integrative reports, such as meta-analyses, decision analyses, and economic analyses that synthesize evidence from multiple primary studies. Practice guidelines - discussed in an earlier Users' Guide [2] - represent a further level of digestion and integration. The guideline may synthesize not only research evidence, but also expert opinion and judgement to help define what a practitioner ought to do when confronted with a particular clinical situation.
For various reasons, we do not always do what the best evidence or a sound guideline suggests we ought to do. Cost containment pressures have also lead to greater concerns about both the effectiveness and efficiency of physicians' practices. Thus, there is a growing cadre of researchers, insurers, administrators, and policy-makers concerned to monitor and document the quality of physicians' work.
From a measurement perspective, the focus of the review can be on outcomes (to be covered in the next Users' Guide) or on assessing processes of care -- the topic of this article. In examining the quality of clinical processes, researchers and managers basically seek to determine whether the right service is provided to the right type of patient for the right reasons at the right time. This can be done by implicit reviews, relying on the individualized judgements of expert clinicians. Practitioners then have the comfort of knowing that their work is being appraised by someone who understands the clinical world and its exigencies. Unfortunately, lack of standardization renders implicit reviews unreliable [3] [4].
Explicit criteria, which form the basis for most process-of-care analyses in the literature, have the advantages of standardization and consistency, as well as transparency. Where necessary, trained staff can apply them retrospectively to medical records without a major time commitment from clinicians. Since explicit reviews are only as good as the criteria used to guide them, the practice community can and should scrutinize both the evidence synthesized and the methods of synthesis. However, even if the criteria themselves pass critical appraisal, they may be poorly or arbitrarily applied. Last, you may also want to determine if the criteria are applicable and practical for assessing your own practice setting.
This Readers' Guide will accordingly assist you in either one of two related goals: to critique a paper purporting to measure the quality of the process of care delivered in a particular setting, and to decide whether, in conducting your own utilization review, you should emulate the methods used in a published study.
In the following discussion, we shall use the American term, "utilization review" and the British term, "clinical audit" interchangeably to describe this type of process-of-care assessment. We shall refer to "panelists" as members of the group of clinical experts that establishes the explicit review criteria in the first place, and to "auditors" as those who review patient charts or interview patients and/or physicians to obtain the key clinical details pertinent to the processes-of-care being scrutinized.
Because we may be interested both in critiquing published results and in using the criteria to assess quality in our own setting, we have modified the basic questions used in earlier Users' Guides to consider three issues: Are the criteria valid? Were the criteria applied appropriately? Can you use the criteria in your own practice setting? Table 1 sets out a method for approaching an article reporting a process-of-care audit that we will apply to the RAND group's appraisal of the appropriateness of PTCA in New York State as described in the opening scenario.
We deliver medical care in the anticipation that we are improving health status. High quality care is therefore care that does the best possible job of improving or maintaining peoples' health. There is controversy about the extent to which economic issues should be factored into any definition of quality care. However, most physicians would accept that cost minimization is desirable provided that health outcomes are not compromised. Therefore, for process-of-care criteria to be valid, one can reasonably argue that they must have a direct link to either improving health or to lowering resource use without compromising health outcomes.
Like practice guidelines, process-of-care assessments require a review of the evidence and an assessment of the implications of the evidence for clinical practice. In essence, they are guidelines for others to use in assessing whether a practitioner made the right decision, as opposed to guidelines aimed at helping a practitioner actually make clinical decisions. The questions for appraising the validity of criteria for a utilization review are therefore similar to those presented earlier for practice guidelines [2].
The optimal approach to identifying, selecting, and combining evidence for process-of-care audit criteria is similar to that for either an overview or practice guideline, and readers can look to previous articles in this series to refresh their memories [2] [5]. If those developing the criteria have not been both explicit and appropriate about how they chose and summarized evidence, the validity of the exercise is in serious question.
The RAND review of PTCA [6] is a background document, and not part of the article that our clinician read. This is a common problem in the literature. Like an iceberg, guidelines and clinical audit criteria may simply represent a "visible tip", supported by a large literature review which most journals don't wish to publish, and most clinicians won't want to read. At the least, however, readers should seek a description of how the literature was assembled and distilled. In this case the RAND review did use an explicit search strategy and inclusion criteria, considered the design of studies and corresponding strength of the evidence about the benefits and risks of PTCA, and aggregated results quantitatively where possible [6].
If the authors have used an adequate process for summarizing the evidence, the reader must still decide on the methodological quality of the evidence itself. Are the criteria based on evidence from high-quality studies, preferably definitive randomized trials or meta-analyses of multiple trials? Are most of the key indications for the service covered by trial evidence, or must observational evidence, inference, and expert opinion be brought frequently into play? When the evidence is weak, those developing audit criteria must integrate the available evidence with expert opinion. Reliance on opinion and inference weakens not only the validity of the criteria for a process-of-care audit, but also the validity of any conclusions derived from applying the criteria.
The RAND group highlights that at the time they conducted their work no randomized-trial evidence of PTCA versus alternative therapies existed [1] [6]. We find no surprise that 38% of patients had PTCA for uncertain indications. While the use of PTCA in patients during and soon after acute myocardial infarction has been exhaustively evaluated, the research community has been slow to determine just when PTCA is useful in patients with stable coronary disease. Investigators have now reported on one randomized trial of medical therapy in stable single-vessel disease, where CABG is seldom indicated [7] and and versus PTCA [7], and four of PTCA versus CABG [8] [9] [10] [11], but controversy continues about whether CABG or PTCA is preferable and in what circumstances.
In the absence of strong evidence, the RAND group tapped expert opinion by having a multispecialty panel rate hundreds of different case scenarios on a risk-benefit scale. The panelists made their ratings based on both literature reviews and their own clinical judgement. Each scenario describes a potential indication for the procedure or clinical service in question. The researchers then use the pattern of panelists' ratings to assign each indication to an appropriateness category. They assign an "uncertain" category if the panelists agree that the scenario is a risk-benefit toss-up, or if there is sharp disagreement among panelists with some favouring the procedure and others not. As Brook et al wrote several years ago in proposing this method [12], "The panel method can synthesize existing knowledge, but it cannot create new knowledge." Thus, if large proportions of indications in practice are rated uncertain, we need more evidence from randomized control trials. That unquestionably was true for PTCA. One weakness of this method is that for any given clinical indication, the researchers never make clear whether the appropriateness ratings rested primarily on research evidence or inference, extrapolation and opinion. Ideally, reports summarizing the results of applying such criteria should show the extent to which findings of inappropriate care rest on high-quality versus lower-quality evidence. This was not done in the RAND report on PTCA.
Any treatment decision involves tradeoffs, and tradeoffs imply value judgements. Ideally, process-of-care criteria will state the source of the values they used, and the nature of those values. When, as is usually the case, the investigators are silent on this issue, one can assume that the values of those developing the criteria determined the decisions they made.
The results of the randomized trials of CABG versus PTCA that were ultimately done highlight this issue. PTCA appears to have a slightly lower early mortality, along with lower initial costs and more rapid recovery from the procedure. However, longer-term mortality data are similar, and CABG patients appear to achieve better symptom relief, have decreased use of medication, and require fewer subsequent procedures [8] [9] [10] [11]. When panelists in the RAND study were rating the appropriateness of different case scenarios for PTCA, they were presumably weighing trade-offs between early and late effects of PTCA and CABG. However, the source or nature of the value judgements is not made explicit.
Experts' inability to agree on criteria is concerning, as it may reflect weak evidence and/or different values and approaches to interpeting available evidence. When the RAND method was being established, Park et al [13] reviewed 5 procedures and found that the expert clinical panelists agreed on ratings for 42-56% of indications and disagreed on 11 to 29%. This is not encouraging. A Canadian group using a similar method has shown that agreement among panelists is greatest when randomized trial evidence is available concerning the indication of interest -- an understandable finding [14]. Also somewhat concerning is the fact that results vary with panel composition. Other studies show that surgeons' ratings of surgical options are more favourable than physicians, and that medical generalists are systematically more negative in procedural appropriateness ratings than medical specialists who provide the procedure [15] [16] [17] [18]. Even when panels have similar practitioner profiles, the nationality of the panel markedly affects the ratings of scenarios [19].
If panelists have restricted themselves to criteria that follow directly from strong evidence from randomized trials, no further validation is necessary. For instance, one might conduct an audit of prescribing practices after myocardial infarction, and simply address whether, in the absence of specific contraindications, clinicians prescribed aspirin and beta-blockers -- two treatments shown by systematic overviews of randomized trials to reduce mortality. The reader could readily conclude that the criteria were valid, and that outcomes may be adversely affected by underuse of these proven treatments. In many audits (including the RAND audit of PTCA in New York) this will not be the case, and our discussion has highlighted the limitations of relying on expert opinion as an alternative.
Under these circumstances, investigators (and users) can add strength to the criteria by determining whether adherence to the criteria improves patient outcomes. How might such a validation study be performed? One option would be to use the criteria in a continuous audit or quality improvement process. Practitioners could receive regular feedback as to how often their decisions failed to meet established criteria, and might change their practices accordingly. More intrusive options would be to institute prospective utilization review mechanisms, wherein the practitioner is unable to offer a particular service unless explicit indications are met.
In these situations where the goal is to change what physicians do, researchers can randomly allocate practices or practitioners to usual care versus a program of concurrent audit, focusing on the service(s) of interest. Although the design is much weaker, another practical expedient may be to use so-called historical controls, whereby earlier patient experience is documented and compared to that after a program of audit or prospective case management is implemented. Any comparison should assess outcomes for referred persons, including those who did and did not receive the service of interest under the two differing approaches -- usual versus "managed" care. Such studies are tantamount to assessing a therapeutic intervention and could be critically appraised using criteria we've suggested in prior Users' Guides [20] [21]. A methodologically strong study showing patient outcomes improve, or similar outcomes with reduced costs, provides powerful validation of any set of audit criteria.
At this point, however, you will recognize that audit-and-feedback mechanisms change practice in part because practitioners internalize the criteria, which thereby function as practice guidelines. It surely makes sense to teach practitioners formally or informally about preferred practice patterns, and encourage their professionalism, rather than relying on the negative reinforcement of audit-and-feedback alone. Thus, in implementation, one commonly finds further blurring of the distinction between practice guidelines and explicit criteria for auditing quality-of-care.
The study by the RAND group [1] is a retrospective record review, not a randomized trial of concurrent audit-and-feedback using explicit criteria. Investigators can nonetheless present weaker evidence from observational studies to provide some validation. As one example, let us say that reviewers examined a large number of practices. If the criteria were valid, we would expect patient outcomes to be worse, or to be similar but with higher costs, in practices where adherence to the criteria was low. Here, 15 hospitals in New York State were found to have severity-adjusted hospital-specific mortality from 0% to 5%; complications, broadly defined, ranged from 4% to 17%. The article reported only that these were not significantly different among hospitals, and did not correlate appropriateness assessments with outcomes.
Even if criteria are developed soundly and on the basis of strong evidence, their application may be seriously flawed. The following section should help you in two tasks. First, if you are critically appraising an article reporting the results of a utilization review, you can ensure that the audit criteria were properly applied. Second, these criteria will also help you understand some pitfalls in applying audit criteria to your own practice setting.
Application of explicit process-of-care criteria often rests on data derived from retrospective chart reviews by professional auditors. Your confidence in their findings should be strengthened if two or more auditors generate the same data from the same patients' records, if the abstraction process is unbiased by knowledge of the involved practitioners or institutions, or if the findings agree with those of a reference auditor with proven expertise. Reproducibility demands very explicit definitions of the clinical variables incorporated into the criteria, e.g. if PTCA is deemed appropriate for refractory unstable angina with 1-vessel coronary disease, then there should be a clear definition of refractory unstable angina.
In addition, if the criteria were applied to a selected or inadequately large sample of practitioners, hospitals, and patients, the conclusions may be misleading.
In the RAND study of New York state hospitals [1], the inter-auditor reliability of the chart review process is not described, there is no mention of agreement with a criterion-standard abstractor, and there is no mention of blinding as to institutional identity. However, the process they used is well-established, with good inter-abstractor reliability for other services [22] and blinding is less of an issue when investigators use their own auditors, explicit criteria, and detailed structured chart reviews. The RAND investigators appropriately selected a random sample of both hospitals and patients. Their sample of approximately 1500 charts may, however, be inadequate. Institutions had between 1 and 9% inappropriate procedures, but the investigators. could not exclude the play of chance as an explanation for the differences. Differences of this magnitude, if real, would be important to patients, payors, and policy-makers. Thus, the sample size may have been insufficient for the investigators to detect clinically significant differences in quality among hospitals.
Limitations of evidence and uncertainty about values may suggest different criteria for appropriateness, and investigators should examine the impact of these different criteria. This may be done in a number of ways. If panelists have disagreed, investigators might present alternative results based on ratings from both the harsher and more lenient raters. Alternatively, one could look at the implications of assuming that ratings of "uncertain" represent adequate, or inadequate, care. This examination of alternative ratings is analogous to the sensitivity analysis we discussed in our Users' Guide to decision analysis [23].
Furthermore, standardization of explicit audit criteria exists in tension with a potential lack of responsiveness to mitigating clinical factors. It may be desirable to treat the results of such audit as tantamount to a screening test, with a "second look" by experienced clinicians when explicit reviews identify potential quality of care problems.
The RAND report on PTCA in New York [1] offers extensive sensitivity analyses including an exploration of how cases were placed in the uncertain category (e.g. by explicit ratings of uncertain risk-benefit ratio; by being rated appropriate for revascularization rather than edical therapy, but with CABG preferred to PTCA; and by panelist disagreement). They did not explore what would happen if they had set different thresholds or cut-points for defining appropriateness [24]. As well, all cases deemed inappropriate by application of the audit criteria were reviewed by physician-researchers with the study team to ensure that mitigating factors were not missed. However, the article did not report on the proportion of cases deemed inappropriate by the explicit criteria, but uncertain or even appropriate on a physicians' "second look" using clinical judgement.
If criteria are adequate in terms of their validity, there are still further issues you must consider before you apply them in your own setting. Here, the issue of reproducibility that we have already mentioned reappears. If investigators have shown that they can train auditors to make reproducible judgements, it increases the likelihood that you could do the same with your own staff. But even if you can train others to make reproducible judgements about the data elements that define appropriateness, an audit may not be feasible if the exercise is too time-consuming.
Can the internist in our opening scenario apply the RAND criteria? Though not formally tested here, it appears likely that the RAND criteria-based assessments of PTCA are reproducible, and they were applied successfully in diverse hospital settings in New York [1]. However, the article does not detail how laborious this process is. The chart abstraction booklets are lengthy and auditors should have a clinical background and undergo special training. It is therefore a challenge for busy hospitals to use them.
Finally, it is crucial that the criteria be up-to-date. What is optimal practice at one time may be malpractice a short time later. One can imagine what process-of-care criteria would look like before and after randomized trials of anti-arrhythmic agents ecainide, flecainide and moricizine (which kill) or carotid endarterectomy (which, despite expectations that it would go the way of intracranial/extracranial bypass surgery, reduces strokes among selected patients with transient ischemic attacks by more than 50%).
This Users' Guide provides an approach to critically appraising quality-of-care studies that focus on the process of delivering a service. The RAND methods critiqued here are meticulously documented and superior to many other approaches to defining and measuring quality of care, an observation that highlights the difficulties inherent in this area of modern medical research and management. Many clinicians may prefer more straighforward approaches. For example, one might conveniently focus on randomized control trial evidence in isolation, and derive indications where the service is either highly effective or definitively proven to be inferior to alternatives. Other indications can be set aside as resting in the "grey zone" of uncertainty where reasonable persons can disagree [19]. While this approach is simpler and less controversial, there are two problems with streamlined criteria. The first problem is that randomized trial evidence is often limited and may never become available for some procedures and clinical situations [19] [25]. A commitment to evidence-based practice cannot preclude the reasonable use of clinical judgement, inference, and extrapolation. The second problem is that trials are better at helping us decide what to do than what not to do. Expert panels, with all their limitations, do permit detailed assessments of inappropriate and uncertain indications [26].
At present, however, the proliferation of quality-of-care assessments has greatly outstripped the credible research in the field [3] [4]. Despite the eager embrace of "managed care", the measurement of quality of care remains difficult. Reliability of implicit assessments is low, while the available evidence for derivation of explicit criteria is often limited. Furthermore, the overall impact of these criteria on clinical behaviours, system costs and patients' health outcomes is difficult to know as they are seldom evaluated in formal prospective studies, and are often coupled with changes in practice organization and/or reimbursement that in themselves may change behaviour.
The resolution of our scenario has you, the next day, dictating a note to the authors of the RAND report, seeking their background documentation, ratings of case scenarios, and a sample chart abstraction form. You revisit the library and do a Medline search using the MeSH headings, PTCA, randomized control trial, human, and english language. Several relevant studies have appeared since 1990. The articles in question are again digested over lunch at your desk and in the evening. At the next Medical Advisory Committee meeting, you are prepared to discuss the RAND study on PTCA, and plan to highlight three concerns - lack of updated criteria, the limited trial evidence on which the criteria are based, and the likelihood that many of the indications will remain uncertain.
As you are about to start your presentation, however, the chief of cardiology informs the committee that she has been to the library, to health records, and to visit colleagues at the two area hospitals with different utilization statistics. She presents data showing that the discrepant utilization profile is almost completely attributable to acute PTCA for myocardial infarction, which your hospital's cardiologists offer as an alternative to thrombolysis for patients presenting early after the onset of symptoms. There are four randomized trials in the recent literature [27] [28] [29] [30], two of which suggest that PTCA is a safe and effective alternative to thrombolysis for recanalizing the infarct-related artery [27] [28]. The chief of cardiology rightly claims "Many reasonable cardiologists favour direct PTCA over thrombolysis when patients present early and are suitable candidates for emergency angioplasty." As the meeting degenerates into a squabble over whether the administrator should apologize to the hospital's cardiologists for his allegation, you drift off into a daydream about how your own practice would stand up to a well-conducted audit.
© 2001 Centre for Health Evidence.
Home.
Users' Guides to EBP.
Webmaster.
Disclaimer.