Where is this article. [As requested.. sorry it is a little under-edited, Adam ps I have seem to be having problems with formating, I assume it is not because I have done something silly - but this might not be a safe assumption] [The formatting problems are probably my fault. I’m in the middle of some changes to the way the wiki program works. Jason]

Asymmetry of EBM

The aim here is just to highlight the fact that EBM + classical statistics is focused almost entirely on testing and establishing the benefits of medical interventions. The consequence of this is that it is (exceedingly) bad at testing and establishing adverse effects.

This is part Sociological Fact (?terminology) and part nature of classical statistical inference. The sociological fact part is that it is multi-national drug companies that conduct and pay for most drug trials and predominately what they are interested in is showing/finding the benefits of interventions (I agree that this is more nuanced and that the drug companies do want to ensure that the drug is safe enough that the drug will be marketable. [Legal question: is a drug company liable for an unforseen side-effect if it can show that it did everything it could to find out about side-effects according to the prevailing scientific paradigm — i.e., according to EBM? Jason. Unsure if this is rhetorical - but my reply would be no - it seems hard enough to pin companies down when they overtly appear a little less than ethical (even then the debate appears to be focussed on the science), I haven’t kept up with all the facts but suicidal ideation with antidepressants would appear a good example].) It is the EBM + classical statistical inference part that I am particularly interested in. (I will continue to call the combination of these two EBM ’ I find it an interesting question as to whether the two are actually separable ’ but that is another wiki page — Does E B M Always Use Classical Statistics).

How is EBM asymmetrically focused on benefits of drugs?

- Classical statistics limits us to the testing of two hypotheses. This is usually just the null and the alternative ’ almost invariably, in large RCTs, the null is that the drug under study does not have a particular benefit and the alternative is that it does have a particular benefit. This means that the entire study is focused on testing this question ’ the randomisation procedure, the power’ everything. - See the paper Yusuf S, Collins R, Peto R., [[Why do we need large simple RCTs?]] Stat Med. 1984 Oct-Dec;3(4):409-22. This paper was very influential. Jason - The limits of this are: - (1) Power. Simply put RCTs (with their focus on benefits) are typically drastically underpowered to pick up any particular adverse effects. There are two problems with this: firstly many ‘real’ adverse effects will not be picked up as significant and secondly any adverse effects that do show up as significant will likely to be exaggerated in magnitude (this seems to follow directly from the power calculation as discussed in the Cox-2 example) - (2)[=EBMs=] hierarchy of evidence provides an additional problem. Given that the vast majority of RCTs are focused only on benefits and [=EBMs=] hierarchy of evidence privileges RCT over alternative experimental design, EBM in effect privileges benefits over adverse effects. If one adheres to EBM and associates methodological rigour with strength of recommendation/belief - our strength of belief is skewed towards benefits. That different questions need different methodology is something that seems to be acknowledged among some proponents of EBM (Glasziou BMJ 2004;238:39-41). - (3) There are a number of what seem to be interesting questions here: Is there something inherently different about adverse effects that requires different methodology? Could a Bayesian methodology be used for both benefits and harm? How would this methodology look? It will be interesting to have a look at the Upsalla WHO centre (see Read Find) in this regard in how they use Bayesian methodology to pick up adverse effect signals. - (4) More’ e.g. randomisation, ?others’ I need to think these through - I should note that there are some recent trials which take a different tack and to explicitly bring adverse effects into the design and power calculation of the trial. In mind are the recent Women’s Health Initiative studies which set out to test the hypothesis of whether the benefits of HRT outweigh the risks. The set up of the trial explicitly took into account the proposed benefits of HRT and weighed this against the proposed risks. This seems a step forward in terms of reducing the asymmetry of EBM but still raises the traditional problems of classical statistical inference, notably stopping rule problems (the trial was ended when the risks outweighed the benefits with a p-value of 0.05). [Terrific example. Jason]

How then does the current model (EBM) find/establish adverse effects?

- This is done in the first instance during drug development. One of the main objectives of drug development is to screen for pharmacological properties which in the past have lead to well known adverse drug effects. The archetypal example being prolongation of QT interval in animal models ’ this suggests the possibility of the drug causing arrhythmia in humans. Much effort is put into this in order to minimise R&D expenditure on drugs which will not be marketable. Some notes: - (1) This method of picking up adverse effects of drugs is clearly limited by what we know to screen for ’ if the new drug causes a new adverse effect this may not be picked up. I need to check on this but I have the impression that the range of things which are being screened for from a toxicology perspective is pretty limited compared to the wide range of things which can go wrong. - (2)The differing methodology for adverse effects picked up in this way as opposed to benefits should also be noted, i.e. we are relying here on pharmacological stories (the bottom of the EBM hierarchy). I should note that at this stage of drug development pharmacological stories are also all we have with regard to the benefits of the drug ’ the difference is that we won’t believe the benefits until we have a large RCT, we are unlikely ever to do a large RCT to establish the potential harms of the medication. This all becomes part of the ‘How does EBM and the basic sciences inter-relate?’ story. [Good points. Jason] - Once the drug is marketed the reliance is on surveillance to find and then (attempt to) establish adverse effects. The unit of information here is typically individual case reports from healthcare professionals and patients using the medications. There are too many limitations of this approach to begin to list [I hope you don’t mean that literally, because if you don’t list them who will? Jason I agree, I was being a little lazy - I think there will be some literature on this] and yet it is the key form of evidence for marketed drugs (compare this to RCTs for benefits and the asymmetry is obvious). Once a signal (pharmacological or from surveillance) is detected the question arises as to how this signal should be established ’ the options are RCTs (with the problems of which listed above); purposefully designed cohort or case-control studies (which of course are lower than RCTs in [=EBMs=] evidentiary hierarchy, while case-control studies, which are the most efficient studies for this purpose are even lower than cohort studies); Bayesian networks (as per the Upsalla centre - Read Find - I am keen to get a feel for how they go about this use of bayesianism) or finally (and this seems to be the typical story) a mixture of each (causal pharmacological story + limited RCT evidence + case-control data) and public hysteria (read Vioxx, and to an extent HRT)

Where does this fit? - ?maybe a chapter; once I have a good search and get my head around the literature which exists on this, I am sure there would be a paper [This is good stuff, worth several papers and a whole thesis! If you’re sufficiently interested in it, that is. Jason] - Focussing on the statistical inference aspect there seems to be a number of topics/questions - (1) I need to explicate precisely the limitations of classical stats + EBM with regard to picking up adverse effects. Which of this are different to the general limitations of classical statistics? - (2) How does the type of statistical inference link to your experimental methodology? [Grossman et al 1994 paper in Statistics in Medicine gives part of an answer. Jason] - (3) Is there a better reply to this problem than flexibility in methodological hierarchy (as seems to be suggested by Glasziou)? Answer: yes, but precisely how?