E-cigarettes and formaldehyde

E-cigarettes and formaldehyde: anatomy of a flawed study 9 September 2015 1. Overview On 22nd January, New England Journal of Medicine published a paper, Jensen et al, Hidden Formaldehyde in E-cigarette Aerosol which gained much media attention, suggesting that formaldehyde exposures to e-cigarette users, and related cancer risk, might be 5-15 times greater than for smoking. The paper was deeply misleading as the e-cigarettes used were operated in conditions that no user would experience - overheating the liquids creates thermal degradation products, but also makes the the vapour taste terrible. Jensen et al, were measuring the thermal degradation products and calculating the cancer risk to humans, but taking no account of the the reality that vapers just don’t use the products in this way - under the well known ‘dry puff’ conditions. We made a complaint to the NEJM and called for the paper to be retracted. Neither efforts of the journal or the authors have amounted to a remotely credible defence of what was done and how it was published. The basic case for retracting Jensen et al. Hidden Formaldehyde in e-cigarette aerosol is set out in this short summary: Summarising the case for retraction. This document provides access to all of all the correspondence and exchanges related to the case. 1.1 Research letter alleges high levels of formaldehyde In January 2015, the prestigious journal the New England Journal of Medicine published a study: ‘Hidden formaldehyde emissions in e-cigarette aerosol’, which investigated emissions from e-cigarettes and found high levels of a particular form of formaldehyde when the device was operated at high power. It then made a comparison with formaldehyde exposures from cigarette smoking based on other studies and concluded with a calculation of cancer risk, comparing smoking and vaping. The suggestion in the paper that the incremental cancer risk for vaping may be 5-15 times higher than for smoking generated predictable headlines. 1.2 Critique of the published study Using laboratory equipment, the study operated the vaping device at a such a high temperature that it produced thermal breakdown products, which the authors measured and used as a basis for calculating formaldehyde exposure and related cancer risk. But in reality no human user would ever be able to vape in this way – the vapour would be too acrid (so- called dry puff conditions). Their measurements were not therefore a reliable proxy for human exposure, but they went on to calculate human cancer risk from these unrealistic machine measurements and presented the data in way that was bound to mislead. Their method took no precautions to ensure that the operating regime and measurements were realistic for humans rather than an artefact of machine setting and puff regime that no-one

would ever experience. This was irresponsible science and careless publishing, with the likely harmful effect of adversely changing the perceptions of the relative risks of smoking and vaping in a way that will cause harm. The paper should be retracted in its entirety. It is insufficient to only respond by publishing critical letters: the paper remains as part of the academic record and continues to be cited and used by governments and public bodies to justify policy proposals and to communicate risks to consumers. It is important to be clear, that misperceptions of risk caused by flawed academic papers do have potential consequences for human health. 1.3 Formal complaint and call for retraction Clive Bates and Konstantinos Farsalinos formulated a complaint to the NEJM and made a case for retracting the research letter. The case draws on the Committee on Publication Ethics Code of Conduct and Retractions Guidelines. The NEJM dismissed the complaint, declined to retract the paper and did not provide details of its complaint procedure or the options available to dissatisfied complainants. In our view, NEJM and the authors have not taken this complaint seriously or met their responsibilities to retract unreliable papers and to withdraw their conclusions. 2. Key documents 2.1 Original research article and publicity 22 January. Publication of research letter. New England Journal of Medicine publishes Hidden formaldehyde in e-cigarette aerosol. The key newsworthy wording in the research letter implied vaping posed significantly higher cancer risk than smoking ….then long-term vaping is associated with an incremental lifetime cancer risk of 4.2×10−3. This risk is 5 times as high (as compared with the risk based on the calculation of Miyake and Shibamoto shown in Figure 1), or even 15 times as high (as compared with the risk based on the calculation of Counts et al. shown in Figure 1) as the risk associated with long-term smoking. 21 January. NEJM tweet. The impression that vaping may be more harmful than smoking may have been reinforced by NEJM social media promoting the paper: which included the sentence: “Authors project higher cancer risk than smoking” NEJM tweet: 21 January 2015

22 January. Portland State University press release: PSU researchers uncover high levels of hidden formaldehyde in e-cigarette vapor 2.2 Letters published in NEJM - an inadequate response 17 April. Letters published in NEJM in response. More on Hidden Formaldehyde in E- cigarette Aerosol. Nitzkin, Siegel & Farsalinos summarise the major flaw in the work in just 128 words. The original authors provide a wholly unconvincing reply to the criticism in 180 words, avoiding the main point - and the exchange ends there. The original research letter remains intact and part of the academic record despite the fatal criticisms and the evasive and unsatisfactory response. 2.3 Complaint sent to NEJM addressing failings in detail Our view was, and remains, that the exchange of nugatory correspondence in which the original authors ducked the main criticism of the work was completely inadequate - as it left this wholly flawed paper on the academic record with accumulating citations. So Konstantinos Farsalinos and I formulated a more complete critique of the letter and raised this as a complaint with the NEJM editors. We also addressed the multiple shortcomings of the authors’ reply to the Nitzkin et al critical letter published by NEJM. Basis of complaint - Committee on Publication Ethics codes. Most major academic journals, including NEJM, are members of the Committee on Publication Ethics (COPE). Members agree to abide by rules for ethical conduct in publishing research, including a Code of Conduct and Retraction Guidelines. We made a complaint to the NEJM with reference to its obligations under the COPE codes and guidelines. 20 April. Detailed complaint. From Clive Bates and Konstantinos Farsalinos sent to NEJM editor Jeffrey Drazen calling for retraction of research letter. Complaint under the Code of Conduct of the Committee on Publication Ethics: Letter in the New England Journal of Medicine Jensen et al (2015) Hidden Formaldehyde in E-cigarette Aerosol. The complaint concludes: The clear message readers were intended to take from the letter is that usage of e- cigarettes can cause more formaldehyde exposure, and related cancer risk, than smoking cigarettes. But this is flatly contrary to substantial existing evidence, and the authors’ unrealistic test does not support this claim. The findings of this letter are unambiguously inaccurate, misleading and wholly unreliable. For that reason the letter should be retracted under the principles established by the Committee on Publication Ethics. The errors made were foreseeable and diligent researchers familiar with their field should have avoided them. Such studies can adversely affect public risk perceptions in a way that can lead to additional ill-health as people who are at risk respond to reports about the research in the media. It is therefore particularly important that great care is taken in conducting and reporting on such research and that academic

journals take a robustly ethical approach when such research is found to be fundamentally flawed. 20 April. Supporting letter. Summary of case signed by 40 academics and experts summarising the concerns and respectfully inviting NEJM to examine the complaint carefully. Letter from 40 academics and experts in support of the complaint about the research letter published in the New England Journal of Medicine on Hidden Formaldehyde in E-cigarette Aerosol However, it appears clear that the device used was operated in unrealistic laboratory conditions and that human subjects would never experience the exposures that were measured and used in the calculation of cancer risk. The letter is therefore based on a major methodological error, and the calculations of cancer risk and comparison with cigarette smoke are invalid and misleading. Cigarette smokers face high risks of disease and premature death. It is essential that they are provided with valid, well-founded information about the options available to them to stop smoking or to cut down, including switching to e-cigarettes. Misleading information about risks of e-cigarettes may adversely influence choices made by smokers in a way that may harm their health. There is, therefore, an especially strong ethical imperative to measure and report risks associated with e-cigarettes with great care. 20 April. Covering letter to NEJM. Complaint regarding research letter: Hidden Formaldehyde in E-cigarette Aerosol. As well as enclosing the two document above, this letter asks to NEJM to set out the process for evaluating the complaint - a requirement of the COPE Code of Conduct. 21 April. Letter direct to editor of NEJM. Fwd: Complaint regarding research letter: Hidden Formaldehyde in E-cigarette Aerosol - this forwards the complaint sent to the general editorial desk and reiterates request for clarity on process - citing the relevant COPE Code of Conduct Provisions. 2.4 Response from NEJM dismissing the complaint The NEJM’s editors decided not to retract, amend, qualify or otherwise indicate the unreliability of the findings. The letter remains part of the academic record and is cited in formal and grey literature. The flawed method continues to be used in other research. 21 April. Acknowledgement. RE: “Complaint under the Code of Conduct of the Committee on Publication Ethics. “The New England Journal of Medicine takes the complaint very seriously. It will receive careful and thorough attention by the editors.” No further information was provided about the complaint process, though we had asked NEJM to set out its procedures. 27 April. Substantive response from NEJM. RE: “Complaint under the Code of Conduct of the Committee on Publication Ethics”

As we read the COPE and ICMJE guidelines, retraction would be warranted if there was evidence of scientific misconduct, inaccurate methods for the determination of the concentrations of the various analytes in the vapor, or the use of inappropriate statistical methods to draw the conclusions. The document sent to us does not indicate concern in any of these areas. Thus, we see no basis for retraction. We appreciate that there are grounds for healthy scientific debate about whether the settings on the device used to generate the analyzed vapors are within the range of realistic use. It was precisely this point that was aired in the Letter to the Editor by Drs. Nitzkin, Farsalinos, and Siegel, published on April 16, 2015. We understand that Bates and Farsalinos (and others) were not happy with the authors’ response, but that is the substance of scientific debate. New data are needed to address this issue 2.5 Reply to the NEJM dismissal We regarded the response from the NEJM as a white-wash that did not address the substantive issues and focussed on irrelevant issues. So we replied highlighting the flaws in the NEJM dismissal. 1 May. Reply to the NEJM. Re: Complaint regarding research letter: Hidden Formaldehyde in E-cigarette Aerosol I set out why the editor’s review had misrepresented the legitimate reasons for retraction which are: Retractions guidance Journal editors should consider retracting a publication if: they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error) I set out again the case for retraction: ● Human exposure and related cancer risks were then calculated using emissions measurements made in these unrealistic conditions for human use. This means that the method used to measure emissions (machine measurement in overheated operating conditions) is the wrong method to characterise the actual quantity of interest (human exposure to formaldehyde). It is standard practice to retract research findings based on use of inappropriate experimental method - what would be the alternative? ● The exposure and cancer risk calculations based on these measurements are therefore miscalculations and unreliable in the meanings used in the COPE retractions guidelines as cited above. The reported cancer risk is the key finding of Jensen et al, but it does not exist in reality. This fundamental failing is not addressed in the response to our complaint from the NEJM. ● The authors took no steps to guard against this experimental design flaw, and their response to the critical letter published in NEJM confirms this. It was their responsibility to show that the cancer risks they calculated were meaningful human risks and therefore fit to publish in a leading journal concerned with human health.

● Publication of a short letter in the NEJM (Nitzkin, Farsalinos, Siegel) pointing out this serious experimental failing does not render the original calculations any less unreliable or diminish the case for retraction. It is not simply a matter of debate and opposing opinions, it is an unambiguous experimental design failure leading to a misleading assertion of risk with potential to cause harm to health. I also requested clarification of the complaints procedure and the options available to dissatisfied complainants, which NEJM is supposed to have as part of its COP obligations. Section 15.1: 15.1. Editors should respond promptly to complaints and should ensure there is a way for dissatisfied complainants to take complaints further. This mechanism should be made clear in the journal and should include information on how to refer unresolved matters to COPE. There was no reply from NEJM to any part of this communication. 2.6 Formal correspondence - Addiction journal Given the harms arising from a misleading paper, we are not content to let the matter rest with an inadequate exchange of letters in the NEJM and a ill-founded dismissal of a carefully constructed complaint to the NEJM and authors. We have therefore written to the journal Addiction to put this case on the record. Addiction accepted this correspondence and published in its October edition and online on 9th September 2015. These are the 9 September. Letter to Addiction outlining the basis of the case for retraction from NEJM. Research letter on e-cigarette cancer risk was so misleading it should be retracted 9 September. Response from some of the original authors (Pankow et al), fails to address or acknowledge the weakness in their method… Formaldehyde from e-cigarettes—it's not as simple as some suggest 9 September. Reply to authors, reiterating the central flaw in their work. E-cigarettes need to be tested for safety under realistic conditions 9 September. Supplementary material. Addiction includes original detailed complaint to NEJM and supporting letter to NEJM from 40 experts. 3. Other relevant material 3.1 Pre-publication warnings I saw a press release for the study prior to publication, and the flaws were immediately obvious - as was the potential to cause harm - I wrote to both the authors and NEJM on the day before publication. I did not expect them to withdraw the publication as that notice, but hoped they may qualify their confidence in the finding or add caveats.

21 January. Letter Clive Bates to authors: Hidden Formaldehyde in E-cigarette Aerosols - some questions and concerns 21 January. Letter Clive Bates to NEJM, copying letter to the authors to the editor. Hidden Formaldehyde in E-cigarette Aerosols - some questions and concerns To summarise the concerns expressed in these letters: To be more direct, I am concerned that: ● This study uses a completely unrealistic puffing regime to create the conditions in which formaldehyde forms with no attempt to calibrate the machine to reflect realistic human use. ● That it presents results from extreme and unrealistic operating conditions which are then built into a 'back of the envelope' calculation of cancer risk. ● That this contrived and artificial cancer risk is misleadingly compared to real human cancer risks associated with smoking. ● That the statements about vaping having 5-15 times the incremental cancer risk associated with smoking are provided without context and could easily be misread as implying that vaping is more dangerous than smoking - it would not be the first time that misreporting of formaldehyde findings have created this impression. ● This study may repeat the harm done through mischaracterisation of 'light' cigarettes by use of unrealistic puffing regimes that did not reflect real human behaviour. The same is likely to apply here, but instead of understating risk of a harmful product, the effect will be to grossly overstate the risk of a relatively benign product - with equally damaging results. 3.2 Online commentary - post publication As the paper was published, many expert commentators attempted to provide a counterbalance through rapid online postings - NEJM does not allow comments on articles published online, and the academic timescales are far too slow to meet media deadlines. The study was published with all the authority of the NEJM and its apparently shocking conclusion gave it great momentum. However, in substantial terms the commentaries were highly critical and robust. Dr Konstantinos Farsalinos: The deception of measuring formaldehyde in e-cigarette aerosol: the difference between laboratory measurements and true exposure The scientific community must realize that variable wattage devices cannot be used at any wattage levels with any available atomizer. Even for naïve users, the harsh taste of the dry puff phenomenon is unbearable. … In fact, it is very easy to produce as much aldehydes as you want in the lab with an e-cigarette device. However, this has nothing to do with exposure from e-cigarette use. Dr Michael Siegel: New Study Reports High Levels of Formaldehyde in Electronic Cigarette Aerosols

Essentially, what this study demonstrates is that if you overheat a vaping system, it will produce high levels of formaldehyde. However, such conditions are not realistic, as they could not be tolerated by an actual vaper. Therefore, extrapolating from this study to a lifetime of vaping is meaningless. Norbert Zillatron: Freaking Formaldehyde …you can’t just simply select an arbitrary puffing regime and expect it to represent an applicable model of real vaping Tom Pruen (ECITA) First burn the barrel, then scrape it – commentary on a letter published in the NEJM Consumers are extremely unlikely to voluntarily inhale high concentrations of formaldehyde; formaldehyde is characterized by its unpleasant smell, and at concentrations of as little as 5 parts per million causes burning sensations in the respiratory tract, and breathing difficulty Professor Peter Hajek: Formaldehyde in e-cigarettes: expert responds When a chicken is burned, the resulting black crisp will contain carcinogens but that does not mean that chicken are carcinogenic. Without overheating the e-liquid, no formaldehyde was detected. Vaping may not be as safe as breathing clear mountain air, but it is much safer than smoking. It would be a shame if this study persuaded smokers who cannot or do not want to stop smoking and contemplate vaping that they might as well stick to their deadly cigarettes. Professor Brad Rodu: Bogus research of formaldehyde in e-cig vapor R. Paul Jensen and colleagues at Portland State University produced the new results by overheating an e-cigarette, a condition (called dry puffing) that is familiar to vapers; the resulting product tastes so bad it cannot be inhaled. In other words, the formaldehyde produced under abusive conditions is not “hidden” at all, because it is in vapor that users find intolerable. Enough data is extracted from the authors to confirm the the measurements were made in extreme and unrealistic ‘dry puff’ conditions. Dr Konstantinos Farsalinos figures out what they actually did: Verified: formaldehyde levels found in the NEJM study were associated with dry puff conditions. An update. It is more than obvious that the findings of very high levels of formaldehyde are a result of overheating. Lack of experience on e-cigarettes and no contact with vapers can result in such erroneous and unrealistic results, which can create confusion and misinformation both in the scientific community and among users and potential users of e-cigarettes. Dr Michael Siegal follows up: Confirmed: Formaldehyde Study Conducted Under Implausible Conditions; Conclusions Invalid

Unfortunately, the alarmist (and incorrect) conclusions of this study have already been widely disseminated in the media. Even if the information is corrected, it appears that the damage is done. I believe that the damage is substantial because many smokers will now become convinced that there is no advantage to switching from tobacco cigarettes to electronic cigarettes. Dr Gilbert Ross of the American Council on Science and Health, Poorly-done e-cig vapor study gets big headlines but means nothing. Dr Ross highlights the additional point that it wasn’t formaldehyde that was measured and formaldehyde is not an especially powerful carcinogen. This flawed study will be used to attack e-cigs as not only not safer than smoking cigarettes, but perhaps even more toxic. Nothing could be further from the truth. Those who promulgate this falsehood should be aware that even if the study were done correctly — which it was not — such a conclusion is the worst kind of destructive fiction. Formaldehyde was not even studied, in fact, but something called formaldehyde releasing compounds. And formaldehyde is a very weak carcinogen, with only a slightly increased chance of cancer among highly-exposed workers over a lifetime. The presence of one such carcinogen in vapor cannot be compared to the toxic stew of carcinogens and other toxins in cigarette smoke. Clive Bates: Spreading fear and confusion with misleading formaldehyde studies draws some of this together. 3.3 Sample press coverage and links ● Reuters: Ramping up e-cigarette voltage produces more formaldehyde -study ● LA Times E-cigarettes can produce more formaldehyde than regular cigarettes, study says ● NPR: E-Cigarettes Can Churn Out High Levels Of Formaldehyde ● NBC news: Before You Vape: High levels of Formaldehyde Hidden in E-Cigs ● CBS News: E-cigarette vapor filled with cancer-causing chemicals, researchers say ● Wall Street Journal: Study Links E-Cigarettes to Formaldehyde, Cancer Risk ● South China Morning Post: E-cigarettes ‘expose smokers to more formaldehyde than regular tobacco’ ● Tech Times E-cigarettes Not Safer than Ordinary Cigarettes, Contains High Concentration of Formaldehyde ● Mail Online: Some e-cigarettes may release more of a cancer-causing chemical than regular tobacco, study suggests ● Consumer Health Day: High Levels of Cancer-Linked Chemical in E-Cigarette Vapor, Study Finds ● Buzzfeed: Study Finds E-Cigs Produce More Formaldehyde Than Regular Cigarettes ● News Everyday: Vaping worse than smoking, formaldehyde study ● Huffington Post (UK): Is Vaping Safe? Cancer-Causing Substance ‘Formaldehyde’ Hidden In E-Cigarettes

Portland University has tracked media and in May 2015 claimed: The story of its findings was picked up by at least 179 news outlets on nearly every nearly doubled in the following weeks. 3.4 Critical journalism Some journalists writing after the release did not take the story at face value: ● Jacob Sullum, Reason, Does Formaldehyde Make E-Cigarettes Worse Than the Real Thing? 22 January 2015 ● Joe Nocera, New York Times, Is vaping worse than smoking?, 27 January 2015 ● Sally Satel, Forbes, New Surgeon General Should Prescribe Scientific Honesty On E- Cigarettes, 2nd February 2015 The study was give satirical treatment on Gutfeld: New e-cig study goes up in smoke “The point: If you create an unrealistic climate for harm you create the harm you seek, which allows researchers to elevate the risk and that lands them in a prestigious journal.” 3.5 The deleted Portland State University article On 13 May, Portland State University published an article (Clearing the Fog: PSU researchers get the world's attention by linking vaping with formaldehyde) including quotes from the authors discussing the study and its reception. By 15 May it had been removed from the internet. At the time of writing it is available in the Google cache and saved in PDF form. attempt at retrospective defence of their methodology: “The critics say whenever you inhale (at the levels where we found formaldehyde), you stop because it tastes bad,” Pankow says. “But that automatically means that you’re exposing yourself to those conditions. This level is a known phenomenon. “Also, it’s pretty well known that inhaling cigars is unpleasant, but people do it,” he says Note that their cancer risk calculations did not assume momentary one-off mouth-only exposure while a user adjusted their settings or changed puffing intensity. Their cancer risk calculation assumed that vaping would always involve this exposure and continue for life at this level. The use of argument by analogy with cigar use simply demonstrates that their method was not suited to measuring the quantity of interest - human exposure to formaldehyde related compounds. Why should dry puff conditions for vaping have the same sensory aversiveness as cigar smoke? Recall, the authors didn't just suggest that these high levels might be a concern if people are actually exposed to them, they made an assertive and unqualified claim about the relative formaldehyde-related cancer risks comparing smoking and vaping - it was this claim that gained the media attention the report with some pride in the article.

3.6 Subsequent academic criticisms Konstantinos E.Farsalinos, Vassilis Voudris, Konstantinos Poulas. (2015) E-cigarettes generate high levels of aldehydes only in ‘dry puff’ conditions. Addiction 110:10.1111/add.v110.8, 1352-1356 CrossRef Electronic cigarettes produce high levels of aldehyde only in dry puff conditions, in which the liquid overheats, causing a strong unpleasant taste that e-cigarette users detect and avoid. Under normal vaping conditions aldehyde emissions are minimal, even in new-generation high-power e-cigarettes. Peter Hajek. Commentary on Farsalinos et al . (2015): E-cigarettes do not expose users to dangerous levels of aldehydes. Addiction 110:10.1111/add.v110.8, 1357-1358 CrossRef Most toasters have a setting which burns the toast to a crisp. The function of this setting presents one of the deeper mysteries of life, but it is there and you can burn your toast if you want to. Although burned toast contains carcinogen, it is highly unlikely that New England Journal of Medicine would publish a paper demonstrating this and warning people that toasts are carcinogenic. This raises an intriguing question of how such studies get to see the light of day and receive so much attention. E-cigs are a disruptive technology which threatens sales of conventional tobacco products and sales of stop-smoking medications, as well as some of the established beliefs and positions in tobacco control. There seems to be a substantial appetite for news of e-cig dangers. Studies which can be interpreted in this way seem to have an easy progress to publication in prestigious medical journals even if they contain obvious flaws; journals tend to publish press releases which further exaggerate the findings; and some public health officials with anti-e-cig sentiments use such reports to warn smokers not to use e-cigs.