2021 Virtual GPF Slides

GPF Day 2 Recap • Breakouts/plenary discussion: – Opportunities for manufacturers to address and respond to uncertainty concerns – Greater training/awareness of novel methods – Can innovation in trial design or other risk adaptations reduce MEA needs? – Elevate not only patient but clinician input – Semi-quantified approach for patient and other contextual feedback – Independent committee action on uncertainty – Opportunities for mediation – Clarify evidence vs. decision uncertainty – Opportunities for rapid evidence generation – Ensure stakeholders feel participation is genuine and trust the process – Acknowledge unresolvable uncertainty – Limits to fixed approaches across settings with different patient/societal preferences and context

GPF Day 3 • Focus on the “output” phase of HTA • What are the key considerations all HTA bodies and stakeholders should have in mind when describing and presenting uncertainty’s role in a decision as well as the actions taken to address it?

Agenda: Day 3 • Patient perspective: Anne-Pierre Pickaert, Patvocates • Communication tools: Jean Slutsky, PCORI • Payer/device/pharma reaction panel – Lynne Nakashima, BC Cancer – Maria Stewart, Boston Scientific – Jake Lebiecki, Pfizer • Coffee Break • Breakout discussions (35 min) • Report back & plenary discussion (40 min) • Wrap-up and topic selection for 2022

Conveying & communicating uncertainty: A patient Perspective Anne-Pierre Pickaert Acute Leukemia Advocates Network (ALAN) February 26, 2021

Conveying and communicating uncertainty: a patient perspective • What is the Acute Leukemia Advocates Network (ALAN)? • Some patients are ready and perfectly capable to take greater risks to gain uncertain greater benefits… • … especially if they are shown the data in a format, which is transparent and easy to understand.

ALAN: a platform connecting ˜110 leukemia patient groups in ˜80 countries

ALAN: changing outcomes of patients with acute leukemias Member capacity building & joint policy-shaping • Create awareness about acute leukemias and how to better support leukemia patients; • Develop patient information and specific support for patients with acute leukemias and their carers in all countries; • Improve education for healthcare professionals serving leukemia patients • Advocate for better treatment, care & access to healthcare services; • Strengthen patient organisations by sharing best practices and providing toolkits in patient advocacy; • Collaborate with other initiatives and stakeholders with similar goals

There is no medicine like hope, no incentive so great, and no tonic so powerful as expectation of something tomorrow. Orison Swett Marden (1848–1924) American author

Population uncertainty: patients are given the same treatment but not all experience the same results Adapted from “Outliers vs extreme responders in oncology, Dr André (Gustave Roussy) at the 2016 Emergency Nursing Association conference.

Individual uncertainty: some patients are ready & perfectly capable to take greater risks to gain uncertain greater benefits The data needs to be shared in a format, which is transparent and easy to understand. Adapted from “Outliers vs extreme responders in oncology, Dr André (Gustave Roussy) at the 2016 Emergency Nursing Association conference.

“Health care should not systematically mislead the public about benefits and about harms” Gerd Gigerenzer, Max Planck Institute, Berlin http://videolectures.net/icots2010_gigerenezer_hdap/; https://www.healthnewsreview.org/podcastmedia/gerd-gigerenzer-podcast-systematic-misinformation-public-health-care/

“Health care should not systematically mislead the public about benefits and about harms” Gerd Gigerenzer, Max Planck Institute, Berlin Relative risks are not transparent for the general public http://videolectures.net/icots2010_gigerenezer_hdap/; https://www.healthnewsreview.org/podcastmedia/gerd-gigerenzer-podcast-systematic-misinformation-public-health-care/

“Health care should not systematically mislead the public about benefits and about harms” Gerd Gigerenzer, Max Planck Institute, Berlin Absolute risks are transparent for most of us http://videolectures.net/icots2010_gigerenezer_hdap/; https://www.healthnewsreview.org/podcastmedia/gerd-gigerenzer-podcast-systematic-misinformation-public-health-care/

“Health care should not systematically mislead the public about benefits and about harms” Gerd Gigerenzer, Max Planck Institute, Berlin Frequent mismatching Relative risk decreases used to show benefits data framing Absolute risk increases used to show harm http://videolectures.net/icots2010_gigerenezer_hdap/; https://www.healthnewsreview.org/podcastmedia/gerd-gigerenzer-podcast-systematic-misinformation-public-health-care/

Collective statistical illiteracy Few physicians, patients and politicians understand health statistics Causes • Non-transparent framing of information • Lack of training in risk communication (schools and medical schools) Some solutions • Teach “the Maths of uncertainty” to all stakeholders, incl. patients • Teach and implement transparent risk communication http://videolectures.net/icots2010_gigerenezer_hdap/

Communicating uncertainty should be transparent and easy to understand E.g. of calculating the probability that a woman who tests positive in mammography screening actually has breast cancer Adapted from https://www.stat.berkeley.edu/~aldous/157/Papers/health_stats.pdf

Communicating uncertainty should be transparent and easy to understand E.g. of calculating the probability that a woman who tests positive in mammography screening actually has breast cancer Adapted from https://www.stat.berkeley.edu/~aldous/157/Papers/health_stats.pdf

Communicating uncertainty should be transparent and easy to understand E.g. of calculating the probability that a woman who tests positive in mammography screening actually has breast cancer: Avoid conditional probabilities Adapted from https://www.stat.berkeley.edu/~aldous/157/Papers/health_stats.pdf

Communicating uncertainty should be transparent and easy to understand E.g. of calculating the probability that a woman who tests positive in mammography screening actually has breast cancer: Natural frequencies More natural representation of uncertainty Adapted from https://www.stat.berkeley.edu/~aldous/157/Papers/health_stats.pdf

Conveying and communicating uncertainty: a patient perspective • ALAN is platform connecting ˜110 leukemia patient groups in ˜80 countries to engage in capacity building & joint policy-shaping • Some patients are ready and perfectly capable to take greater risks to gain uncertain greater benefits… • … especially if they are shown the data in a format, which is transparent and easy to understand. Thank You | [email protected] | https://acuteleuk.org/ Follow us on Twitter, Facebook, Linked In @AcuteLeuk

Research Fundamentals and Public Reporting of Results Jean Slutsky Senior Advisor to the Executive Director

PCORI’s Original Mandate “The purpose of the Institute is to assist patients, clinicians, purchasers, and policy-makers in making informed health decisions by advancing the quality and relevance of evidence… through research and evidence synthesis... … and the dissemination of research findings with respect to the relative health outcomes, clinical effectiveness, and appropriateness of the medical treatments, services...” — from PCORI’s authorizing legislation • Reauthorization language strengthens and formalizes PCORI’s dissemination and implementation mandate. • Directs GAO to review and report to Congress on these activities.

How is PCORI different? Research PUBLICATION Most funders, even if undertaking D&I, Funding have separate programs for research and implementation. Research Funding PCORI is unique in extending the pathway from funded research directly to dissemination and implementation. Publication, Dissemination Initial Broader ROAD TO Release of Implementation Implementation* UPTAKE Findings *Scale up to larger numbers and/or scale out to different audiences or implementation sites

Release of Findings: Getting Started Addressing PCORI’s Legislative Mandate • PCORI’s authorizing law requires us to “convey the findings of research in a manner that is comprehensible and useful to patients and providers in making health care decisions” and to make findings available within 90 days. o Must Include limitations of findings • Within 90 days of PCORI’s acceptance of the investigator’s Draft Final Research Report (following peer review) we post: o Public abstract o Professional abstract o Also: Summary of peer-review process, Conflicts of interest 174

Release of Findings: Posted Results: The Public Abstract Public Abstract: • 6th-8th grade reading level • Q&A style headings, leads with results; personal relevance • Shortens words, sentences, and paragraphs; uses bulleted lists • Avoids jargon; conversational tone • Cognitive testing for all abstracts • Spanish translation, audio, downloadable versions Professional Abstract: • Resembles std. research abstract • Features PICOT box

Release of Findings Return of Results • Returning aggregate research results (i.e., the overall findings from a study, as distinct from individual, patient-specific results) to trial participants fulfills an ethical responsibility to those who make the research possible. • PCORI directs awardees to make every reasonable effort to return aggregate research results to study participants. (Guidance for Peer Review and Public Release of Research Findings, 2015) • May be accomplished by sharing lay language results summary PCORI posts on completion of each study. 176

Efforts to Increase Comprehension • https://www.pcori.org/engagement/research-fundamentals 177



Knowledge Needs Identified • What is research? • What are PCOR and CER? • How is PCOR different from traditional research? • Safeguards/participant protection in research • Engagement in research 179

Research Fundamentals Structure Learning Need Navigation Tool Interactive PDF Video-Based Module Developing Designing the Planning Patient- Sampling, Recruiting, Understanding and Research Questions Research Study Centered Consent and Retaining Study Sharing Research and Study Protocols Participants Findings Five Interactive Learning Modules 180

Research Fundamentals User Experience 181

Research Fundamentals User Experience 182

183

Thank You • https://www.pcori.org/engagement/research-fundamentals

Breakout Discussions

Breakout Question 1. What key considerations should all HTA bodies have in mind as they describe, visualize, and explain the role that uncertainty played in a decision or recommendation?

Group 1 Friday February 26th

What key considerations should all HTA bodies have in mind as they describe, visualize, and explain the role that uncertainty played in a decision or recommendation? • Consistency in decisions • Deliberative processes, but evidence should be considered consistently • Fairness – treating all the same • Challenge: Accounting for individual patient goals; broader patient goals • Medical need taken into consideration • Predictability & Timing - communicating in sequences as new data emerge – a “continuous dialogue” • Timing: HTA is a continuous process  reduce risk over time; include evidence as it emerges • Reassessment with new evidence: what is the timeframe? • “More data needed” – what data could be generated to overcome critical data challenges • Challenge: changing standard of care over time – offer a timeline for a re-review • Health literacy principles for patients AND decision-makers • Complexity of evidence and data sources • Who are we trying to communicate to? • Available and understandable information • One size does not fit all • Transparency • Forthcoming regarding challenges • Methodological/standards – where do they come from? Are they relevant? Developed with rigor (thresholds, measures) • Totality of evidence • What was considered? What was not considered? Including how did patient experience data contribute • Public trust in the decision-maker framework

Break-Out Group 2 HRTeApcoomrtmOunuictation of uncertainty during the “output” phase Friday, February 26 (08.00-08.35 MST / 16.00-16.35 CET).

Breakout Group 2 What key considerations should all HTA bodies have in mind as they describe, visualize, and explain the role that uncertainty played in a decision or recommendation? Key Considerations that Vary by Stakeholder Resource Implications for effectively communicating uncertainty • sUtsaikneghpolladienrsla,ningculaugdee aendducfaigtiuorneasl tmo amtearkiaelsresults understandable to • cCoommmmuunniiccaattiioonns team; include team with appropriate expertise in health • Clarify consequences (payer burden, trade-offs in risk / benefit) • urCenogcnaenrredtcaintinigotniweswh?aittHhiosrwensedeeoadrwcehdecawonomdrmkwuahcnoricoisastscioosnltlae;kcwetihhnoogldhaeadrddcittooiomnbmaulruddneainttiaetso?reCsloalrvitey • aFancdemtoanfaucfaec,tturraenrsrpeaprreenstecnotamtimveusnipcraetfioenrawbliythbsetfaokreehroelldeears;einocf lruedseulptsatient • Resources used to defend a ‘no’ decision are more intensive because of • Explain importance of patient representative in interpreting results • Describe next steps to deal with uncertainty, pragmatic requirements to defend although uncertainties exist in both • ppCaoorpvteuicrlauatlgaioernlyws aitrhouevniddeisnscueesdeovfealopppmroepnrtia, taedndeistsionanaldccollalericftyioanpporfoepvriidaetence, positive and negative decisions. Additional Innovations in this Space Lessons from COVID-19 (If time permits) • Early bi-directional discussions across different stakeholder groups to • iCnodoupsetrrya)t,iotnhibnektinwgeeans astagkroeuhpoldaesras c(roengsuelqaurteonrcse, poafywerosr,krdeuserianrgchtehres,pHroTcAess,s ustnadkeerhsotaldnedr different perspectives and outcomes that are relevant for each instead of working in sequence • More predictability in the process • More tools are needed to communicate risk and benefit to general • Ncoenesdidaedreddit;iosntaalnddiasrpdlaizyastiaonnd summaries that more clearly show factors HTAs • Translate in reports what can be done about uncertainties that exist populations • AdeftsecrriHbiTnAg rheopwornt,ecwondtaintauearteo dtrimaciknisthheinegvuonlvcienrgtaeinvitdyence development,

Group 3 Friday February 26th

What key considerations should all HTA bodies have in mind as they describe, visualize, and explain the role that uncertainty played in a decision or recommendation? -Failure to get the uncertainty in context: what matters and what not? What drives the context or decision: what uncertainty needs to be solved and what is less important - Characterizing all uncertainty sometimes goes too far, characterization should be done better: decisions seems sometimes driven by budget and not by uncertainty - More balanced way of talking about uncertainty needed, not only negative also value of hope - Differences between agencies in how they handle and value uncertainty: acceptance of statistical methods is varying - Difference between regulatory and HTA agencies/payers: should be taken into account - More focus needed on managed entry agreements/coverage with evidence development, lessons can be learned from regulatory agencies. Think about most suitable agreement: not just price/volume agreements and try to harmonize them across agencies - When uncertainty is changing over time: new agreements or decisions needed. Incentives in place to reduce uncertainty, some examples available e.g. new data on overall survival - Show key drivers (2 or 3) of uncertainty in the final decision preferably visualize them: perspective not only of decision makers but also from patients and professionals - Important to visualize the uncertainty, still discussion how to make it understandable for everyone involved - CADTH: starting to connect with patients, what are they looking for and what does the data mean for them

Thank You 1-780-448-4881 | [email protected]| | htai.org Follow us on Twitter, Facebook, Linked In @HTAiorg

Group 4 Friday February 26th

Breakout Question: What key considerations should all HTA bodies have in mind as they describe, visualize, and explain the role that uncertainty played in a decision or recommendation? • Uncertainty is inherent – can’t eliminate, question is how much • Innovative ways to communicate – visualizations - infographic (ex. for uncertainty to address, different areas (Clinical outcomes, Parameter COVID vaccines), story telling, social media, decision trees inputs, Cost inputs) • Routes for communication, consider different channels (patients, public, etc). Social media may only reach limited • Prioritize through early dialogue what will have biggest impact to • Public panels, advisories groups – are there networks can that different stakeholders (payers, patients, etc). Look for aeras of disseminate information agreement across stakeholders. • Template for communicating decision informed by input from broad stakeholders– more of a focus now on how decision is • At scoping stage – where are key uncertainties and gaps, can made, not on decision and uncertainty around decision, why companies turn to stakeholders to fill gaps decisions differ across countries. • Disagreement about importance of uncertainty – ex. payers, • Considerations around communication patient community may not be aligned on the importance. • People think in terms of people – difficult to think in terms of Can there be a deliberative process in place to bridge the risk, statistics etc, how to translate concepts gap? • Testing communication – are we conveying what we think we’re conveying, get feedback on type of information that would be • Clearly communicated process for incorporating risk tolerance into most useful deliberation. There are different perceptions and tolerance levels – • Resource implications – adapt standard outputs, consistent understand risk preferences of those involved in deliberation. tools/tables etc that can be re-used. • Framework is difficult to develop and consistently apply, but • Public/patients – should be included in framing the value need to be clear about areas where more data can be assessment framework, everyone should understand the gathered, need flexibility framework at beginning. Decisions may vary due to contextual • Shift from uncertainty to risk factors, decision frameworks differ across countries • Need Consistency – information that is shared, actions • Uncertainties for full population, premium payers vs individual patients – what kind of decision is being made and who will info be communicated to

Group 5 Friday February 26th

Breakout Group 5 Facilitator: Nicole Mittman Today’s session focuses on HTA communication of uncertainty during the “output” phase – describing to stakeholders and the general public what effects uncertainties had on a decision or recommendation and any steps being taken to address them (e.g., managed entry agreements). We would like the discussion to focus on the following question only, based on the presentations you’ve seen today and your own experiences:

Breakout Group #5: W hat k ey considerations should all HTA bodies have in mind as they describe, visualize, and explain the role that uncertainty played in a decision or recommendation? 1. How patient perspective was considered and how weighed against uncertainty 2. How patient’s willingness to take on risk was acknowledged/considered 3. Patient-focused transparent communication regarding risks, harms, benefits – and in plain language 4. Awareness of perspectives of stakeholders such as researcher, manufacturer, governmental body and potential conflicting/aligned interests 5. Subsequent recommendations on how/what would help resolve those uncertainties for them 6. Opportunities for collaboration to resolve 7. Explain clearly to decision-makers who are frequently the target audience/adopters (for example: “More evidence” means….?) 8. Describe what were the uncertainties, how identified, how addressed through deliberative process in ways that all stakeholders may understand 9. What were the trade-offs, e.g. incremental v. transformational improvements

Group 6 Friday February 26th

Communication of uncertainty during the “output” phase-Group 6 • Need to engage patients and other end users on issues of uncertainty and include/describe input in final recommendation • Need to communicate not just the science but how deliberations on recommendation/decisions influenced by uncertainty • Explain why recommendations/decisions differ by country • Need to consider specific audience in communicating uncertainty; need lay language for patients and public • Need to communicate opportunity cost of a decision • Communicate all clinical evidence supporting recommendations • Explain interaction of multiple uncertainties on decisions