2021 Virtual GPF Slides

What First methods consultation: Completed December happens next? 2020 consultationT: Lhiivseinsoaws! ample First process quote layout page… MethodsAndProcess @nice.org.uk Second stage: January - S“eLoprteemmipbsuemr 2do0l2or1sit amet, TChoenshuolwta:tiforonmoncausneiffioerdcmhaanngucsdeaoaagnpltisoi(tbJetuiucpstnsredo-qtaJugupurrailabaymmud) siepm.uiDsemcoinmnaeguacrenisaluitta. Sl ed - - ultricies diam.” Name and job title here Publication and implementation: Sept 2021 onward https://www.nice.org.uk/about/what-we-do/our- programmes/nice-guidance/chte-methods-consultation 101

Managing uncertainty in HTA HTA Case Studies: PBAC

Presentation outline Context Objective of approach Development of approach Initial trials of approach and assessment of its wider generalisability

Context Australia has two main HTA-based committees advising on listings for national funding programs in health • Pharmaceutical Benefits Advisory Committee (PBAC) • medicines for the Pharmaceutical Benefits Schedule (PBS) • vaccines for the National Immunisation Program (NIP) • Medical Services Advisory Committee (MSAC) • consultations, procedures, pathology, imaging, etc for the Medicare Benefits Schedule (MBS) • blood products for the National Blood Authority (NBA) • high cost therapies for the National Health Reform Agreement (NHRA) with the States and Territories • >100 committee considerations per year • advice is tailored to different characteristics of different funding programs • HTA advice encompasses a wide range of health technology types • persuasiveness of the supporting evidence has proven to be crucial

Objectives of standardised descriptors approach Initiated by PBAC • Need consistent approach to identifying the extent of influence of determinative factors • help inform the committee’s advice on listing a proposed health technology for funding • identify options for managing the implementation of any advice to list a health technology • Need consistent approach to communicating the extent of influence of determinative factors • enable applicants and the public understand the influence of these different factors in determining the committee’s advice for a proposed health technology • help monitor the consistency of decision-making across factors, across applications and across committees • Uncertainty is usually a determinative factor

Development of approach • Adapts the GRADE approach from the Cochrane Collaboration to HTA factors • List the 4 consistently repeating determinative factors 1. Status of the health technology (new type or niche population/ new option of same type or same population/ variant of an existing option) 2. Need for the health technology (high and urgent need/ moderate need/ limited additional need) 3. Overall benefit/harm (a matrix of 5 extents of benefit against 4 outcome types) 4. Certainty/Uncertainty (a matrix of 4 levels of confidence in relying on estimates of trial effects, modelled benefits, modelled costs, incremental cost-effectiveness ratio, and budgetary impact) • Define standardised categories for each determinative factor • Apply the standardised terminology for a proposed health technology • in documents prepared to inform the deliberations of the committee • in documents prepared to explain the advice of the committee

Standardised descriptors for confidence Level of Trial effects Modelled benefits Modelled costs ICER Budgetary confidence impact Summary estimates of Modelled estimates of Modelled estimates High benefits and harms in benefits and harms in of costs in economic Robustness of ICER Size of population key trial(s) economic evaluation evaluation results to sensitivity and risk of use Moderate Low likelihood that the All parameters make Cost offsets and scenario analyses outside that intended Low effect will be clinical sense, different considered robust Very low substantially less than modelling approaches and plausible Sensitivity analysis Patient population trial estimates provide consistent range <$5k (or % well defined, known outcomes Cost offsets highly equivalent) on key growth, little Very high likelihood that implausible factors likelihood of use the true effect is Clinical parameters outside that intended substantially less than incorrect trial estimates Sensitivity analysis High patient numbers range >$50k (or % with high likelihood equivalent) on key of use outside that factors intended

Trial of approach 1. Trial in commentaries prepared in response to applications to PBAC 2. Trial in minutes of PBAC deliberations and advice 3. Potentially trial in commentaries and assessment reports prepared for MSAC • further adaptation to encompass a wider range of health technology types and purposes • beyond therapeutic intent to also include investigative intent • the factors and their categories should be largely consistent This approach is still in its early stages • likely to evolve further during these trial periods

Thank You Any questions? 1-780-448-4881 | [email protected]| | htai.org Follow us on Twitter, Facebook, Linked In @HTAiorg

HTAi 2021 Virtual Global Policy Forum Virtual Meeting; 24 February 2021 (Day 2) Assessing and describing uncertainty – Case studies of IQWiG Stefan Sauerland, MD MPH Head of Department of Non-Drug Interventions

Role and selected tasks of IQWiG within German healthcare  Split of responsibilities  Drugs  HTA done by IQWiG (Cologne)  Early benefit assessment (dossier)  Decision-making done by G-BA (Berlin)  Standard assessment of drugs  Budget impact analysis (BIA)  Legal foundations of IQWiG (or health economic evaluation)  Independence (by law)  Financed from statutory health  Non-drug interventions insurance (SHI) levies  Assessment of in-hospital and/or  Cooperation with medical experts and out-of-hospital interventions involvement of affected persons  Assessment of „potential“  Transparent and evidence-based  Assessment of new high-risk medical methods device interventions

Different reasons for uncertainty and possible conclusions (1) Imprecision more less (6) Very large effect (2) Heterogeneity Uncertainty ? (7) Dose-response (3) Risk of bias gradient (4) Missing information (8) Opposite (5) Indirectness confounding Inconclusiveness Hint Indication (i.e. mo- Proof (or potential) (i.e. low certainty) derate certainty) (i.e. high certainy) https://www.iqwig.de/en/about-us/methods/methods-paper/ Schünemann H, Brożek J, Guyatt G, Oxman A: GRADE handbook (2013) https://gdt.gradepro.org/app/handbook/handbook.html

Reasons for uncertainty: Example No. 1 (1) Statistical imprecision g pp g( g ) Studie TM Kontrolle P: Patients with heart failure IN-TIME n/N OR (95%-KI) Gewichtung OR I: Dedicated telecardiology TELECART n/N 0.36 C: No telecardiology TIM-HF 8/333 0.62 O: Cardiovascular mortality TIM-HF2 3/89 21/331 12.8 0.86 40/354 5/94 4.3 0.65 39/765 38.7 46/356 44.2 59/773 0.10 0.32 1.00 3.16 10.00 TM besser Kontrolle besser Heterogenität: Q=3.28, df=3, p=0.351, I²=8.5%  Metaanalysis 1 (DerSimonian-Laird) significant => Moderate certainty  Metaanalysis 2 (Knapp-Hartung) non-significant => Inconclusive  Algorithmic qualitative synthesis confirmed effect => Low certainty Low certainty (after downgrading for imprecision)

Reasons for uncertainty: Example No. 2 (4) Missing information P: Patients with early prostate cancer I: Brachytherapy C: Prostatectomy, external radiation, or watchful waiting O: Survival (amongst many others)  1 RCT (with data on 5-y DFS for only 159 patients) => Data on survival insufficient  30 Non-RCTs (with confounder adjustment) => Short-term benefits (HRQoL) Inconclusive risk-benefit profile (because of missing or insufficient information on main outcome) https://www.iqwig.de/en/projects/n10-01.html

Reasons for uncertainty: Example No. 3 Hint of a positive effect (4) Missing information (downgraded by 1 level P: Patients with chronic wounds because of missing data with I: Negative pressure wound therapy C: Conventional wound therapy possible publication bias) O: Wound closure (amongst others)  30 RCTs (4315 patients) => significant effect => indication of a benefit  18 RCTs (with data for at least 1386 patients) missing Zens Y, et al. https://pubmed.ncbi.nlm.nih.gov/33038929/

Reasons for less uncertainty: Example No. 4 (6) Very large effect + (7) Dose-response gradient P: Babies with SMA (spinal muscular atrophy) I: Earlier treatment (after screening) C: Later treatment (without screening) O: Motor function Two indirect comparisons (of trial subgroups)  Therapy at symptom onset is better than after symptom onset => Low certainty  Therapy before symptom onset is better than at onset => (Very) low certainty Moderate certainty (after upgrading for very large effect size and time-effect-relationship) https://www.iqwig.de/en/projects/s18-02.html

Options for reducing uncertainty: Example No. 5 RCT initiated (and Coverage with evidence development (CED) financially supported) by Federal Joint Committee P: Retinitis pigmentosa I: Transcorneal electrical stimulation C: Standard of care O: Visual acuity  Application from manufacturer or user group  2 sham-controlled RCTs (24 and 63 pts.)  positive effect on visual field (1 RCT)  negative effect on colour vision (1 RCT)  no difference in visual acuity (2 RCTs)  Inconclusive risk-benefit profile, but “potential” h, ttps://www.iqwig.de/en/projects/e14-07.html https://www.g-ba.de/studien/tes-rp-studie/ Image source: https://www.okuvision.de/de/therapie/das-okustim-system/

Some insights and conclusions  Assessment of uncertainty is a key element of HTA, and there are several distinct, but additive sources of uncertainty.  Most aspects of uncertainty require some degree of judgment. HTA thus benefits from including the views of patients and clinicians.  In the first step, it is important to objectively describe uncertainty, regardless of the nature of the intervention (risks, costs, unmet need, etc.). In the second step, the reimbursement decision should made after considering all relevant criteria.

Institute for Quality and Efficiency in Health Care (IQWiG) Im Mediapark 8 D-50670 Cologne Phone +49 221 35685-0 Fax +49 221 35685-1 [email protected] www.iqwig.de www.gesundheitsinformation.de www.themencheck-medizin.de Twitter: @iqwig and @iqwig_gi XX.XX.20XX Autor / Titel 119

Breakout Discussions

Breakout Questions 1. What are the approaches that HTA bodies can use to best present and convey uncertainty to HTA committees during deliberation? 2. What steps can patients, manufacturers, and other external stakeholders take to explain and provide context around uncertainties in the data and potential uses of a new technology to better equip HTA committees for deliberation?

Breakout Session Group 1 February 24, 2021

Breakout Questions 1. What are the approaches that HTA bodies can use to best present and convey uncertainty to HTA committees during deliberation? 2. What steps can patients, manufacturers, and other external stakeholders take to explain and provide context around uncertainties in the data and potential uses of a new technology to better equip HTA committees for deliberation?

Discussion Notes 1. What are the approaches that HTA bodies can use to best present and convey uncertainty to HTA committees during deliberation?  Discussed approaches from Germany and Australia mainly  The deliberative process in Australia is good as the feedback goes directly back into the committee. However, it is more qualitative, but perhaps it could be quantified more. It feeds into the process/deliberation but is not the definitive answer.  Important to distinguish clinical uncertainty versus health economic uncertainty. HTA bodies need to be clearer about how large the uncertainty is in each of these areas and the impact on outcome  It may be better for the appraisal committees to request areas of uncertainty instead of them being proposed by the agency  Patients would appreciate predictability, but the process should also still be flexible across disease states  Transparency and collaboration are important to broader stakeholders like patients and industry  MEAs are complex and hard to manage, but you get good quality data. It does help to reduce uncertainty over the long-term, but usually only for a specific population.

Discussion Notes 2. What steps can patients, manufacturers, and other external stakeholders take to explain and provide context around uncertainties in the data and potential uses of a new technology to better equip HTA committees for deliberation?  One of the demands on patients is to resolve uncertainty which has been known about for a long time with data, but they need timeframes and clear questions  Clinicians could be more involved and could do interviews or participate similar to patients  Patient groups should have a specified way to insert their perspective, and it should be an integrated role in the discussion  Increased documentation (both clinical and health economic) will aid in decreasing uncertainty  QoL and caregivers are important to patients and those discussions enrich the process (need long-term engagement and potentially outside of just the HTA decision-making)  Some attempts to derive information from social media to gather patient insights are ongoing, but rigor is an issue  In the US, the National Health Council has launched core health sets to better understand diseases impact on patients

Breakout Group 2

What are the approaches that HTA bodies can use to best present and convey uncertainty to HTA committees during deliberation? - Scoring system or mechanism to assess the patient input - Having a mediator, if not available then find a good translation and stick to the wording (using lay language) - Differentiate between different types of uncertainty and how they can be solved (more research, better advice)  which stage to do this? - Measure the level of uncertainty and the potential impact of uncertainty - Adaptation of the HTA framework for certain health technologies (e.g., rare diseases or genetic therapies)  looking at common characteristics instead of certain types of health technology? (to be inclusive with precision oncology and other technologies) - To extend the options for advice from “yes or no” to make them more future proof

What steps can patients, manufacturers, and other external stakeholders take to explain and provide context around uncertainties in the data and potential uses of a new technology to better equip HTA committees for deliberation? - Importance of early advice / early dialogue to get people together, more patients to get involved - To continue the collaboration in the post-approval settings - Patient stories for the right decision context backed up with data - Scoping meeting with medical professionals and patients before the appraisal to get a feel in what the disease really entails - Start the conversation about uncertainties when designing the clinical trial

Thank You 1-780-448-4881 | [email protected]| | htai.org Follow us on Twitter, Facebook, Linked In @HTAiorg

Breakout Group 3 Uncertainty Throughput Phase

Q1. What are the approaches that HTA bodies can use to best present and convey uncertainty to HTA committees during deliberation? • Presenting: • Consider matrix option (e.g. Australian) cover page with opportunity to zero in • Uncertainty as part of a life cycle story • Evidence uncertainty and decision uncertainty • Visual representation of categories (red, yellow, green) • Components of uncertainty • Health Literacy • Conveying: • Education and Awareness • Committee / Decision Maker - Gains and / or losses in health (net health benefit) as well as uncertainty linked to decision • In a similar way across bodies, or at least by layer (clinical evidence)

Q2. What steps can patients, manufacturers, and other external stakeholders take to explain and provide context around uncertainties in the data and potential uses of a new technology to better equip HTA committees for deliberation? • Acknowledgement: evidence and corresponding uncertainty emphasis may differ by HTA (different emphasis may result in different decisions) • Patients and Manufacturers: engage early and identify key areas of evidence (and gaps) • Patients and Clinical Experts: rapid evidence design and generation via survey or other approaches within key gaps

Thank You 1-780-448-4881 | [email protected]| | htai.org Follow us on Twitter, Facebook, Linked In @HTAiorg

HTA management of uncertainty during the “throughput” phase– deliberating on the evidence and arriving at a decision or recommendation Breakout Group 4 Facilitator: Seth Clancy

What are the approaches that What steps can patients, HTA bodies can use to best manufacturers, and other external present and convey uncertainty stakeholders take to explain and to HTA committees during provide context around uncertainties deliberation? in the data and potential uses of a new technology to better equip HTA • Description uncertainties (semi-quantitative approach); using committees for deliberation? a common framework for each type of uncertainty (clinical, economic); • Presentation of patient preference tools and how those are integrated in the assessment • Description of the reasons and context around uncertainties • Prioritizations of uncertainties • Involve patients earlier in the HTA at the time of • Zero level of uncertainty does not exist. Regulators has good scoping the skeleton of the assessment. Explain to patient level of evidence existing, outcomes and methods for trade-off, unmet needs need to be balance with uncertainties uncertainty • Time-horizon about uncertainties, tolerance for promising • Patients preferences and Values help committees technologies with positive short- term outcomes • Using specific regional researches on PRO and • Communication: not only about sharing uncertainties but how it is interpreted by different stakeholders (patients, HCP, recognized that those are not perfect but still relevant etc) to understand context, trade-off and evidence gaps • Simple language, communication, summaries, more • Context implementation framework for new therapies visualization tools, patients oriented that can help complement the assessment

Breakout Group 5 Facilitator: Adrian Griffin Rapporteur: Amadou Diarra

Discussion Notes – Group 5 • Building trust by communicating systematically uncertainty (Transparency) • Small effects can make a difference to patients. Look at it in a disaggregate way. Resolvable vs Not? • Explaining/providing context about uncertainty to patients (MNF, HTA sides) • Uncertainty there to stay. Manage it vs fight it. Manage entry agreements can help reduce risks. • Critical role of stakeholders to put uncertainty in context, need to move from technical to judgement steps • Communicating uncertainty, using “confidence”, Yes or No vs grey (criteria for funding). HTA cannot be just for perfect data. Solutions: what type of risks can be taken in terms of patients/affordability? Learning system. • Arguments to allow more uncertainty in some situations. Take a range to consider more opinions. Devil in details. • Acceptability to patients (re trust). Same treatment not same effects. Distrust. Transparency vs statistical literacy. • Detect uncertainty, find other methodologies. Stakeholders vs risk: what does it mean? Individual vs collective. • Confidence: Sometimes more certain about answers than evidence supports. Need for timely decisions. • Need to translate Technical decisions in plain language for stakeholders. • Patient Outcomes Vs Uncertainty • Uncertainty vs diversity of patient population

Breakout Session Group 6 February 24, 2021

Managing uncertainty in HTA - “throughput” phase Clarify the problem statement (what are we trying to solve when talking about “uncertainty”?) • Strong focus on “technical” aspects, modelling etc.; BUT: process related/context specific issues matter greatly (e.g. committee composition) • Distinguish uncertainty in evidence itself and in how uncertainty is dealt with in the deliberation • Disentangle the different dimensions of uncertainty (What do we mean when using the term “uncertainty”; clarify!) • Implications of uncertainty very context specific (-> tendency to jump into details) • Storytelling / Ensure we keep the “big picture” and do not get lost in technicalities • Use qualitative evidence? Patient involvement • More genuine participation would help • Issue: knowledge translation (e.g. find common language, translate “expert discussions” to lay language) • Novel technologies (-> limited evidence “by nature”) pose particular communication challenges

Managing uncertainty in HTA - “throughput” phase Opportunities / What can be done / Steps forward • Timing matters: when is technology assessed; which uncertainties can be reduced and when? • And how is patient access handled “during period of uncertainty” (e.g. GER approach: access during HTA process, w price/reimbursement adjustments later on) • Science: can handle some types of uncertainty. But “price” is more complex (not only “science” driven) and relates also to trust relationship between supplier and payer (HTA bridges). • Different types of value / Value dimensions • Better adherence to appropriate methods (and good-practices) • Some inconsistency (conservatism?) wrt novel methods • E.g. cure-rate modelling, surrogacy, … • Can we learn (more) from other settings (how do other, non-HTA settings, manage uncertainty)? • Other industries? • Portfolio decisions?

Plenary Discussion

What’s Next? • Communicating the role uncertainty played in decision or recommendation: – Does this differ by stakeholder type? • Are you aware of any current innovations? • Are there lessons learned from COVID-19 that can be applied? • Day 3 – 26 Feb, 0700-0930 MST (1500-1730 CET)

Thank You 1-780-448-4881 | [email protected]| | htai.org Follow us on Twitter, Facebook, Linked In @HTAiorg

18th HTAi Global Policy Forum Considering and Communicating Uncertainty in HTA 22, 24, and 26 February 2021 Virtual

Communicating Uncertainty Friday, February 26

Ground rules/housekeeping • Chatham House Rule: - Participants are free to use the information received, but neither the identity nor the affiliation of the speaker(s), nor that of any other participant, may be publicly revealed - This includes social media! - Presentations, Zoom recordings, etc. can be shared within member organizations

Ground rules/housekeeping - Please change your name so it shows your first & last name + affiliation by referring to the “participants” tab in the Zoom tool bar - Please change your view to “speaker view” in the top right had corner of your Zoom screen - Attendee microphones and cameras will be turned off while speakers are presenting - We need to be vigilant about keeping to our allotted time. We apologize in advance if we need to interrupt comments or presentations to move on to the next agenda item!

Ground rules/housekeeping OR - If you have a question, please use the “raise hand” feature in Zoom and you will be called on in the order in which you’ve raised your hand - You may also use the “chat” feature to ask questions or make comments, we will also refer to the chat periodically

Sli.do You are welcome to open slido for a “meeting summary” poll Visit sli.do or www.slido.com (best to do on your mobile device) Enter the event code #2021GPF Poll title: “The 3 Most Important Changes to HTA Consideration and Communication of Uncertainty in Your View”

GPF Day 2 Recap • Standardized documentation of patient experience, interpretation of Rx effects, info on unintended consequences may reduce uncertainty and benefit HTA deliberations • Acknowledging uncertainty for patients must be an “investment” to avoid distrust • HTA case studies: – Process and methods changes – Visual aids / color coding – Timing of new evidence – Accelerated/flexible pathways – Standardized categorization of determinative factors – Matrix for level of confidence in results for deliberators and external stakeholders – HTA-generated studies to mitigate uncertainty – Explicit involvement of patients and clinicians