2021 Virtual GPF Slides

Uncertainty as a feature of HTA “Considering uncertainty is a fundamental and inherent component of HTA” – HTAi background paper • There are different degrees of uncertainty for different technologies and disease areas • Developers need to consider several ‘input’ uncertainties when designing studies and preparing dossiers for submission • Clinical uncertainty: • Study design: single-arm vs Randomized Controlled Trial • Variability from person-to-person, and different outcomes in subgroups • Effects in a clinical trial vs real-world setting, e.g. positive and negative externalities and study length • Chosen comparator may not reflect standard of care in the country assessing the technology, leading to uncertainty around relative effectiveness • Choice of surrogate endpoints 51

The case of OMPs • To accommodate uncertainty, value assessment and pricing and reimbursement decisions for OMPs should be adaptive subject to the need and availability of information over time. • TRUST4RD is an approach developed with Orphan Medicinal Products (OMPs) in mind, which are associated with higher input uncertainty • Developed through multi-stakeholder dialogue • Aims to identify and discuss appropriate approaches to resolve uncertainties by developing an iterative and informed dialogue 52

From TRUST4RD to RWE4Decisions • TRUST4RD is being followed up by RWE4Decisions, which is a multi- stakeholder initiative that acknowledges that collection of real world evidence is a shared responsibility, to inform decisions for highly innovative technologies. • The initiative brings together HTA authorities, payers, regulators, patient representatives, researchers/clinicians, academics and industry (EUCOPE is a participant). • Rationale: Real world evidence can play a critical role in assessing transformative innovative treatments that come to market via expedited regulatory approvals • Establishes a learning network on RWE involving all stakeholders, to realize the potential of RWE to inform HTA decision-making 53

Multi-stakeholder participation Wider Thought leadership stakeholder (INAMI-RIZIV CEO Jo De Cock) community Multi-stakeholder Steering Group HTA/payer Public research community bodies HTAs Patients KCE, FIMEA, NICE EURORDIS, EPF, ECPC G-BA, AIFA, HAS, Clinicians: EBMT, INFARMED, NICE, relevant ERNs Academic Industry University of Edinburgh EUCOPE & companies FIMEA Relevant (Connection to EURORDIS Clinicians/Researchers FINOSE), ZIN, KCE, members EORTC, ERNs NoMA, NCPA, TLV, AOK, Slovenian EMA Supported by RWE4Decisions Secretariat health insurers, Austrian insurers

The case of ATMPs • Offer groundbreaking new opportunities for the treatment of disease and injury • ATMPs such as gene and cell therapies tend to use single arm trial design due to ethical considerations and inability to blind investigators and participants to treatment allocation. • Some therapies can change the life trajectory of patients and carers, however many HTA processes do not account for these externalities (input uncertainty). • Uncertainty could be addressed through managed entry agreements and additional data collection and reassessment after 5-10 years 55

A joint responsibility to manage uncertainties • Developers are asked to address a number of input uncertainties as part of the HTA submission • Industry has a responsibility to seek to enter into early dialogue with HTA bodies, in order to address uncertainties that could arise during the HTA process (e.g. early scientific advice, EMA) • HTA bodies have a responsibility to offer opportunities for dialogue and to consider iterative appraisal processes with greater acceptance of real world evidence (RWE). • This is particularly the case for OMPs and ATMPs, but uncertainties in general can effectively be addressed through iterative dialogue • Such approaches could be structured according to the TRUST4RD or RWE4Decisions projects. 56

Thank you 57

Plenary Discussion

Plenary Discussion • How can the HTA community better prepare for increased uncertainty from accelerated approvals, specialized technologies, etc.? • What are the key tradeoffs in resolving uncertainty for HTA? • Are there steps that can be taken to increase alignment on uncertainty between HTA and regulators? And industry?

What’s Next? • Consider how HTA can best: – Present key uncertainties to deliberators and stakeholders – Fully integrate uncertainty into discussion and debate • Networking and cocktail-making! – Option 1: Today, 1400-1530 MST (2200-2330 CET) – Option 2: 24 Feb, 0300-0430 MST (1100-1230 CET) • Day 2 – 24 Feb, 0700-0930 MST (1500-1730 CET)

Thank You 1-780-448-4881 | [email protected]| | htai.org Follow us on Twitter, Facebook, Linked In @HTAiorg

18th HTAi Global Policy Forum Considering and Communicating Uncertainty in HTA 22, 24, and 26 February 2021 Virtual

Managing Uncertainty Wednesday, February 24

Ground rules/housekeeping • Chatham House Rule: - Participants are free to use the information received, but neither the identity nor the affiliation of the speaker(s), nor that of any other participant, may be publicly revealed - This includes social media! - Presentations, Zoom recordings, etc. can be shared within member organizations

Ground rules/housekeeping - Please change your name so it shows your first & last name + affiliation by referring to the “participants” tab in the Zoom tool bar - Please change your view to “speaker view” in the top right had corner of your Zoom screen - Attendee microphones and cameras will be turned off while speakers are presenting - We need to be vigilant about keeping to our allotted time. We apologize in advance if we need to interrupt comments or presentations to move on to the next agenda item!

Ground rules/housekeeping OR - If you have a question, please use the “raise hand” feature in Zoom and you will be called on in the order in which you’ve raised your hand - You may also use the “chat” feature to ask questions or make comments, we will also refer to the chat periodically

GPF Day 1 Recap • Expansion of early advice, solving the “firewall”, making dialogue iterative and lifecycle-oriented • Uncertainty and opportunity costs (for both positive and negative decisions) • Risk tolerance and acceptability vary by setting • Uncertainty “discount” vs. certainty “bonus” • Cross-country collaboration for data collection • Communication “wins” now? Investment in more resources for HTA “management” later?

GPF Day 2 • Focus on the “throughput” phase of HTA • What are the key considerations all HTA bodies and stakeholders should have in mind when contemplating how uncertainty affects deliberations and decision-making? • In addition to data/evidence, what other uncertainties are present? From the patient perspective? From industry’s?

Agenda: Day 2 • Keynote recap and Q&A with Ann Single • HTA case studies – Helen Knight, NICE – Andrew Mitchell, PBAC – Stefan Sauerland, IQWiG • Coffee Break • Breakout discussions (40 min) • Report back & plenary discussion (40 min)

Summary of Information for Patients Uncertainty at throughput Ann Single, HTAi Patient & Citizen Involvement Interest Group Chair; Patient Voice Initiative (Australia)

What do we mean by patient involvement in HTA? Patient-based evidence from robust research into patients’ needs, preferences and experiences e.g. primary or secondary research, Qualitative Evidence Synthesis, patient preference studies, … Patient participation in the actual process of HTA e.g. attending meetings, being consulted, submitting patient perspectives, … Facey K et al. (2010) ‘Patients’ perspectives in HTA: a route to robust evidence and fair deliberation.’ Int. J. Tech Assess Health Care, 2010, 334-340; Staniszewska S & Werkӧ (2017) ‘Patient-based evidence in HTA’ and Facey KM (2017) ‘Developing the mosaic of patient participation in HTA ’. In: Facey KM, Hansen HP, Single ANV, eds. Patient involvement in health technology assessment. Singapore: Springer Nature, 43-66.

Where Patients can Contribute to DecisionUncertainty (adaption of Facey et al. 2017 Box 37.1 summary of Menon et al. 2015) Clinical benefit uncertainty  Are benefits observed in trial generalizable to patient population within relevant jurisdiction?  Does level of health gain observed vary across patient subtypes?  Which outcomes should be measured?  What is natural progression of disease?  What is known about effect of drug compared to that of current best practice?  What health gain is meaningful to patients (is a clinically small outcome of some value to patients/carers)?  How patients view/ trade-off side effects/QoL Facey KM, Bertelsen N, Wale JL, Single ANV (2017) ‘Reflections for future development’. In: Facey KM, Hansen HP, Single ANV, eds. Patient involvement in health technology assessment. Singapore: Springer Nature, 419-428; Menon D, Stafinski T,Dunn A, et al. (2015) Involving patients in reducing decision uncertainties around orphan and ultra-orphan drugs: a rare opportunity? Patient 8, 29–39.

Where Patients can Contribute to DecisionUncertainty (adaption of Facey et al. 2017 Box 37.1 summary of Menon et al. 2015) Value for money (economic) uncertainty  What are broader implications associated with drug, beyond clinical benefit (including non- health benefits)?  What opportunity costs are associated with funding drug?  What is known about society’s willingness to pay for expected gain? Affordability uncertainty  How many patients are expected to benefit from drug?  What is expected cost per patient per year? Facey KM, Bertelsen N, Wale JL, Single ANV (2017) ‘Reflections for future development’. In: Facey KM, Hansen HP, Single ANV, eds. Patient involvement in health technology assessment. Singapore: Springer Nature, 419-428 and Menon D, Stafinski T, Dunn A, et al. (2015) Involving patients in reducing decision uncertainties around orphan and ultra-orphan drugs: a rare opportunity? Patient 8, 29–39.

Where Patients can Contribute to DecisionUncertainty (adaption of Facey et al. 2017 Box 37.1 summary of Menon et al. 2015) Adoption/diffusion uncertainty  How will access to drug be managed?  Who has expertise to decide on starting and stopping criteria?  Are mechanisms compelling patients and physicians to ensure appropriate use required?  Are there other drugs in the pipeline that may affect utilisation of this drug in near future? Facey KM, Bertelsen N, Wale JL, Single ANV (2017) ‘Reflections for future development’. In: Facey KM, Hansen HP, Single ANV, eds. Patient involvement in health technology assessment. Singapore: Springer Nature, 419-428 and Menon D, Stafinski T, Dunn A, et al. (2015) Involving patients in reducing decision uncertainties around orphan and ultra-orphan drugs: a rare opportunity? Patient 8, 29–39.

Meaningful patient engagement provides information unavailable from other sources . . . a misalignment of expectations detracts from meaningful engagement Rozmovitz L, Mai H, Chambers A, et al. (2018) What does meaningful look like? A qualitative study of patient engagement at the pan-Canadian Oncology Drug Review: perspectives of reviewers and payers. J Health Serv Res Policy 23, 72-79. … there is a significant gulf between patients’ and consumers’ expectations of the HTA process and what decision makers want to know from these groups Wortley S, Wale J, et al. Moving beyond the rhetoric of patient input in health technology assessment deliberations. Aust Health Rev 2017; 41: 170–172. See also Berglas S, Jutai L, MacKean G, et al. (2016) Patients’ perspectives can be integrated in health technology assessments: An exploratory analysis of CADTH Common Drug Review. Res Involve Engag 2,21 for exploratory analysis re integration of patient insights at CADTH, including additional information regarding progress of recovery and sustainability of health and focus of assessments.

Uncertainties about HTA for patients and their groups How their input contributes to determining the value What they know that HTA body/ committee doesn’t know What evidence HTA body has and assumptions it’s based on What is being assessed and who it is for Who HTA body is and consequences of its work

Scottish Sponsor completes Medicines Summary template Consortium Approach SMC check it and give to patient group(s) Patient group uses it to target input Scottish Medicines Consortium (2017) Guidance to manufacturers for completion of summary information for submitting patient groups form https://www.scottishmedicines.org.uk/media/2771/guidance-on-summary-information-for-patient-groups.pdf (accessed 30 January 2021)

PCIG Project Sub-Committee: International Template Summary of Information for Patients Co-leads Nigel Cook/ Susan Vallow (Novartis) Heidi Livingstone (NICE) Members Jennifer Dickson (Scottish Medicines Consortium) Sally Wortley (Office of HTA, Australia) Maria Jose Vincente (Aragon/IACS) Kate Morgan (Myeloma Patients Europe) Elisabeth Oehrlein (National Health Council, USA) Franz Waibel (Consultant, CH) Martin Coombes (BMS, UK) Barry Liden (Edwards Lifesciences, US) See https://htai.org/interest-groups/pcig/projects/current-projects/

PCIG Project Sub-Committee: International Template Summary of Information for Patients Content areas • Name of medicine • Administration & dosing • Authorisation • Efficacy • Company/HTA body contact • Quality of life impact/patient person preference information • Disclosures • Safety & side effects • Condition • Current clinical trials • Diagnosis of condition • Summary of key benefits • Current treatment options • Value & economic considerations • Patient-based evidence • Further information • How medicine works • Glossary of terms • Combinations • References See https://htai.org/interest-groups/pcig/projects/current-projects/

NICE adaptation and pilot Part of proposals for public consultation … a lay summary of industry’s submission to NICE, known as the 'Summary of Information for Patients' (SIP) … primarily to help them understand the evidence more easily to write their submission, … and provide a better understanding of the evidence for technical engagement and the committee meeting. The SIP is written by the company and included with the rest of their submission and sent on to the other participating stakeholder organisations by NICE and can be used to inform later submissions and statements. The company may also be asked to provide a SIP to support their response to technical engagement, or submission of additional evidence in response to additional consultations during guidance development. NICE (January 2021) Review of the health technology evaluation processes. Public board meeting agenda and papers. https://www.nice.org.uk/get- involved/meetings-in-public/public-board-meetings/agenda-papers-january-2021 (Accessed 3 February 2021)

Australian trial adaptation Pharmaceutical Benefits Advisory Committee Dr Sally Wortley, Consumer Evidence and Engagement Unit, Office of HTA, Dept. of Health Bristol Myers Squibb Australia Greg Cook, Director Access, Policy & Advocacy Hayley Andersen, Head of Patient Advocacy Nicole Patten, Head of Market Access Lara Donovan, Market Access Manager Consultant Ann Single

Australian trial adaptation  Same content boxes, reorganised for reduced text  Inbuilt sponsor guidance: ! Use diagrams (eg treatment options, dosing schedules) and tables (safety & side effects new vs comparator)  Prompts developing with trials  Value & economic considerations • any assumptions about quality of life used in the model • key costs/drivers used in any models, • any assumptions about the uptake of the medicine • any cost implications (positive or negative) for the health service (e.g. number of days in hospital) arising from implementing the new medicine • any other financial implications that may be of relevance to HCOs. Detailed health economic justification (e.g. cost/QALY) not required. Beginning this section with a sentence similar to the following will avoid the need to re-state the benefits already described: In addition to the benefits already described, the following elements informed the value and economic considerations . . .

How their input contributes to determining the value What patients know that HTA body/committee doesn’t know What evidence HTA body has and assumptions it’s based on What is being assessed and who it is for Who HTA body is and consequences of its work Sample of Australian trial supporting document for patient groups (CEEU,2020)

Preliminary observations from Australian trial Preparing summaries = iterative sense-making process of re-drafting, a dialogue teasing out what each knows that the other needs/wants to know between sponsor, HTA body and patient groups Meaningful is more important than simple, depth required Substantially reduced burden of preparation time for experienced group and altered its style of input supporting targeted information about less desirable/preferred side effects Potential to reduce misunderstandings/misalignment of expectations when used with wider communication support Potential to assist any time poor stakeholder (untested) Alternative formats (eg one-page infographic summary template) & individual consumer use

Informed patient involvement & uncertainty Known knowns Known unknowns Unknown unknowns Lack of balanced Balanced information Experience, capacity information for and support for building with balanced patients/groups patients/groups information and support Patient as mouth piece Patient as input to for patients/groups of industry? address HTA identified Patient as agent asking uncertainties? different question, seeking different evidence, identifying different uncertainties?

Thank You 1-780-448-4881 | [email protected]| | htai.org Follow us on Twitter, Facebook, Linked In @HTAiorg

Exploring uncertainty: reviewing the NICE methods of health technology evaluation Helen Knight Programme Director, NICE © NICE 2021. All rights reserved. Subject to Notice of rights.

Changes Topic This is a sample Healthcare and we are selection quote layout page… policy environment making to health Methods“Lorem ipsum dolor sit amet, technolog consectetur adipiscing elit. Sed y dapibus quamProecuemssaeusris evaluation sagittis dapibus. Donec at ultricies diam.” Name and job title here Programme manual

Uncertainty • A long-standing challenge, and growing • Earlier licensing, earlier evaluations, evidence limitations • Changing technologies: • precision, advanced treatments and treatments for rare diseases • Significant concern for stakeholders • Perception that uncertainty creates barriers – misalignment with regulators, delays, negative recommendations • Uncertainty = risk → to treated patients, displaced care and NHS Methods Understanding the uncertainty Describing, characterising, quantifying review Decision-making → How uncertain? In what way? In the context of that uncertainty → Should we recommend? 89

Types of uncertainty in current methods Uncertainty in choice of data source: Decision uncertainty: • Choice between different data sources • Probability committee • Uncertainty about quality of data makes the wrong decision source (e.g. trial methodology) (i.e. recommending a cost- ineffective technology or Parameter uncertainty: not recommending a cost- • Difference between observed value effective technology) used in cost-effectiveness analysis and ‘true’ value of the variable Structural uncertainty: • Uncertainty about assumptions made in the structure of the model (e.g. categorisation of health states, type of model) 90

Areas reviewed 1. Parameter uncertainty 6. Decision Key literature 2. Structural modifier: uncertainty uncertainty and risk Workshops Treasury guidance International HTA methods Academic experts Committee survey 3. Extrapolating 5. Presenting uncertainty beyond the evidence 4. Value of information analysis 91

• To ensure that uncertainty is fully and robustly characterised (both quantitatively and qualitatively) to clearly identify the nature, size and effect of uncertainties across evaluations Understanding • Key components:  the nature and effects of different types of uncertainty , characterising  whether it is captured in quantitative analyses  to what extent it may be resolved, reduced or mitigated by and presenting uncertainty additional evidence or expert input • The second stage of the methods review will give further details of how this assessment will be implemented 92

Understanding, characterising and presenting uncertainty – key highlights Parameter and structural uncertainty • Probabilistic analyses used as the starting point for economic analyses (committees’ preferred analyses and scenarios) • Enough simulations to minimise error and appropriate methods to account for ordering and correlation in parameters • Importance of plausibility of parameters and scenario analyses • Scenario analyses exploring effects of long-term benefit assumptions (possible threshold analysis) • Combined analysis of parameter and structural uncertainty (e.g. by parameterising structural uncertainty) is encouraged but not mandatory Value of information • Expected value of perfect information should be routinely presented – tool to understand likelihood and consequences of decision error • Further detailed work to understand its use in decision-making

Understanding, characterising and presenting uncertainty – key highlights Visualising and presenting uncertainty • Several options identified – looked at different industries • Net benefit approaches – identified from looking at ATMPs, and HICTs and subgroups • Net benefit rankings • Categorisations of uncertainty – inherent, resolvable in reasonable timeframe, unresolvable • Evidence generation over time – graphical presentation • More work needed but not necessary to restrict to a methods review • See next slide

Example of colour coded uncertainties 95

Example of colour coded tornado diagram 96

Example of expected evidence generation over time 97

Uncertainty • gCeonnedriatitoionns fisorcowmhipclhexitains dredciofgfincuislet,dstuhcahtaesvridareencdeiseases should remain an important  Limited evidence found that society values health benefits for rare consideration in diseases more highly…. decision making A greater degree  But important to ensure people with rare diseases are not of uncertainty disadvantaged and risk should be accepted in  Adopt a more accepting attitude to uncertainty defined circumstances… • Innovative technologies  Difficulties interpreting and applying current methods for innovation  No case for specific modifier – no evidence to show that society values health benefits from an innovative technology more….  Supporting innovation important to NICE – can provide valuable benefits to patients and society  Appropriate to accept greater risks for highly innovative technologies 98

Uncertainty • Technologies that provide large benefits should remain an important  Appropriate that greater degree of risk can be accepted when the consideration in potential benefit is higher decision making A greater degree  Further explore whether this includes large benefits to individuals, of uncertainty smaller benefits across a large population, or both and risk should be accepted in • When the uncertainty and risks can be monitored and defined controlled circumstances…  For example, through managed access arrangements • In all cases need to account for the nature, scale and consequences of uncertainty and risk • More work to define precise situations where greater uncertainty can be accepted  Understanding and definition of innovation; no overlap with size of benefits 99

Uncertainty • Regulatory changes – other non- • Launch of Innovative Licensing and Access Pathway (ILAP) from MHRA methods • Innovation Passport leading to Target Development Profile (TDP) - a approaches unique and tailored UK roadmap • Integrated pathway – expertise from across the MHRA , NICE, SMC, patients and companies • Process changes • Early identification of topics suitable/needing further evidence collection/managed access • Is full HTA needed at that point? Alternative approach? • Technical/patient/clinical engagement • Broadening of Cancer Drugs Fund to Innovative Medicines Fund • Commercial framework with NHS England and Improvement 100