CQI Program Manual

CONTINUOUS QUALITY IMPROVEMENT PROGRAM MANUAL March 31, 2021



An electronic version of this document can be obtained by visiting the following website: http://www.nd.gov/dhs/info/pubs/family.html MISSION STATEMENT To provide quality, efficient, and effective human services, which improve the lives of people. North Dakota Department of Human Services Children and Family Services Division 600 E. Boulevard Avenue, Dept. 325 Bismarck, ND 58505-0250 Phone: (701) 328-2316 Toll Free: (800) 245-3736 Relay ND TTY: 711 Fax: (701) 328-3538 Email: [email protected] Website: www.nd.gov/dhs Facebook: @NDHumanServices Twitter: @NDHumanServices

Continuous Quality Improvement Program Manual TABLE OF CONTENTS 4 4 Section I: Introduction 5 • Overview of CQI 6 • Purpose and Expectations for Use 6 7 Section II: CQI Vision for North Dakota Child Welfare System 8 9 • History of North Dakota Child Welfare CQI Program 9 • Core Values of North Dakota CQI Model 10 • Strategic Priorities for North Dakota Child Welfare System • Role of CQI in Meeting Strategic Priorities and Achieving Positive Outcomes 11 o CQI Cycle of Learning and Theory of Constraints 11 o Key Performance Indicators 11 12 Section III: CQI Teaming Structure 12 13 • Purpose 13 • Overview of North Dakota CQI Team Levels 14 15 o State CQI Council 15  Purpose 16  Membership 16  Establishing a Team Charter 16  Team Lead Responsibilities 17  Meeting Frequency 17  Meeting Agenda 18  Application of CQI Cycle of Learning (link to Section 6) 18  Communication of CQI Activities 18  State CQI Ad hoc Sub-groups 18 18 o Cross-Zonal CQI Teams 18  Purpose 19  Membership 19  Establishing a Team Charter 19  Team Lead Responsibilities 19  Meeting Frequency  Agenda  Application of CQI Cycle of Learning  Communication of CQI Activities o State CQI Support Team  Purpose and Membership  Responsibilities March 31, 2021 Page 1

Continuous Quality Improvement Program Manual Section IV: Staff and Stakeholder Participation in CQI Activities 20 • Purpose 20 • CQI Roles for Staff and Stakeholders 20 Section V: Training and Resources to Support CQI Activities 24 • Purpose 24 • CQI Training Overview 24 • Foundation Training for North Dakota CQI Members 24 • CQI Training Curriculum Outline 24 • Case Review Staff Training and Certification 27 • Additional CQI Resources 28 Section VI: CQI Cycle and Theory of Constraints 29 • Purpose 29 • Overview of CQI Cycle 29 • Overview of Theory of Constraints 30 • Integration of CQI Cycle and Theory of Constraints 30 • When to Apply 30 • Implementing Six Steps in CQI Cycle 31 31 1. Identify and Understand the Problem 32 2. Research the Solution 33 3. Develop a Theory of Change 34 4. Adapt or Develop the Solution implementation 35 5. Implement the Solution 35 6. Monitor and Assess the Solution 37 Section VII: Data Collection and Dissemination 37 • Purpose 37 • Data Pathway for North Dakota Child Welfare 37 • Data Entry and Extraction 38 • Data Sources 39 • Data Reports 41 • Tips for Interpreting Data 41 • Data Quality 42 • Methods to Ensure Data Quality 42 • Data Integrity 43 • Case Record Review Data 43 • Analysis and Dissemination of Data March 31, 2021 Page 2

Continuous Quality Improvement Program Manual Section VIII: Case Record Review 44 • Purpose 44 • Principles of North Dakota Case Review Process 44 • Steps in Case Review Process 45 • Workforce Structure for Implementing Case Review 46 • Competencies for Case Review Workforce Members 47 • Case Review Resources 47 • Sampling for Case Review 47 • Case Review Preparations 48 • Completing the Case Review Process 48 • Resources and Use of Onsite Review Instrument 48 • Ensuring Data Quality of Case Review Results 50 • Dissemination and Use of Case Review Data as Part of CQI Activities 53 Section IX: Key Performance Indicators for CQI 55 • Purpose 55 • Connection of Key Performance Indicators to North Dakota Strategic Priorities 56 • Data Reporting 56 • Table of Key Performance Indicators 57 Appendix: (under development) 59 • CQI Scenarios 60 • CQI Action Plan Worksheet 62 • Status Report 69 March 31, 2021 Page 3

Continuous Quality Improvement Program Manual I. INTRODUCTION Overview of CQI Continuous quality improvement (CQI) is a comprehensive continuous learning process implemented by an organization to identify, describe, and analyze strengths and problems and then test, implement, learn from, and revise solutions. CQI is: • A Philosophy, as it places a high value on improvement • A System, as it is a coherent set of components that supports the continuous improvement process Key features of a CQI system are: • It is data-informed, systematic, and proactive • It is inclusive, involving stakeholders and staff at all levels • It is holistic, incorporating information about multiple aspects of the system and establishing linkages It is essential for any child welfare agency to have a well-functioning CQI system to accomplish its key goals which is to ensure the safety, permanency, and well-being of children and promote the resiliency of families. In the field of child welfare, CQI is a process for: • Evaluating the quality of work with children, youth, and families • Measuring outcomes of the work with children and families (safety, permanency, wellbeing) • Gathering information to guide program improvement efforts, and reinforces best practices in child welfare The Children’s Bureau in its 2012 Information Memorandum suggests that there are five components that are essential to a well-functioning CQI system. These include: o Administrative Structure o Quality Data Collection o Method for On-going Case Reviews o Process for Analysis and Dissemination of Quality Data o Process for Providing Feedback to Stakeholders and Adjusting Programs and Processes as Needed Additionally, an organization’s culture and climate are critical to developing and maintaining a CQI system that can monitor and promote the agency’s outcomes. This includes the essential role that agency leadership plays in modeling CQI practice and supporting CQI activities, the meaningful engagement of staff and stakeholders, and communication practices that clearly articulates and messages CQI processes and goals as well their connection to the agency’s mission, vision, and guiding principles at all levels of the agency and with key stakeholder groups. March 31, 2021 Page 4

Continuous Quality Improvement Program Manual Purpose and Expectations for Use of Manual: This manual is intended for use by child welfare agency staff, system partners, and stakeholders and for anyone who wants or needs to understand how to participate in North Dakota’s CQI process and activities. This includes but is not limited to: • All child welfare staff, across all program areas and at all levels, inclusive of State agency leadership, Regional/Zonal management, caseworkers and supervisors, IT and Data staff, and CQI and Case review staff • All child welfare partners, including: o Public and private providers o Courts and the Division of Juvenile Services o Tribes o Youth, parents, caregivers – those with lived child welfare experience Orientation to this manual will be provided as part of the child welfare certification process for agency staff and core components will be integrated into new and ongoing staff training. Key elements of the manual will also be highlighted as part of ongoing orientations to the North Dakota CQI process with child welfare partners and key stakeholders groups. An electronic version of this document will be available once final approval has been received by the federal Administration for Children and Families. System partners will be able to access the document at http://www.nd.gov//DHS/info/pubs/family.html. Employees within the North Dakota Department of Human Services will have access through the DHS Insider at https://ndgov.sharepoint.com/sites/DHS- ChildrenAndFamilyServices. For additional information regarding the Continuous Quality Improvement program, please contact: CQI Administrator North Dakota Department of Human Services Children and Family Services Division 600 E. Boulevard Avenue, Dept. 325 Bismarck, ND 58505-0250 Phone: (701) 328-2316 Email: [email protected] March 31, 2021 Page 5

Continuous Quality Improvement Program Manual II. CQI VISION FOR NORTH DAKOTA’S CHILD WELFARE SYSTEM History Of North Dakota’s Child Welfare CQI Program North Dakota’s efforts to create a continuous quality improvement program began with the federal Administration for Children and Families’ issuance of the August 27, 2012 Informational Memorandum. The Children and Family Services Division went through reorganization that Fall and a position was created for further evaluation and development of the state’s CQI system. During the Winter and Spring of 2013, an assessment of the state’s infrastructure and readiness to implement a CQI program was completed and the results shared with the Children’s Bureau in the 2014 Annual Progress and Services Report. Design and implementation of a fully functioning CQI program became a goal in the 2015-2019 Children and Family Services Plan. Throughout the planning process in 2014 and implementation for the 5-year plan, the Division and stakeholders focused on strengthening the quality assurance/case review process, improving data collection ability, and CQI training. In 2016, the state completed the 3rd Round Children and Family Services Review. Results continued to show the state struggling with the lack of a fully functioning CQI program. The Division drafted a program improvement plan that included implementing quality assurance and CQI programs. During the same time, technical assistance was requested to assist with these endeavors. In April 2017, with assistance from the Capacity Building Center for States, a Systems of Care Evaluation Team was established as a stakeholder-involved process for moving ahead with the development on a CQI process and to assist in developing strategies for addressing the CFSR Performance Improvement Plan using CQI principles. That stakeholders representing both private and public child welfare entities, including Division staff and those who assisted with the federal CFSR, were invited, and agreed to be part of this effort. This group went through the CQI Academy during Spring 2017 – Fall 2017. The Division began administering the child and family services reviews in 2003 and continuing through 2015. In 2017, the Division contracted with the Children and Family Services Training Center at the University of North Dakota to manage an Onsite Case Review process that was in place through 2018. In 2019, CFS created a new Quality Assurance Unit to monitor child welfare services, including conducting case reviews. The QA unit also supports continuous quality improvement efforts for the Division. In 2018 the Department adopted the Theory of Constraints (TOC) as its CQI process department wide. Theory of Constraints is a methodology for identifying the most important limiting factor (i.e., constraint) that stands in the way of achieving a goal and then systematically improving that constraint until it is no longer the limiting factor. The primary focus of Theory of Constraints is to identify the March 31, 2021 Page 6

Continuous Quality Improvement Program Manual constraints, believe there is hidden capacity, and apply the “rules of flow” to measure the work output and the quality of work. TOC has been, and will continue to be, applied to all child welfare programs so that holistic change can occur in each area of service. In 2020, Division engaged with the Capacity Building Center for States to further define and operationalize its CQI approach which led to the integration of key TOC principles and concepts into a more fully comprehensive CQI cycle of learning which reflects the current foundation and structure for North Dakota’s CQI model effective in 2021. Core Values of North Dakota CQI Model The continuous quality improvement process for the North Dakota Child Welfare System requires the involvement of all agency staff and its partners providing child welfare services, children, youth, families, and stakeholders to implement and sustain system transformation. The North Dakota Department of Human Services is committed to an organizational culture that understands the purpose and importance of collection and utilization of accurate data in the process of improving the services for children and families. It also needs the commitment at all levels of the organization to the continuous quality improvement philosophy for the process to be successful. With a commitment throughout the public child welfare system to continually improve services, all levels work as one for the continual improvement of the North Dakota Child Welfare System. Engaged: System staff and stakeholders at all levels must have a clear understanding of the North Dakota Department of Human Services child welfare vision and work continuously and collaboratively to achieve and sustain improvements in practice and outcomes. Data Informed: The success of the CQI process is dependent on a commitment by System leadership, staff, and stakeholders to the ongoing use of data and evidence to examine problems and support decision-making across the child welfare system. March 31, 2021 Page 7

Continuous Quality Improvement Program Manual Solution Focused: CQI activities must be routinely focused on identifying strategies that will best address or solve the underlying issues or constraints that are impacting system and agency performance in terms of case practice, service delivery and child and family outcomes. Strategies must be regularly monitored until the desired results are achieved and modifications made as needed. Strategic Priorities for the North Dakota Child Welfare System CQI activities will primarily focus on but are not limited to monitoring and facilitating the effective implementation of the following key Department strategic priorities (Note: This CQI manual, including strategic priorities, will be reviewed, and revised as needed in accordance with annual CFSP planning): 1. Successful completion and continuation of all outstanding Round 3 CFSR PIP strategies and related activities, and outcomes: a. Design and implement a Continuous Quality Improvement (CQI) process to identify the strengths and needs of the service delivery system; monitor and evaluate the system changes to positively impact outcomes for children and families. b. Ensure safety for children and well-being for children and families by improving caseworker’s skills and engaging the court to increase family engagement, thoroughly assessing, and addressing identified risk and safety factors and providing services quickly and effectively. c. Realign the service delivery system to engage and empower families earlier in the case to improve outcomes and inform practice. d. North Dakota will experience increased achievement of timely permanency. e. Strengthen and reframe the statewide foster and adoptive parent diligent recruitment plan to support the recruitment of families who meet the needs of the children they serve and who reflect the ethnic and racial diversity of children served by the foster care program. 2. Progress towards the achievement of ND 2020-2024 CFSP goals and objectives: a. Prevention: Utilize primary, secondary, and tertiary prevention strategies to address child abuse and neglect, promote family preservation, and divert children from foster care by supporting and engaging families early during service delivery. b. Timely Permanency: Increase achievement of timely permanency by engaging with the family to meet identified needs through individualized service plans, by implementing a formal level of care assessment to improve placement stability, and by supporting kinship caregivers. c. Legal Community Training and Education: Collaborate with the North Dakota Court Improvement Project (CIP) to support, engage, and educate the child welfare and legal communities to develop knowledge, skills, and expertise to support best practice. d. Quality Assurance: Design and implement a practice improvement process using Theory of Constraints (TOC) to identify the efficiencies and hidden capacity of the service delivery system, monitor and evaluate the system changes to positively impact outcomes for children and families. March 31, 2021 Page 8

Continuous Quality Improvement Program Manual 3. Focus on Organizational Policy Priorities: a. Reducing institutional placements by increasing family supports allowing children to remain safely in the home. b. Reducing out of state placements by increasing family supports allowing children to remain safely in the home. c. Increasing collaboration between agencies and systems who serve families affected by substance use disorders and involved in child welfare services. d. Increasing children and families receiving evidence-based supports to decrease the need for out-of-home care, allowing families to remain together. e. Reducing severity of school-based behaviors by partnering with educational and behavioral health partners to expand outpatient and community-based services; and enhance and streamline the system of care for children and youth. f. Successful implementation of CQI trained workforce to increase the understanding and use of quality/timely data to enhance service delivery. g. Enhancing involvement with family and youth voice in the CQI process. Role of CQI in Meeting Strategic Priorities and Achieving Positive Outcomes Through its commitment to an engaged, data informed, and solution focused continuous quality improvement process, North Dakota will be able to implement and sustain actions to address these and other system priorities. This will be accomplished through the effective utilization of: • CQI Cycle and Theory of Constraints: North Dakota has adopted both of these methodologies and intends to integrate them as the primary means to being able to systematically make decisions throughout the child welfare system that will lead to desirable change and improved outcomes for children and families. o The CQI Cycle includes six steps that begins with a team using data or other evidence to develop a deep understanding to the underlying or root causes to any problem, identifying and implementing an intervention that is the best possible fit for the problem, and then monitoring and adjusting the intervention as needed until the problem is resolved or desired outcome is achieved. o The Theory of Constraints is a methodology for identifying the most important limiting factor (i.e., constraint) that stands in the way of achieving a goal and then systematically improving that constraint until it is no longer the limiting factor. (https://www.leanproduction.com) The primary focus of both of these methodologies is for an organization to be able to achieve its goals. Both also encourage a team-based approach that includes input from those most impacted by the problem, and those who have the most influence over the problem. Though not identical, these two approaches are not at all contradictory. North Dakota therefore intends to integrate key concepts from Theory of Constraints, particularly as it relates to identifying and redesigning work practices that limit efficient progress towards agency goals, into the six-step CQI cycle and utilize it as the foundation for the state child welfare system’s CQI model. March 31, 2021 Page 9

Continuous Quality Improvement Program Manual • Key Performance Indicators (KPI’s) for North Dakota Child Welfare: The North Dakota CQI process will focus on the continual generation, use, and reinforcement around a set of key performance measures and outcomes that are in alignment with the agency’s vision, strategic priorities, and Federal requirements. Measures will be updated as needed and well as the establishment of performance baselines and benchmarks to be utilized as a primary driver for all Department CQI activities. March 31, 2021 Page 10

Continuous Quality Improvement Program Manual III. CQI TEAMING STRUCTURE Purpose: The North Dakota Child Welfare System CQI teaming structure provides a framework to regularly bring agency staff and stakeholders together and ensure a consistent application of CQI activities across the state that is in alignment with the state agency’s mission, vision, and strategic priorities. Overview of North Dakota CQI Teaming Levels The North Dakota Child Welfare System CQI Teaming structure will be composed of two primary team levels – the State CQI Council and a total of four Cross-Zonal CQI Teams representing each of the state’s Human Service Zones, Division of Juvenile Services, and Tribal partners from across the state. The CQI Support Team serves to support both the State CQI Council and Cross-Zonal CQI Teams in implementing CQI activities. Guides Supports Informs Agencies (i.e., DJS, Tribes, HSZ’s) will be assigned to one of four (4) Cross-Zonal CQI teams accordingly to their positioning with the state’s four (4) Juvenile Court Units. If an agency appears in more than one Judicial Unit, the HSZ will be assigned to the judicial unit aligned with the Zone’s administrative county. March 31, 2021 Page 11

Continuous Quality Improvement Program Manual Team 1: Turtle Mountain Child Welfare, Spirit Lake Social Services, DJS NE/LR offices, Northern Prairie HSZ, Mountain Lakes, Northern Valley, Grand Forks County HSZ, Eastern Plains Team 2: DJS SE/SC offices, Central Prairie HSZ, Buffalo Bridges HSZ, Agassiz Valley HSZ, Cass County HSZ, RSR HSZ, South Country HSZ Team 3: Standing Rock Social Services, DJS WC/SW offices, Burleigh County HSZ, Dakota Central HSZ, Three Rivers HSZ Roughrider HSZ, Southwest Dakota HSZ Team 4: MHA Nation, DJS NW/NC offices, North Star HSZ, Mountrail McKenzie HSZ, Ward County HSZ See the map below for additional information regarding Cross-Zonal Team alignment and composition. Cross Zonal CQI Team 4 Cross Zonal CQI Team 1 Divide Burke Renville Bottineau Rolette Towner Cavalier Pembina NORTH STAR HSZ NORTHERN NORTHERN PRAIRIE HSZ VALLEY HSZ Williams Mountrail MOUNTAIN Wals h LAKES HSZ Ward WARD HSZ McHenry Pierce Ramsey Nelson Grand Forks MOUNTRAIL - MCKENZIE Bens on EASTERN PLAINS HSZ McKenzie McLean Sheridan Wells Eddy GRAND FORKS HSZ Dunn CENTRAL PRAIRIE Griggs Steele Traill DAKOTA CENTRAL HSZ Fos ter AGASSIZ VALLEY HSZ Mercer Golden Billings Burleigh Kidder S tuts man Valley Oliver Barnes Cas s ROUGHRIDER NORTH HSZ BURLEIGH HSZ Morton BUFFALO BRIDGES HSZ CASS HSZ Stark Slope Hettinger Emmons Logan La Moure Rans om THREE RIVERS HSZ Richland SOUTHWEST HSZ Grant SOUTH COUNTRY HSZ RSR HSZ Bowman Adams Sioux McIntos h Dickey Sargent Cross Zonal CQI Team 3 Cross Zonal CQI Team 2 1. State CQI Council a. Purpose: The State CQI Council is the primary driver for North Dakota’s statewide CQI process and is comprised of a wide range of state-level agency staff and stakeholders. Members of the State CQI Council will come together on a quarterly basis, using data and other March 31, 2021 Page 12

Continuous Quality Improvement Program Manual sources of information, to bring to light and gain a better understanding as to what is working well and what is not in relation to core agency child welfare goals and strategic priorities. The State CQI Council will also work together to utilize the CQI Cycle and Theory of Constraints in order to identify and implement effective strategies and solutions that address areas in need of improvement and monitor and adjust strategies through the use of data as needed over time to ensure successful implementation. b. Membership: Membership of the State CQI Team is inclusive of agency staff across multiple program areas and key child welfare system partners, as well as representatives from each of the Cross-Zonal CQI Teams. Membership will represent the voices of youth and families and others with lived experience. Membership should include but is not limited to the following: • Representatives of the CFS Leadership Team:  CFS Director and Assistant Director  CQI Administrator and QA Manager  CPS Administrator  Foster Care Administrator  Well-being Administrator  CFS Field Service Specialist Supervisors • Decision Support Services Data Analysts • CFS Training Center Director • NATI (Native American Training Institute) Director • AASK (Adults Adopting Special Kids) Director • DJS Director of Community Operations • Children Behavioral Health Program Administrator • Adult Mental Health Program Administrator • Substance Abuse Program Administrator • Supreme Court representative (i.e., CIP Director) • Juvenile Court representative (i.e., Juvenile Court Directors) • Tribal Child Welfare Directors or designees • Cross-Zonal CQI Team Leads • Family and youth representatives c. Establishing a Team Charter: Members of the State CQI Council are encouraged to work together to develop a team charter. A team charter is a formal document that will provide the foundation and direction for the team’s work, particularly by: • Documenting the team’s purpose • Clarifying member’s roles, authority, and responsibilities March 31, 2021 Page 13

Continuous Quality Improvement Program Manual • Defining operating rules – i.e. How the group will work together in carrying out CQI activities • Establishing communication methods and reporting structures • Facilitating member buy-in around the decision-making process Potential elements to consider for inclusion in the team charter are: • Team mission statement • Goals and objectives for the team’s work • Expected deliverables • Defining a process for identifying and recruiting team members • Team consensus and decision-making • Member roles and responsibilities, such as: o Expectations regarding member attendance and participation o Preparation for meetings by reviewing meeting minutes, data, or other relevant materials o Providing timely feedback o Adhering to established deadlines • Scheduling of meetings, locations, and/or meeting formats (i.e., in-person vs. virtual) • Description of communication strategies and frequency of sharing information, both within the team and with Cross-Zonal CQI Teams, subcommittees, agency staff, and stakeholders • Strategies for managing conflict The team charter should be reviewed regularly and revised as needed to ensure it is current and represents the real time implementation of North Dakota’s CQI model and the role of the State CQI Council in guiding and supporting the process. d. State CQI Council Team Lead Responsibilities: State CQI Council co-leads have responsibilities that include but are not limited to the following: i. Agenda planning and meeting preparation: This includes establishing agenda items, identifying and notifying as appropriate members responsible for presenting data or sharing other information, arranging for the preparation of data reports, and disseminating the agenda and related data and other relevant information to team members in advance of CQI meetings. ii. Meeting facilitation: Team leads are responsible for setting a productive environment for CQI meetings by supporting and encouraging participation, helping the team to communicate effectively, and encouraging members to focus on problem resolution and achieving outcomes. iii. Meeting documentation: Team leads should ensure there is a scribe assigned to record meeting minutes, including documentation of attendees, agenda items, team discussion, identification of problems or action items, and resolution of action items as appropriate. March 31, 2021 Page 14

Continuous Quality Improvement Program Manual iv. Follow-up on action items: This includes ensuring that all action items identified by the State CQI Council as well as any issues raised by Cross-Zonal CQI teams are being effectively tracked and monitored as a part of CQI team activities towards resolution. e. Meeting frequency: The State CQI Council and Cross-Zonal CQI Teams are expected to meet on a quarterly basis. Though not required, state and Cross-Zonal teams are encouraged to consider utilizing a schedule that provides the opportunity for Cross-Zonal CQI Teams to meet in advance of the State CQI Council so that local trends in need of higher-level input can be shared as appropriate for the purpose of promoting effective statewide program improvement planning activities. A visualization of this quarterly CQI team meeting schedule is provided below. Quarterly CQI Schedule of Key Activities f. Meeting Agenda: Standing meeting agenda items for the State CQI Council should connect to current child welfare agency and system strategic priorities (see Section 2) along with a focus on the use of available data and evidence. Potential standing agenda items include but are not limited to: i. Review and evaluation on the progress of PIP strategies, CFSP goals and other agency priorities and recommend program adjustments as needed to support successful implementation towards improved outcomes. ii. Review of agency key performance measures, including safety and permanency outcome data such as AFCARS, NCANDS, NYTD, and other reports. iii. Review of semiannual case review (OSRI) results and discussion of state and regional initiatives and issues in need of improvement. iv. Provide ongoing consultation and collaboration to Cross-Zonal CQI Teams re; issues being raised from the local level. March 31, 2021 Page 15

Continuous Quality Improvement Program Manual v. Determination for the need of state or Cross-Zonal level action plans based on review of performance data and use of CQI Cycle/TOC where appropriate. vi. As needed, develop, and implement state level action plans using the CQI cycle. g. Application of CQI/TOC Cycle: State CQI Council members are expected to utilize the Six- Step CQI Cycle along with key concepts from the Theory of Constraints as outlined in Section 6 of this manual as part of quarterly team meetings and related CQI activities as the primary framework for identifying problems and implementing effective solutions throughout the North Dakota child welfare system. This includes a focus on: i. Reviewing KPI’s and other available data to identify trends and potential problems and engaging team members to understand the underlying conditions that are driving the problem. ii. Identifying and implementing effective solutions that address the root causes of the problem. iii. Ongoing monitoring – Working as a team to close the loop by determining if the solution has been successful and whether to close the matter or if future monitoring is needed. Team members should also utilize the CQI Action Planning Worksheet to document use of the full CQI cycle from problem identification to successful resolution. (See appendix) h. Communication of State CQI Council Activities: State CQI Council members should determine as part of the Team Charter development process how information from team meetings and related state CQI activities will both be documented and shared with CQI team members as well as the broader agency and child welfare system in order to keep everyone informed and engaged in the CQI process. Elements to consider for the communication plan should include: i. Identifying key types of CQI information to be regularly shared, both internal and external, to staff and stakeholders. This includes such information as CQI meeting minutes, data reports, findings from data analysis, project milestone updates. ii. Methods for sharing information, frequency of sharing information and communication, and who will be responsible for ensuring that all key information is being disseminated as planned iii. Protocols for ensuring multi-directional communication by determining how information will flow to and from agenda leadership and Cross-Zonal CQI teams, and among any related subcommittees. i. State CQI Ad hoc Sub-groups: Short or long-term sub-groups may be formed as needed to target or support specific areas in need of improvement and facilitate activities such as: i. Conducting statewide analysis or overseeing specific change projects or initiatives, including utilization of Theory of Constraints. This includes the following TOC Redesign Sub-groups, each of which meet monthly or as needed, dependent upon the phase of the project. 1. CPS TOC Redesign Workgroup: The primary purpose of this workgroup is to research, identify constraints, and implement solutions to improve performance and outcomes to provide children and families being served by child protection the right service at the right time, and at the right frequency and intensity. March 31, 2021 Page 16

Continuous Quality Improvement Program Manual 2. In-Home Case Management TOC Redesign Workgroup: The primary purpose of this workgroup is to research, identify constraints, and implement solutions to improve performance and outcomes for children and families receiving in-home case management services. 3. Safety Framework Practice Model Champions of Change TOC Workgroup: The primary purpose to this workgroup is to research, identify, and implement a new practice model for child welfare in North Dakota. 4. CPS Central Intake Champions of Change TOC Workgroup: The primary purpose of this workgroup is to research, develop, and implement a centralized intake process for CPS. ii. Additional Ad hoc subgroups: The State CQI Council should assess the need for additional ad hoc subgroups: This may include establishing workgroups or committees designed to: 1. Provide targeted technical assistance as needed to Cross-Zonal CQI teams for specific project areas 2. Provide data support such as aggregating and analyzing relevant data to the State CQI Council and/or Cross-Zonal CQI teams. 3. Provide targeted support to CFSR related projects such as finishing out Round 3 successfully and overseeing Round 4 related activities (i.e., Statewide assessment, stakeholder feedback sessions, PIP development and monitoring. 2. Cross-Zonal CQI Teams a. Purpose: Cross-Zonal CQI Teams are the drivers for the local CQI process with a focus on improving child welfare agency case practice, service delivery and the achievement of outcomes for ND children and families. Cross-Zonal CQI Teams are comprised of CFS FSS, Tribal, DJS and HSZ agency staff and stakeholders and will meet on a quarterly basis and are responsible for reviewing regional/zonal data and implementing the CQI Cycle and Theory of Constraints at the local level. Other responsibilities include: • Review of data from various sources including CFSR/QA data and agency data reports. • Review of case review data and discussion of local initiatives or action plans to address areas needing improvement. • Design and implementation of CQI projects to improve local outcomes throughout the child welfare service delivery system. • Provide ongoing consultation and collaboration to review and evaluate the progress of relevant PIP strategies and CFSP goals and recommend program adjustments to support effective implementation and the successful completion of CFSR/CFSP plan requirements. • Provide for or arrange for ongoing training for agency staff on CQI principles. • Promote a Cross-Zonal culture that values service quality and continual efforts by team members, agency partners and contractors to achieve program goals and positive outcomes for service recipients. March 31, 2021 Page 17

Continuous Quality Improvement Program Manual b. Membership: Membership of each of the four Cross-Zonal CQI Teams includes, but is not limited to representation from: • Children and Family Services Division (CFS Field Service Specialists, including at minimum one from CPS and one from Case Management to ensure both practice areas are represented) • QA Unit representative • Human Services Zones • Tribal Social Services • Division of Juvenile Services • Private and private provider agencies • Youth and family • Juvenile Court Supervisor or their designee Local agencies will participate in the CQI process by serving as members on their respective Cross-Regional CQI Teams. This ensures that local agency needs/issues are seen and addressed at the State CQI Council level. c. Establishing a Team Charter: Each Cross-Zonal CQI team is encouraged to work together to develop a team charter (See previous State CQI Council section for information on purpose and steps for developing a team charter). d. Cross-Zonal CQI Team Co-Lead Responsibilities: (See previous State CQI Council section for a list of team lead responsibilities as it relates to agenda planning, meeting preparation and facilitation, documentation of meetings and follow-up). e. Meeting Frequency: Cross-Zonal CQI Teams are expected to meet on a quarterly basis, in advance of the State CQI Council, following the meeting cadence as outlined previously in this manual. f. Meeting Agenda: Standing meeting agenda items for Cross-Zonal CQI Teams should connect to the state’s strategic priorities as outlined in Section II of this manual along with a focus on local or zonal issues that are impacting practice, service delivery, and achieving positive outcomes. Potential agenda items include: i. Review of zonal and Cross-Zonal performance data in relation to progress towards achieving PIP goals. ii. Review of zonal and Cross-Zonal case review (OSRI) results and discussion of local areas in need of improvement. iii. Determination for the need of Cross-Zonal or zonal action plans and use of the CQI Cycle and Theory of Constraints where appropriate. iv. Determination for the need to raise challenges or barriers to the State CQI Council for assistance in resolution. g. Application of the CQI Cycle and Theory of Constraints: Cross-Zonal CQI Teams are expected to utilize the Six-Step CQI Cycle along with key concepts from the Theory of March 31, 2021 Page 18

Continuous Quality Improvement Program Manual Constraints as outlined in Section 6 of the manual during quarterly team meetings as the primary framework for identifying problems and implementing effective solutions. h. Communication of Cross-Zonal CQI Activities: Team members should determine as part of the Team Charter development process how relevant information will documented and shared. (See previous State CQI Council section for information on key communication plan elements). i. Cross-Zonal ad hoc sub-groups: Short or long-term sub-groups may be formed as needed to target or support specific zone or Cross-Zonal areas in need of improvement. 3. CQI Support Team Purpose and Membership: The primary purpose of the CQI Support Team is to provide or arrange for technical support and needed resources to the State and Cross-Zonal CQI teams. The CQI Support Team will chaired by the CQI Administrator. The team will include members of the CQI Council including, at a minimum, the Director of the UND Children and Family Services Training Center, the QA Unit Manager, a CFS Field Services Specialist Supervisor, a Decision Support Services research analyst, designated CQI staff, representatives from DJS such as field training officers, and representative from Tribal agencies. The CQI Administrative Team will meet quarterly or more often if needed. Responsibilities include: • Drafting policies and procedures for the CQI process within child welfare services • Arranging for or provide initial and ongoing education and technical assistance on CQI principles and processes. • Guiding ongoing assessment and evaluation of the overall public child welfare system • Overseeing and supporting statewide and regional/zonal data analysis or change projects, including utilization of the Theory of Constraints model • Drafting and disseminating data reports March 31, 2021 Page 19

Continuous Quality Improvement Program Manual IV. STAFF AND STAKEHOLDER PARTICIPATION IN CQI ACTIVITIES Purpose: The CQI vision for the North Dakota Child Welfare System is dependent upon the active engagement and participation of staff at all levels of the agency, as well as children, youth, families, and key stakeholders such as courts, tribes, and providers. To achieve this vision, staff and stakeholders must have opportunities and be supported to actively participate and assume meaningful roles in all phases of the CQI process. This includes: 1. Identifying and understanding the problem; 2. Researching solutions; 3. Developing a theory of change; 4. Adapting or developing a solution; 5. Implementing the solution; and 6. Monitoring and assessing the results. CQI Roles - Responsibilities and Opportunities Potential roles for staff and stakeholders to participate in the CQI process can be distinguished by “responsibilities” and “opportunities” with both roles being essential to an effective CQI model. (Note: See appendix for CQI Roles – Responsibilities and Opportunities matrix) “Responsibilities” as it relates to CQI pertains to agency staff and specific stakeholder groups such as private providers, courts, and tribes where there are clear expectations for the participation in some aspect of the CQI process or in carrying out specific CQI related activities such as producing and disseminating data or in accurately entering data into agency data bases. “Opportunities” relates to staff or stakeholders having the option as well as being provided access and support to be able to serve in various CQI capacities such as being a member or serving in an advisory capacity on a CQI team, having access to and reviewing data, participating as case reviewers or in focus groups, or in other aspects of the CQI process. State and Cross-Zonal CQI teams are expected to ensure that team membership is representative of the perspectives of multiple agency staff and stakeholders across the child welfare system. The Department is committed to seeking out opportunities to have an inclusive CQI process with particular focus on engaging youth, parents, and others with lived experience in CQI activities. All stakeholders should be prepared by CQI Team leads or agency leadership in advance of participating in CQI activities by explaining the purpose of their participation, their role to provide feedback based on their experience, and to make recommendations related to improving service delivery. Key Agency Staff and Stakeholders in the North Dakota CQI process include but are not limited to the following: • State Level Agency Leadership • Regional/Zonal Management • Agency Caseworkers and Supervisors • Agency Information Technology/Data March 31, 2021 Page 20

Continuous Quality Improvement Program Manual • CQI/Case Review Staff • Public Providers • Private Providers • Courts and DJS • Tribes • Youth, Parents, Caregivers • Foster and Adoptive Parents Roles for staff and stakeholders to play in the North Dakota CQI process include but are not limited to the following: • Contributing to a high-quality North Dakota Data Pathway: o Data entry and collection – This refers to timely and accurate entry as well as collection of data to ensure overall data quality and includes both administrative and qualitative data. Agency staff and child welfare system partners are encouraged to consider the role each plays in ensuring the quality of data that will be utilized as part of the CQI process.  Caseworkers and Supervisors  Agency Leadership and Regional/Zonal Management  Agency IT staff  Case Review staff  Public and Private Providers  Courts and DJS  Tribes o Data extraction and production – This refers to the importance in generating user-friendly data reports that can be utilized by agency staff and stakeholder, as well as CQI teams, as part of the CQI process.  Caseworkers and Supervisors  Agency Leadership and Regional/Zonal Management  Agency IT staff  Case Review staff  Public and Private Providers  Courts and DJS  Tribes o Data dissemination – This refers to the sharing of data with staff and stakeholders in ways that promote a common understanding of what is being presented and connects practice to outcomes.  Caseworkers (opportunity) and Supervisors March 31, 2021 Page 21

Continuous Quality Improvement Program Manual  Agency Leadership and Regional/Zonal Management  Agency IT staff  Case Review staff  Courts and DJS  Tribes (opportunity) o Use of Data – This refers to the regular review and analysis of data in an effort to understand and monitor agency practice, service delivery, and outcomes.  Caseworkers and Supervisors  Agency Leadership and Regional/Zonal Management  Agency IT staff  Case Review staff  Public and Private Providers  Courts and DJS  Tribes  Youth, Parents, Caregivers, Foster, and Adoptive Parents (opportunity) • CQI Teaming – Agency staff and stakeholder should be encouraged to seize the opportunity to serve as a member on the State CQI Team, Cross-Zonal CQI Teams, ad hoc CQI sub- groups/teams, or by actively participating in other structured CQI related activities. CQI meetings provide a structured approach for staff and stakeholders to contribute to the process of identifying problems, utilizing data to understand root causes, generating solutions and monitoring progress towards improving outcomes. o Caseworkers and Supervisors o Agency Leadership and Regional/Zonal Management o Agency IT staff o Case Review staff o Public and Private Providers (opportunity) o Courts and DJS o Tribes o Youth, Parents, Caregivers, Foster, and Adoptive Parents (opportunity) • Case Review/OSRI – Agency staff and stakeholders have both the responsibility and opportunity to participate in North Dakota’s case review process by serving as a case reviewer, case interviewee, or in an OSRI technical support capacity. o Caseworkers and Supervisors o Agency Leadership and Regional/Zonal Management o Agency IT staff (opportunity) March 31, 2021 Page 22

Continuous Quality Improvement Program Manual o Case Review staff o Public (responsibility) and Private Providers (opportunity) o Courts and DJS (opportunity) o Tribes (opportunity) o Youth, Parents, Caregivers, Foster, and Adoptive Parents (opportunity) • Focus Groups and Surveys – Agency staff and stakeholders who are not actively serving as members on a CQI team can still ensure that their voices are represented as a part of the CQI process. This may include serving on child welfare system committees and workgroups, responding to surveys, or by participating in interviews or focus groups as part of the case review process which contributes to the gathering of valuable feedback and informs program improvement planning efforts. o Caseworkers and Supervisors o Agency Leadership and Regional/Zonal Management o Agency IT staff o Case Review staff o Public (responsibility) and Private Providers (opportunity) o Courts and DJS o Tribes o Youth, Parents, Caregivers, Foster, and Adoptive Parents (opportunity) • CQI Cycle of Learning/Theory of Constraints – Staff and stakeholders serving on CQI teams should are expected to participate in structured problem-solving activities as part of CQI Teaming efforts or ad hoc problem established workgroups. (Note: See Section 3 re: CQI Teaming responsibilities in use of the CQI cycle and Theory of Constraints). March 31, 2021 Page 23

Continuous Quality Improvement Program Manual V. TRAINING AND RESOURCES TO SUPPORT CQI Purpose: Staff and stakeholders at all levels of the North Dakota child welfare system need to be prepared and supported to fully participate in all phases of CQI process. This includes the provision of adequate resources, staff, time, and training. CQI Training Overview All new and existing staff child welfare staff will receive CQI training to prepare them to lead or assist with CQI projects in the agency. Training and other CQI information will be provided through a variety of methodologies including but not limited to: • Child Welfare Certification Training Program through the University of North Dakota’s Children and Family Services Training Center • Web-based module basic training and refresher training hosted by University of North Dakota Children and Family Services Training Center • New employee orientation • Periodic educational newsletters CQI Foundation Training for North Dakota CQI Members: Department staff and stakeholders who will be playing a key role as leaders and facilitators in launching implementation of North Dakota’s CQI model will have the opportunity to participate in training during 2021 that is co-designed and facilitated with the Capacity Building Center for States (CBCS) with a goal of ultimately integrating key content into existing training curricula accessible to all child welfare staff in the future. The planned training agenda is intended to be delivered virtually (or in-person if deemed appropriate) over the course of four sessions. The training modules presents an overview of key information relevant to helping child welfare agencies build agency capacity for continuous quality improvement (CQI) and will be customized to the core components of ND’s CQI model. CQI Training Curriculum Outline Session 1: Welcome, Culture and Climate, Administrative Structure for CQI (3.5 hours) Unit 1: Welcome and Introductions • Name, position, where located • How do we want to work together today? • Overview of Key Features of a CQI System • Quality Assurance vs. Quality Improvement • Overview of Theory of Constraints March 31, 2021 Page 24

Continuous Quality Improvement Program Manual Unit 2: Culture and Climate Count Training Competency: Participants will know how to create a CQI-receptive culture and climate. Learning Objectives: (Note: Curricula will incorporate ND CQI vision and strategic priorities) 1. Increase knowledge about how culture and climate together create a CQI-receptive agency. 2. Increase knowledge of the characteristics associated with a CQI-receptive culture and climate. 3. Increase knowledge of strategies for enhancing an agency’s culture and climate to promote effective CQI activities. 4. Develop a plan describing specific actions participants can take when they return to their jobs to enhance the agency’s CQI-related culture and climate. Unit 3: Administrative Structure for the CQI Process Training Competency: Participants will know how to establish an administrative structure that supports the CQI process. Learning Objectives: (Note: Curricula will incorporate North Dakota’s CQI teaming structure) 1. Increase knowledge about what is required for an administrative structure to support the CQI process. 2. Increase knowledge about the various strategies necessary to ensure a supportive CQI administrative structure. Session 2: Leading with Data, Promoting High-Quality MIS Data, Promoting High-Quality Case Review, Survey, and Interview Data (3.5 hours) Unit 4: Leading with Data Training Competency: Participants will be aware of strategies for leading with data and using data to drive decision-making. Learning Objectives: (Note: Curricula will incorporate a visual “data walk” using North Dakota child welfare key performance measures to familiarize participants with agency performance) 1. Develop a plan that describes specific actions participants can implement in their jobs to enhance an agency’s capacity to lead with data. 2. Increase knowledge about: a. Why using data when making decisions is critical to achieving desired outcomes for children and families served by child welfare agencies b. Factors associated with successful efforts to lead with data c. Strategies for enhancing an agency’s capacity to lead with data (or to make decisions based on data) Unit 5: Promoting High-Quality Data from your MIS Training Competency: Participants will know how to promote high-quality data from an agency’s management information system (MIS) for the CQI process. March 31, 2021 Page 25

Continuous Quality Improvement Program Manual Learning Objectives: (Note: Curricula will incorporate focus on the responsibilities for agency staff at all levels to ensure data quality) 1. Increase knowledge of the factors that affect the quality of data in an MIS and how the quality of that affects the CQI process. 2. Increase knowledge of the strategies for enhancing the data quality in an MIS. 3. Develop a plan describing specific actions that participants can implement in their jobs to enhance the quality of data in their MIS. Unit 6: Promoting High-Quality Data through Case Reviews, Surveys, and Interviews Training Competency: Participants know how to promote high-quality data from case reviews and from other types of data collection involving samples such as surveys and interviews. Learning Objectives: 1. Develop a plan that describes specific actions participants can take in their jobs to enhance the quality of data collected through case reviews, surveys, and interviews. 2. Increase knowledge about factors that affect the quality of data collected through a case review, survey, or an interview process and how data quality affects the CQI process. 3. Increase knowledge about the strategies for enhancing data quality in a case review, survey, or an interview data collection effort. Session 3: Data Analysis, Staff and Stakeholder Engagement, (3 hours) Unit 7: High-Quality Data Analysis Training Competency: Participants will know how to promote a high-quality data analysis process. Learning Objectives (Note: Curricula will include focus on methods to analyze North Dakota data as part of CQI Cycle) 1. Increase knowledge about what constitutes a quality data analysis process. 2. Increase knowledge about the kinds of factors that affect the quality of data analyses and dissemination. 3. Increase knowledge about the strategies for enhancing the quality of the data analysis process. 4. Develop a plan describing specific actions participants implement in their jobs to enhance the quality of the data analysis process. Unit 8: Staff and Stakeholder Engagement Training Competency: Participants will know how to engage the child welfare agency community in the agency’s CQI process. Learning Objectives: (Note: Curricula will incorporate the identification of key North Dakota child welfare systems stakeholders, highlight importance and ways to effectively engage in CQI activities) 1. Increase knowledge of the benefits of engaging the child welfare agency community to both the community and the agency. March 31, 2021 Page 26

Continuous Quality Improvement Program Manual 2. Increase knowledge of strategies for enhancing engagement of the child welfare agency community and how strategies may differ depending on the target population. 3. Develop a plan describing specific actions participants can take in their jobs to enhance engagement of the child welfare agency community. Session 4: CQI Cycle and Theory of Constraints (3 hours) Unit 9: Application of CQI Cycle and Theory of Constraints Training Competency: Participants will know how to effectively utilize the six-step CQI Cycle and Theory of Constraints key concepts as part of CQI teaming activities. Learning Objectives: 1. Increase knowledge about how to apply all six-steps of the CQI cycle. 2. Increase knowledge about how to integrate key concepts from the Theory of Constraints as part of the CQI process. 3. Develop a plan describing specific actions participants can implement in their jobs to enhance the effective utilization of the CQI cycle and Theory of Constraints towards improved outcomes for North Dakota children and families. Closing: • Communicate next steps Case Review Staff Training and Certification (Note: See also Section VII – Case Record Review) Initial Certification: All QA Case Review Workforce members will participate in an initial certification training appropriate to their role. Certification training for Reviewers will include the following topics: QA Case Review Overview, OSRI Overview, Orientation to the OMS, Interviewing Skills, and a mock case activity. Participants must pass a certification quiz following the training. QA Leads will be certified as a Reviewer plus complete an additional training specific to QA duties to complete their required training. Participants must pass a certification quiz upon completion of training. Training will be offered as needed to ensure a sufficient workforce. Ongoing Training: Ongoing training will be available to the Case Review Workforce members through formal and informal opportunities, such as but not limited to ‘deeper dive’ learning opportunities during staff meetings, use of the E-Learning site on the CFSR Information Portal, and statewide training opportunities designed to enhance knowledge and skills of topics impacting the child welfare system. March 31, 2021 Page 27

Continuous Quality Improvement Program Manual Ongoing Certification: Once certified, all QA Case Review Workforce members complete a biannual recertification activity and successfully pass a follow up quiz to maintain certification. The exercise will be designed to ensure all workforce members remain up to date on the OSRI and OMS. Additional CQI Resources The Department is still in the process of building capacity in the area of CQI. As such, CQI teams at both the state and Cross-Zonal levels are advised to contact the CQI Support Team as needed for assistance in supporting CQI activities. (see CQI Support Team responsibilities in section 3 of this manual). March 31, 2021 Page 28

Continuous Quality Improvement Program Manual VI. CQI CYCLE AND THEORY OF CONSTRAINTS Purpose: Continuous Quality Improvement (CQI) is the process of identifying, describing, and analyzing strengths and problems and then testing, implementing, learning from, and revising solutions. It relies on an organizational culture that is proactive and supports continuous learning1. In North Dakota, this CQI process is informed by principles from the Theory of Constraints (TOC). North Dakota adopted Theory of Constraints as a method for systematically improving operation of the Department of Human Services’ Children and Family Services Division. The intention was to identify work practices that were limiting efficient progress towards agency goals and redesign those practices to increase capacity and to improve the quality of completed work. While the two strategies are similar in that they both use data to inform targeted improvements, CQI is often broader in the types of data used, such as case reviews, and includes a wider variety of outcomes, such as expanding successful programs. North Dakota intends to integrate key concepts from Theory of Constraints, particularly as it relates to identifying and redesigning work practices that limit efficient progress towards agency goals, into the six-step CQI cycle and utilize it as the foundation for the state child welfare system’s CQI model. A brief overview of each methodology is provided below. Overview of CQI Cycle “A continuous quality improvement approach allows States to measure the quality of services provided by determining the impact those services have on child and family level outcomes and functioning and the effectiveness of processes and systems in operation in the State and/or required by Federal law “. ACYF-CB-IM-12-07 The purpose of the six-step cycle of CQI is to 1. Identify and systematically make decisions that will lead to Understand the desirable change. The steps are: (1) identify and understand what is not working, (2) identify Problem solution options and determine the fit of the solution to the identified problem, (3) design an 6. Monitor and 2. Research the actionable plan based on what is available, (4) Assess the Solution Solution determine if the solution is matched to the situation, (5) implement the solution, and lastly 5. Implement the 3. Develop a (6) monitor for successful implementation of the Solution Theory of Change solution. 4. Adapt or Develop the Solution 1 2012 CQI Memo Page 29 March 31, 2021

Continuous Quality Improvement Program Manual Overview of Theory of Constraints (TOC) The Theory of Constraints is a methodology for identifying the most important limiting factor (i.e., constraint) that stands in the way of achieving a goal and then systematically improving that constraint until it is no longer the limiting factor. In manufacturing, the constraint is often referred to as a bottleneck.” WHAT IS THE THEORY OF CONSTRAINTS? (https://www.leanproduction.com) Five focusing steps are used to redesign practice under a theory of constraints model: 1. Identify the constraint: the constraint is the step in the work that slows down (or limits) what can be accomplished. 2. Maximize the constraint: change the way that work is completed to make the most of available resources. 3. Subordinate all other functions to the constraint. That means that no work can out-pace the constraint. No step in the process can “get ahead” of the constraint and cause a back-log of tasks that are waiting for the constraint. 4. Elevate the constraint: To elevate a constraint is to make it the number one priority and work on increasing the amount of work that can be accomplished in that section of the process. And finally, 5. Focus: continue to monitor the overall goals of the agency and monitor the efficiency of the work it takes on. The methods that make up Theory of Constraints are useful as options for teams that are evaluating the quality of services and improvements on an ongoing basis. Integrating CQI Cycle and Theory of Constraints The process described in this section is intended to include input from those who are affected by the problem and those who have influence over the problem. Including many voices can lead to solutions that are better adapted to the system and can also encourage a culture of growth. Applying the CQI steps is an opportunity to improve the system, rather than work that is done in addition to day-to-day tasks. Each of the six steps in the CQI cycle detailed in this section of the manual also includes a series of “TOC Tips” which are intended to highlight key activities or concepts from the Theory of Constraints model that teams may find useful in effectively moving an identified problem towards resolution. When to apply the process Problems that are explored in depth can arise from a variety of sources. Some problems may be identified through regular monitoring of agency priorities and legal requirements as a part of quarterly State and Cross-Zonal CQI meetings. Other problems may be identified when staff in the field draw attention to ongoing barriers or struggles. No matter where the problem is identified, the steps are designed as a cycle of improvement based in a review of data or other available evidence. The CQI cycle is intended to produce a solution that improves the effort needed to reach desirable outcomes, making it worth the effort. The communication between the state office, zonal administrations, and frontline staff increases the likelihood that solutions will benefit everyone. Zonal offices play a key role in helping to identify March 31, 2021 Page 30

Continuous Quality Improvement Program Manual problems that are meaningful to them and ensuing that solutions benefit their work. Communication between zones will encourage successful strategies. Zones will be able to benefit from improvement cycles completed in other areas and gain insight into their own work. CQI teams are strongly encouraged to use the CQI Action Plan Worksheet located in the appendix of this manual as a way of continually facilitating, documenting, and tracking progress towards the resolution of identified problems throughout the CQI cycle. 1. Identify and Understand the Problem: The goal of the first step in the CQI cycle is to understand the root causes of the problem, rather than a focus on only the symptoms. The first step in the CQI cycle is to identify and assess the nature of the presenting problem by gathering and fully examining all of the relevant information in order to ensure there is shared understanding as to the contributing factors and root causes to the problem. The following key tasks to identifying and understanding a problem include: a. Assemble a team: It is essential to bring together a team TOC Tips for Step 1 of internal and external stakeholders who are most likely to support the agency in successfully identifying and ultimately  Use “process mapping” by addressing the root causes of the identified problem as putting relevant work-tasks in part of the CQI process. State and Cross-Zonal CQI order with the aim of defining teams (depending on whether the identified problem is a the process from initiation to state or local issue) should assess whether additional completion. perspectives, experiences or expertise are needed on the existing CQI team to ensure that the underlying issues to  Identify any spots where work is the problem are understood and are utilized to inform building up; this may be a decision-making in terms of identifying a possible solution. “primary constraint”. b. Clearly articulate the problem statement: This begins by  Explore what is limiting any considering: How do you know it is a problem? It is identified constraint, and what important that there is shared understanding among the unused capacity exists outside of team as to their understanding of the problem, therefore the constraint. the team should discuss what is currently known about the problem. A problem statement explains what needs to change in order to meet agency priorities. Problems may reflect identified needs or opportunities for building on successes to improve agency functioning or outcomes. The team should be prepared to adjust the problem statement as information is uncovered and understanding regarding the problem grows. c. Define goal/outcome of problem: The team should seek consensus as to the goal or outcome that will be achieved once the problem is addressed. What is supposed to be happening? It is okay to be aspirational, connecting desired outcomes to the strategic priorities of the agency March 31, 2021 Page 31

Continuous Quality Improvement Program Manual and relevant key performance indicators. (See Section 9 for the list of Key Performance Indicators). d. Identify the target/affected population: In determining the target population, the team should consider who is most effected by the problem, who completes the task, and who benefits from the desired outcome. e. Identify key research questions: An important step to gaining a greater understanding of a problem is to come up with a set of research questions. This will help inform how data will be collected and analyzed, in order to discover the root cause(s) to the problem. The team should brainstorm areas worth exploring, including: i. What do we need to know that will help us better understand the problem? ii. When is the problem happening? Why is it happening? iii. Is everyone experiencing the problem the same? Are the any differences we should explore (i.e., variations by population groups or geographic differences)? The team should seek consensus and prioritize the top areas to explore and translate those areas into a concise set of research questions to pursue through a comprehensive review of data. f. Assess available data sources and determine data gaps: The next task is to begin the process of determining the types of data that are currently available to help answer the research questions and gain a better understanding as to what is driving the problem. Teams are encouraged to take time to identify and explore the full array of available agency data and information, both quantitative and qualitative. This may include data sources such as: i. State or federal data sets and measures, such as AFCARS/NCANDS/NYTD ii. CFSR case reviews iii. Agency administrative (MIS) data (i.e., FRAME) iv. Targeted case reviews v. Staff and stakeholder surveys, interviews, and focus groups vi. Process mapping Once the team has come to consensus on a set of available data sources, they should also identify any data gaps or needs and begin the process of developing a plan to access or gather needed information. g. Collect and analyze multiple sources of data: How can you use data to inform the question? h. Explore evidence of the problem to identify and understand root causes of the problem: What happens if the problem requires a larger system change? 2. Research the Solution. In this next step, the goal is to identify solution options and determine the fit of those options to the identified problem. It will help to identify any existing evidence-based or evidence-informed solution or intervention that will adequately address the problem. This step may require team members to take the time to research available solutions that have worked in other situations or may require that the team strategize to develop new solutions. March 31, 2021 Page 32

Continuous Quality Improvement Program Manual a. Identify potential solution options: The team may look at similar TOC Tips for Step 2 situations in other states or in other parts of their agency. Potential solutions may be adapted from procedures or policy in other fields. A procedure that has been developed elsewhere may speed up time to implementation but may require • Explore alternative ways of adaptation in order to work in this unique situation. Newly completing the task that is most developed solution can be customized to meet an exact need constrained. but may require additional time to develop and evaluate for • Revisit the system map, look at success. potential solutions and how b. Search for evidence-based or evidence-informed solutions: each will impact the problem. The team will want to consider whether the solution needs to be • Make sure the measurement of established as evidence based and consult with reputable change is included in the clearing houses for lists of services that meet this standard. solution. (i.e., what percentage Generally, clearing houses can provide information on the improvement can you expect). targeted population and resources that may be needed to apply the solution. c. Determine the fit of the solution to the identified problem: The team should research possible solution options and determine how it will address the target population and problem, and whether adaptions to the intervention will be needed. Whenever a solution is being considered, members are encouraged to use the list of questions below to help guide consideration of whether the solution is right for the identified problem. i. Will the solution meet the population’s needs? ii. How does the solution fit within your agency? iii. What is the level of readiness for replicating the solution within your agency as intended and to support sustainability of the solution? 3. Develop a Theory of Change: After a solution or intervention has been identified for the problem, the third step in the CQI cycle is for the CQI team to ensure there is a clear link between the root causes of the problem and the identified solution, also known as the “causal pathway”. This step helps to confirm that the selected solution will influence change in a way that resolves the original source of the identified problem. The theory of change can be thought of as a roadmap that identifies the desired destination and the paths or routes the team will take to arrive at that destination. CQI teams are encouraged to take the time to work together and develop a theory of change as a way to achieve buy-in with staff and stakeholders regarding steps needed to achieve shared goals and to guide strategic planning and decision-making. March 31, 2021 Page 33

Continuous Quality Improvement Program Manual The key steps to developing a theory of change include: a. Gathering information on the problem statement, root causes and target population: These elements will define the actions steps needed to move from root-cause to desired change therefore it is essential that there is consensus among the team as to their understanding of the identified problem, TOC Tips for Step 3 root cause(s), and target population. b. Identifying the long-term outcome: The eventual outcome is a description of the resolved situation. In other  It may be helpful to describe the words, what will success look like? Developing a workflow using any new version description of the outcome also helps determine what of the primary constraint. needs to be monitored to determine successful change.  Determine the steps needed for CQI team members may find it useful to first brainstorm a list scaling up the new process to of desired outcomes which can then be refined into a clear the entire system expect). outcome statement. c. Identifying the requirements (causal links) needed to achieve the long-term outcome: Each step in the chain is an action that resolves the root cause issues and builds a foundation for the desired change. CQI team members are encouraged to brainstorm and document all of the “requirements” they can think of that will be needed in order to meet the desired long-term outcome, specifically by considering: (1) What results must happen before the long-term outcome can be reached and (2) What results are needed to address the root cause(s)? d. Organizing the causal links to develop the pathway of change: Once the causal links have been identified, the CQI team will need to organize the causal links or requirements into a pathway of change. The order of the causal links will help guide the implementation process by ensuring that each segment of change supports the movement toward the intended outcome. There may be intermediate steps in between the starting place and the long-term outcome that can be used to monitor progress towards the goal. 4. Adapt or Develop the Solution Implementation: The goal of this step is to assess evidence about the fit and feasibility of implementing the solution. The aim is to ensure that implementation of the intervention is possible and reasonable within the context of existing agency resources and other agency initiatives and mandates. The steps of the theory of change may suggest different resource needs at each stage of change during set up and implementation of the new solution. a. Consider what resources are needed to achieve successful implementation of the selected solution: Resources may include time or access to work supports. It may require increasing technical expertise on the team or access to data sources. b. If adaptions are needed, determine how they may affect fidelity to the intervention and expected outcomes: For example, if you are applying a solution to a new population then the process for verifying efficacy may need to be repeated. March 31, 2021 Page 34

Continuous Quality Improvement Program Manual c. Assess evidence, fit and feasibility of the solution: TOC Tips for Step 4 Ensuring that implementation of the intervention is possible and reasonable within the context of existing agency  Each person’s actions may resources and other agency initiatives and mandates change the solution differently. i. Operationalize the solution/intervention by further Consider the role of each person defining the “it”, specifically: in the flow of work. ii. Purpose, goals, & guiding principles of intervention iii. Design and operationalize core components  Nuances in solutions can be iv. Define what else is necessary to make the intervention defined as part of the process to work ensure they are consistent (i.e., built into the system map). 5. Implement the Solution: The goal is to engage in a logical  Make sure the new process does process for successful implementation of the solution. The state not create a new constraint CQI and case review teams can be a source of support. They elsewhere in the larger system. can summarize the trends that they find through case reviews. Consider whether existing reports can be leveraged to monitor the process of change. TOC Tips for Step 5 a. Assess agency readiness: Think about the resources available to implement the solution including motivation and organizational capacity. The team may need to have  Define key performance, a plan in place to for what to do if readiness is not present outcome, and fidelity measures (e.g., staging the roll-out or waiting until a better time). that will be used to evaluate b. Identify and address the supports needed: Be able to implementation success. list the resources needed to collect data. Focus on  “Re-design Champions” may defining evaluative measures. help organize the i. Engage key stakeholders and provide guidance to the implementation plan. field re: implementation activities ii. Develop an Evaluation and CQI plan - Define key performance, outcome, and fidelity measures that will be used to evaluate implementation success iii. Collect and disseminate baseline and ongoing performance data c. Consider whether to conduct initial implementation: There may be a need to start the solution on a limited scale with testing for results and workability. This activity may be related to the information gathered in the readiness assessment process above. d. Develop an implementation plan: If needed, a team can develop a plan for bringing the solution to scale. The new solution will need to fit in with existing practice. A project plan can aid in tracking decisions. The state team may provide support for planning timelines and plans for scaled implementation. 6. Monitor and Assess the Solution: The final step is to carefully track information to observe that the intended change is occurring. Watching for the first indications of change can ensure successful implementation of the solution and achievement of desired outcomes. Building in an expectation for March 31, 2021 Page 35

Continuous Quality Improvement Program Manual ongoing discussions around data and program evaluation will promote a culture of learning within the organization. a. Manage implementation of the monitoring and/or TOC Tips for Step 6 evaluation plan: Continually monitor data and assess the need for adjustments. Consider who has oversight in determining success of the change. Key measurement  Remember to update the “system points may be found by examining: map” with any procedure i. Pre-determined outcomes and performance indicators changes, including any changes ii. Benchmarks of progress at regular intervals due to ongoing monitoring. iii. Fidelity to the solution/intervention iv. Implementation process and reasons for progress or lack of progress b. Collect necessary data: Regular tracking of data will ensure data are accurate and reliable. The team may need to consult with technical staff who oversee management of FRAME. The state team will need to coordinate access to data for the regions. c. Analyze the interpret data: Team will need to provide regular feedback to agency management, staff, and stakeholders on progress of implementation. Consider what actions will be taken is the necessary movement toward change is not happening as planned. March 31, 2021 Page 36

Continuous Quality Improvement Program Manual VII. DATA COLLECTION AND DISSEMINATION Purpose: To ensure the continuous quality improvement of the North Dakota Child Welfare System, it is critical to have ongoing access to reliable data about how the system is operating. The type of information needed will be driven by the agency’s goals, objectives, and desired outcomes. Access, sharing, and the use of quality data with Department staff and stakeholders is essential to informing and improving child and family outcomes for the state and to support CQI activities. Data Pathway for North Dakota Child Welfare The data pathway for North Dakota’s child welfare system refers to how data moves throughout the agency’s information systems and includes how child and family client information is entered, stored, extracted, and utilized to improve performance. For North Dakota’s CQI model to be effective, it is essential that Department staff and key stakeholders have a shared understanding regarding roles and responsibilities for: • Entering information into the system • Extracting data from the system (I.e., data reports) • Overseeing the quality of data in the system and the accuracy of data reports • Interpreting the data that comes from the system • Disseminating to multiple stakeholders (Note: See Section 4 for additional information regarding staff and stakeholder roles and responsibilities regarding data). Entering Extracting data Overseeing the Overseeing the Interpreting the Disseminating to from the system information into quality of data in accuracy of data data that comes multiple system the system reports from the system stakeholders Data Entry and Extraction Primarily, data is entered into the FRAME data base by intake CPS workers, foster care workers, in home workers, administrative staff, supervisors, and field services specialists. Prior to entering data into the FRAME data base, it is important to understand what the data means, how to enter it accurately and reliably, and what processes it will inform. All users have access to a few basic reports within FRAME such as the Foster Care Demographic Report. To extract specific data reports, a work order to the North Dakota Information Technology is required. Once a request is made, a centralized team completes the task. Administrative staff can also extract the data on key performance indicators and build reports. March 31, 2021 Page 37

Continuous Quality Improvement Program Manual Data Sources Information and data from various sources are used to identify problems, solutions, monitor progress, and adjust solutions as needed. These include: • FRAME (i.e., ND’s Child Welfare Information System) • Comprehensive Child Welfare Information and Payment System (CCWIPS) • National Youth in Transition Database • Adoption and Foster Care Analysis and Reporting System • National Child Abuse and Neglect Data System • QA Case Review • Odyssey • Juvenile Court Database • Village Family Services, LSS-ND, contracted vendor data • MMIS, SPACES, Childcare Licensing Data System • Public assistance database (LIHEAP, etc.) Data Source Description Purpose Users FRAME ND CW Info Location for CW data for State/HSZ (CPS,IH,FC)/DJS – FC System Federal/State/County Data Comprehensive reporting State/HSZ/Licensing) CW Info and Statewide System Facilitate payments to foster care Payment System for foster caregiver providers. (Foster parents, adoptive (CCWIPS) licensing data parents, FC Facilities) NYTD National Youth in Federal reporting system designed Federal/State/HSZ/FC/Transition to AFCARS Transition to collect information on youth Independent Living Coordinators Database transitioning out of FC who are (PATH IL) NCANDS served by state agencies Adoption/FC administering the Chafee Program Federal/State/HDZ/DJS/Tribal QA Case Review analysis and Reporting System Federal database for FC statistical Federal/State/HSZ information for analysis of ND Foster National Child care/adoption and Tribal Title IV-E Abuse and foster care Neglect Data Systems Federal data collection system to Qualitative Case examine trends in CA/N across the Record Review nation Measure performance in eligible Federal/State/HSZ (CPS, FC, child welfare cases to ensure IH)/DJS/Tribes adherence to identify Safety/Permanency/Wellbeing strengths, areas needing improvement, and systemic barriers impacting Children/Family outcomes March 31, 2021 Page 38

Continuous Quality Improvement Program Manual Odyssey Case Database for input of legal court State (Court system) Court (Deprivation, Management case information regarding the Improvement Project, State Attorney’s, Guardianship and System for ND state’s legal involvement defense attorneys, GAL TPR) Court System Juvenile court directors, juvenile court Juvenile Court Juvenile Court Database for input of case officers, CIP, juvenile court data Database CMS case management information regarding the state’s analyst. Various system juvenile court involvement State/HSZ/Contractors Contractor Data Contractor Vendor Inform decision making process to Systems data for Village, improve or modify MOUs based on State/HSZ/Eligibility LSS-ND, and feedback on effectiveness of MMIS similar applicable services being provided. Inform State/HSZ/Eligibility parties state as to effectiveness of service SPACES application as it relates to State/HSZ/Child Care Licensing Medicaid contracted services Child Care Management State information system to inform State/HSZ/Eligibility Licensing Information System efficiency of Medicaid services and HSZ its effectiveness to provide services LIHEAP Self-Service Portal to local population Data files kept and Consolidated State information system to inform internal to zone to Eligibility System efficiency of SNAP, TANF, and support entry into other eligibility services and its FRAME CD Licensing Data effectiveness to provide services to System local populations State information system to inform Public Assistance efficiency of Childcare services and Database providers within the state. Provide Zone Specific database for licensing reports. information Collaboration with CPS efforts to systems to assess childcare providers supplement Information system which informs FRAME public assistance access Information systems used by counties to supplement data needs. State does not have direct access to this information nor do all Zones maintain the same data Data Reports Annual Progress and Service Report (APSR) The Annual Progress and Service Report (APSR) is a summary report regarding the engagement of statewide partners in substantial ongoing, and meaningful collaboration in implementing the state’s five- year Children and Family Services Plan (CFSP). This data is aggregated by state, county, agency (HSZ, DJS, or Tribal). Data is disseminated through the Department’s website and sent to Children’s Bureau, Stakeholders (CIP), the State Library, and the Intranet. This information can be accessed by the Department’s website, the Children’s Bureau or by contacting the CQI Administrator. March 31, 2021 Page 39

Continuous Quality Improvement Program Manual Child and Family Services Review (CFSR) Final Report The Child and Family Services Review (CFSR) is a final report submitted to the Children’s Bureau on the progress and status of the Child and Family Support Plan (CFSP). It is aggregated by state, county, agency (HSZ, DJS, or Tribal). Data is disseminated through the Department’s website and sent to Children’s Bureau, Stakeholders (CIP), the State Library, and the Intranet. This information can be accessed by the Department’s website, the Children’s Bureau or by contacting the CQI Administrator. Child and Family Services Plan (CFSP) The Child and Family Services Plan is a five-year strategic plan that sets forth the vision and the goals to be accomplished to strengthen the states’ overall child welfare system. It is aggregated by state, county, agency (HSZ, DJS, or Tribal). Data is disseminated through the Department’s website and sent to Children’s Bureau, Stakeholders (CIP), the State Library, and the Intranet. This information can be accessed by the Department’s website, the Children’s Bureau or by contacting the CQI Administrator. FRAME Agency Open Cases by Supervisor Report The FRAME Report is a report on agency opened cases by the supervisor. Its purpose is to inform HSZ agency supervisors on their worker’s workload and case distribution. It is aggregated by State and HSZ (CPS, IH, and FC). These FRAME Reports listed are some of the reports that are available to anyone using FRAME at any time. FRAME Foster Care Report: Demographic Report This report identifies the total number of children in FC based on selected criteria of birthday, age, and open foster care program dates. This report offers sub-report features to capture information related to youth in need of credit reporting, NYTD surveying, and Chafee Independent Living participation. Data is available from 2011 to present and aggregated by State HSZ for foster care. It is only available for internal use only, unless used for aggregating reporting purposes. FRAME Foster Care Report: Indian Child Welfare Act (ICWA) This report identifies foster youth which are covered under the Indian Child Welfare Act (ICWA). This report identifies the tribal affiliation and the custodial agency of the youth as well as the removal and placement dates of the youth. Note: Youth whose parents voluntarily TPR are excluded from the report. This data is aggregated at the State/HSZ level for Foster care and only available through internal use unless there is need for aggregated reporting. There are some ICWA cases disseminated in Odyssey. FRAME Child Protective Services Report: Cases Pending This report highlights the number of reports pending and the number of reports open longer than 62 days. It highlights workload standards for CPS cases and will be updated to reflect current CPS case timelines for reports. This data is aggregated by State HSZ Child Protective Services and only used internally, unless aggregated for reporting purposes. This report contains no personal identifiable information; it only contains performance measures. This report is accessed annually by leadership and directors for review. March 31, 2021 Page 40

Continuous Quality Improvement Program Manual Tips for Interpreting Data • Understand how the data is gathered and connected to the question being asked. • Evaluate the context and ensure there is clear communication about its impact on the data. • Ask questions about the validity and reliability of the data---- where was the sample drawn, how was it determined, how was it validated? • Gather supporting data around the report being reviewed. • Acknowledge data limitations upfront so they can be better assessed and understood. Data Quality All workers are responsible for ensuring data quality and entering data accurately and on time. Data is entered into FRAME within 24-48 hours of case initiation. Supervisors provide important oversight to ensure data quality. At each data entry point, there should be a clear understanding about key elements including but limited to the following: • Reporting requirements • Sequencing • Case entry notes • Date of face-to-face contact with children at initial CPS stage • Date initiated • Date of collateral contact • Date of case opening • Date of face-to-face visits with parent • Placement moves • Date of removal • Date of court hearings • Submission of report to review for supervision • Date case staffed • Date correspondence was mailed • Date of case closure • Date of case handoffs To validate data entered into FRAME, Supervisors enter a supervision note in the case activity log. A supervisor has the responsibility for the entire data set and provides oversight to ensure the workers are entering the dates accurately and appropriately sequencing their work, so no case is missed. To ensure quality, supervisors review the case before submission and facilitate monthly one-to-one supervision with workers to track updates to the case. Daily and weekly team meetings are facilitated for the purpose of acknowledging case progress using the safety framework. March 31, 2021 Page 41

Continuous Quality Improvement Program Manual Methods to Ensure Data Quality Training: A two-day training for staff is provided to learn responsibilities for the data related to cases and the expectations. All attendees receive a FRAME manual, which contains the necessary information to perform all data entry activities in FRAME. Job Shadowing: Job shadowing with supervision is variable across sites. Specific case circumstance requires hands-on training with CPS cases that go into data systems. Shadowing opportunities are based on maltreatment types. Case Record Review: Case record reviews are performed by Quality Assurance who identifies a sample of cases to select from FRAME. Quality Assurance validates each case by interviewing the worker and supervisor to ensure accuracy. When the reviewer meets with the worker or supervisor, discrepancies are identified and the agency has the responsibility to ensure corrections are made. A report is compiled on the results of the Case Record Review and provided to agencies and the state. Data gathered focuses on the accuracy of demographics, permanency goals, reasons for removal, placement history, permanency hearings, and AFSA TPR requirements. Central Office Data Integrity Review: The CQI Administrator and the Systems Integrity and Reporting Program Manager review AFCARS, NYTD, and NCANDS data monthly to ensure data integrity and compliance with Federal data requirements. Workers are notified of data issues and are required to correct prior to report submission. Data Integrity Inaccurate or incomplete data will prevent accurate identification and interpretation of problems as well as the development of appropriate solutions to resolve them. Ultimately inaccurate or incomplete data will negatively impact the CQI process. The following activities will be used to help increase the integrity of agency data: • The CQI Administrator will review child welfare data – including AFCARS, NCANDS, and NYTD data – monthly to identify data integrity issues. • The CQI Administrator will forward a report of data issues to the field services specialists to allow them to work with the child welfare agencies and workers in their catchment area to correct the data issues. • Child welfare agencies and workers shall correct data issues when identified and implement training activities, individually or organizationally, to raise awareness of the identified data integrity issues and implement plans to rectify. • The CQI Administrator shall report data integrity information to the CQI Administrative Team and the CFS Management monthly to identify trends and possible change projects for data integrity. March 31, 2021 Page 42

Continuous Quality Improvement Program Manual • Provide training to case workers and supervisors on issues surrounding data integrity. (A lot of the issue centers around incomplete data – Training may be a way to improve this) • CRR process within case review process. Checking errors for demographics, etc. and working with case worker to correct. • Collaboration, cross compare data (ICWA applicable child versus what is flagged in court) Case Record Review Data and Process: (Please refer to Section VII Case Record Review for additional information). Analysis and Dissemination of Data A productive CQI system requires a mechanism that promotes circular feedback and communication among staff, stakeholders, and teams. These feedback loops permit an ongoing, multi-directional information exchange across all levels of the agency, which in turn facilitates the change process. Equally important is sharing data with agency staff, consumers, and external stakeholders. By sharing data and information and then using staff, stakeholder, and consumer feedback as a starting point, the agency can create a dialogue about improvements it should make in policies, practices, systems, planning, services, and in its CQI program. Frontline staff, particularly workers and supervisors, will show an increased understanding of how their day-to-day actions, as revealed by data, impact short- and long-term outcomes for children and families, and how their practices can be enhanced as a result. Thus, all system participants should receive information and actively analyze and interpret data, connect data to practice, and identify trends and key findings. Through this process of data-based decision-making, the CQI process is subject to continued examination and evaluation and can be adjusted as needed to better meet agency needs. This ongoing adjustment is one of the key factors in maintaining the momentum of effective systemic change. Ongoing, multi-directional communication will occur through the following: • The CQI Administrator and the State CQI Council will review child welfare system data monthly to identify system issues and develop any needed plans to address them. • The CQI Administrator will review the child welfare system data and recommendations made by the State CQI Council with the Leadership Team of the CFS Division monthly. • The CQI Administrator will submit a report of child welfare system data on a quarterly basis to: o North Dakota Department of Human Services’ Executive Office o Field Services Specialists o Human Service Zone Directors o Tribal Social Service Directors o Division of Juvenile Services Administrator o Other entities as deemed appropriate March 31, 2021 Page 43

Continuous Quality Improvement Program Manual VIII. CASE RECORD REVIEW Purpose: The overarching purpose of the case review process is to support practice improvement to strengthen the North Dakota state child welfare system’s ability to achieve its vision of “Safe Children, Strong Families”. The goal of child welfare is to promote, safeguard and protect the overall well-being of children, to intervene on behalf of abused and neglected children, and to work with children and families to assure that every child has a permanent, safe, and nurturing environment in which to achieve their maximum potential. The QA Case Review is one tool to ensure child welfare services are achieving this outcome for the state. Principles of North Dakota QA Case Review Process The North Dakota QA Case Review model is based on the following central principles and concepts: • The QA Case Review goes beyond compliance monitoring. The QA Case Review intends to gather and assess a range of information on quality with emphasis on implementing needed practice improvements on an ongoing basis. • The reviews examine child welfare services from two perspectives. First, they assess the outcomes of children and families served by local child welfare agencies. Second, they examine identified systemic factors that affect the ability of the state’s child welfare system to help children and families achieve positive outcomes. • Through the reviews, the QA Case Review promotes the state’s Child Welfare Practice which supports positive outcomes for children and families. These principles include family-centered practice, community-based services, individualizing services that address the unique needs of children and families and strengthening parents’ capacity to protect and provide for their children. • The review process collects information from a variety of sources so indicators about the state’s performance can be demonstrated. These sources include the case record; case-related interviews with children, parents, foster parents, caseworkers, and other professionals; and interviews with Tribes, partners, and Stakeholders, as necessary. • The reviews are a collaborative effort between the state and local child welfare agencies (Human Service Zones, Division of Juvenile Services, and Tribal Social Services). • The QA Case Review is one part of the state’s overarching continuous quality improvement approach so that regular measurement of service quality can promote ongoing improvement in outcomes for the children and families served by the state. • The QA Case Review is intended to support the state’s overall collaboration with the Children’s Bureau. When needed, the QA Case Review will provide the process to support the federal Children and Family Services Reviews (CFSR). A foundation principle of the QA Case Review is to review the state’s child welfare service according to the standards for which the state will be accountable to the Children’s Bureau. March 31, 2021 Page 44

Continuous Quality Improvement Program Manual Steps in Case Review Process The QA Case Review process is conducted by Children and Family Service’s Quality Assurance Unit remotely on an ongoing basis in order to meet federal requirements. The general framework for reviews will include the following overall steps: 1. Case Sample: Foster Care cases and In-Home Services cases during a defined period will be subject to a random sampling process. This step includes a case elimination process in which the CFS QA Unit will work with local agencies to identify a final sample of cases, including alternate cases. Case Record Reviews (CRR) are completed on the foster care cases pulled for outreach in the sample. 2. Case Review Preparation: Local agencies will receive an orientation to each review event and resources to aid the preparation of selected cases. 3. QA Case Review: The review process will include a review of the case file and interviews with key case participants for each case, first level quality assurance, an opportunity for the agency and CFS Field Service Specialist to receive preliminary feedback regarding the results, second level quality assurance. Secondary Oversight from the Children’s Bureau will occur as outlined in a Program Improvement Plan (PIP) Measurement Plan. 4. Stakeholder Feedback: The QA Unit will seek input regarding systemic factors through a series of online survey and/or remote exchanges during each review event. 5. Reporting and Sharing of Findings: Cumulative case review data and Stakeholder feedback will be evaluated and compiled into a final report completed by the QA Manager. Results will be submitted to all agencies for use in their ongoing continuous quality improvement efforts. The QA Case Review requires collaboration that focuses on identifying shared goals and activities. Most important, this collaborative process should result in changes that promote improved outcomes for children and families. Core principles guiding this collaborative process include: • The safety, permanency, and well-being of children is a shared responsibility. Local agencies must make every effort to reach out to courts, Tribes, and other partners who can help achieve positive results with respect to the outcomes and systemic factors subject to review. • Local agencies do not serve children and families in isolation. They work in partnership with policymakers, community leaders, and other public and private agencies to improve outcomes for children and families in their states. This includes partnering with organizations that directly serve children, youth, and families, and those whose actions affect family and community life. • North Dakota Child Welfare Practice polies are integral to improving outcomes for children and families. As such, collaboration with families, including young people, is important in identifying and assessing strengths and barriers to improved outcomes for children, youth, and families. March 31, 2021 Page 45

Continuous Quality Improvement Program Manual Workforce Structure for Implementing Case Reviews The QA Case Review process includes many roles which all point to the central value of ensuring that ratings are accurate and consistent throughout the comprehensive review of child welfare cases in North Dakota. The responsibility for well-functioning case review process is shared by the following entities: • Case Reviewer: Case Reviewers have the role of reviewing the case file and assessing casework practice in accordance with the OSRI. As part of the case file review, Case Reviewers will conduct interviews with key case participants, providing an overview of the case findings during a debriefing meeting and sharing preliminary results with the agency. • First Level Quality Assurance (FLQA) Leads: Quality Assurance (QA) Leads have the role of conducting First Level Quality Assurance (FLQA), ensuring accurate, and consistent application of the ORSI instructions during a case file review. QA Leads will be paired with a designated number of review teams to assist and oversee the completion of their teams’ case file reviews. • Second Level Quality Assurance (SLQA) Lead(s): Provide technical assistance to QA Leads to ensure consistent and accurate case ratings. Retain final authority for ensuring that ratings are accurate, which may include changing case ratings when joint resolution during the review cannot be reached. • Secondary Oversight: During any PIP related monitoring periods between ND and the Children’s Bureau (CB), a negotiated percentage of QA Case Review OSRI’s will be identified for Secondary Oversight. • QA Manager: The QA Manager oversees all aspects of the ND QA Case Review. • Research Analyst: The Decisions Support Services Division within North Dakota IT assigns a Research Analyst to provide CFS support for the QA Case Review process. • Supplemental Case Review Workforce: Certain QA Case Reviews, such as supporting a federal Child and Family Services Review, provide a valuable opportunity for community child welfare staff to participate in the state’s CQI process. By bringing Quality Assurance staff and partners providing child welfare services together for the reviews, a strong collaboration is formed to incorporate the quality assurance aspects of the OSRI into local and statewide practice. To best support practice improvement and ongoing CQI connections, Supplemental Case Review Workforce members will be sought from human service zone child welfare agencies, Division of Juvenile Services, AASK, Juvenile Court, CFS, licensed child-placing agencies, QRTP/PRTF staff, Child Advocacy Centers, Guardians Ad Litem, State’s Attorneys, and other community child welfare staff involved in the provision of child welfare services. Community child welfare staff are an invaluable resource during certain QA Case Reviews and the success of quality assurance in our state. March 31, 2021 Page 46

Continuous Quality Improvement Program Manual Competencies for Case Review Workforce Members All Case Review Workforce members will demonstrate the following six competencies (Note - Skill sets under each competency differ depending on the role): 1. Exhibits professional conduct when performing Case Review activities/responsibilities. 2. Works respectfully and in partnership with others as needed. 3. Attends and completes all assigned review activities timely. 4. Exhibits appropriate proficiency with all required technology. 5. Demonstrates adequate knowledge and use of the QA Case Review reference materials. 6. Uses the OMS OSRI and Instructions effectively as applicable to role. Case Review Resources Through the review process, a full understanding of what occurred that affected child and family outcomes in a particular case is paramount. It is critical to obtain information from a variety of sources before making determinations about safety, permanency, and well-being outcomes. The information received from child welfare case files and key case participant interviews is captured using the Onsite Review Instrument (OSRI) on the Online Monitoring System (OMS). Key information sources include:  Child welfare case review file  FRAME data system  Agency case records  Case preparation documents  Onsite Review Instrument (OSRI): A reviewer must complete the instrument based on a review of the case record and interviews with key case participants, which may include the children, parents, foster parents, caseworkers, and other professionals involved with the child and family.  Online Monitoring System (OMS) and OMS Reports Sampling for Case Review The case sample list is a randomized collection of cases from the timeframe of the PUR, which through following a number of critical steps in the sampling process, result in the finalized case sample schedule. These steps include: • Cases Subject for Review and Case Sample Methodology: The case sample methodology will be jointly determined by the Children’s Bureau and CFS QA Manager. The sampling methodology will ensure that all public child welfare agencies (zone, tribal IV-E foster care, DJS) will have an equal opportunity for case review. This will result in a sample representing a cross sector of the state’s child welfare population. The total number of cases will be reviewed on an annual basis and adjusted based on actual caseload data. However, a minimum of 65 cases will be reviewed annually statewide. The Quality Assurance Unit will ensure the appropriate number of metro in-home and foster care (Cass as the admin county) cases in the case review sample. Metro foster care cases can include designated DJS foster care cases. March 31, 2021 Page 47