eBook _ The Decision Maker’s Checklist for Rolling Out End-to-End Assessment Technology (1)

The Decision Maker’s www.taotesting.com Checklist for Rolling   Out End-to-End Assessment Technology  

The Decision Maker’s Checklist for Rolling Out 65% of schools say that End-to-End Assessment Technology leaders at the school have Growing up in the digital age, today’s students are technology natives who expect a high level of sophistication and authenticity from the tools they are using, especially those which have a direct flexibility or leeway in impact on their learning and education. choosing the types and While the education industry has been traditionally slower to adopt technology solutions into the fold, the last few years have seen a rapid uptick in digitization as schools around the world implement amount of learning and EdTech software to support new learning modalities. teaching technology to be This transition to digital assessment in particular has introduced engaging, complex item types that rolled out. 2 provide a more authentic learning experience for students and allow institutions to access rich data that allows them to gain better insight into student performance. In this eBook, we lay out a step-by-step guide for strategizing your digital assessment rollout However, with the implementation of new technology, numerous studies have proven that end-to-end, from vision, to access, to integration the way technology is used and embedded into the learning experience is crucial to its and analysis. effectiveness. What’s Inside The effective implementation of assessment software for institutions requires careful consideration and strategy. Software cannot simply be purchased and rolled out without a well thought out project How to get started creating a project plan, which takes into account the bigger picture and factors like the overall need and vision for the plan for your institution’s eAssessment use of the software, including learning goals, how it will be made available to users, and how it will be rollout, including the questions you implemented and monitored. must be able to answer before moving forward. Mckinsey’s analysis of the 2018 international PISA assessment reiterates these overarching considerations, finding that “globally, students performed better in schools where there were sufficient Considerations for integration with other numbers of devices connected to fast internet service; where they had adequate software and online tools and systems. support platforms; and where teachers had the skills, professional development, and time to integrate digital devices in instruction.”1 The ideal qualities to look for when choosing a digital assessment solution.   How to measure the effectiveness of your implementation. and more.   1 of 13 

The Path Forward Steps for creating a successful assessment technology rollout plan: Many modern educational institutions recognize digitizing assessment as a key component and often a first step in an overall digital transformation strategy. The steps below detail the considerations and actions education leaders need to take to effectively implement a first-phase rollout for digital testing that empowers educators and students to succeed. 1 Set a vision for the need and use of the software: Establish the drivers to make a shift toward digitizing assessment, both within your institution and across the education industry as a whole. As you define an overall vision, you’ll want to address questions, including: o How does digital assessment fit into your institution’s greater transformation plans? o What are the goals and implications for teaching and learning? o What do you need the assessment platform to perform?   2 of 13

2 Analyze existing policies, processes & systems: An early step in the planning process for rolling out assessment software includes taking an inventory of what systems and processes already exist to develop a needs analysis for your institution. You’ll also need to understand any existing policies that may influence your planning, including Acceptable Use Policies (AUP) and student data policies, as well as overarching business policies that could require you to provide additional analysis. There may also be procurement policies in place you are required to follow. At this stage, you’ll need to consider the following: o How is assessment being handled at your institution today? o How will technology improve the process? o Which systems will the new technology need to integrate with? How will they be secured and exchange data? o Are there technology policies in place that can be used as a framework for carrying out your implementation plan? o What challenges might you anticipate? Being able to answer these questions is key to developing a bigger picture for your project plan regarding the need for implementing new assessment solutions.   3 of 13

3 Work with your stakeholders for institutional buy-in: Who is involved in your institution’s process for implementing technology? At which levels will you need buy-in to roll out new solutions? Are there digitization initiatives already taking place at the government level that could be expanded to schools? Stakeholders will differ across varying educational bodies, like K12 districts vs. higher education organizations. It’s important to identify who the internal and external champions for change are within your institution in order to position the goals and benefits of digitizing assessment and ultimately secure a budget for your implementation. Knowing who you will need to involve will help you craft a clear and concise project plan to move your assessment technology rollout forward. Internal stakeholders may include: ministry, school External stakeholders often include: parents, or district leaders, directors of assessment, Chief students, taxpayers, policymakers, and other Technology Officers, Chief Information Officers, head government agencies. of IT, test administrators, school boards, teachers, or anyone else with direct control over assessment within your institution. At the national level in countries like Japan, Lithuania, and 4 of 13 France, the implementation of open source digital assessment for school exams serves as a low-cost gateway to further digital transformation within the education system.  

4 Create a detailed implementation plan and define the key milestones of your assessment program: A successful technology rollout strategy relies on research and evidence, and addresses three major areas: a. Vision: The expectations for the use of the software b. Access: The access and availability of the solution for users c. Integration: The implementation of the software and strategies/milestones to accomplish the vision When implementing assessment technology, a clear project plan should reiterate your research from steps 1 and 2 including: o The drivers for implementing assessment technology Conducting research into how other o The needs analysis for your institution institutions with similar profiles to yours o The vision for the use of the software and what it will need to accomplish have successfully implemented digital o Integration requirements assessment in the past will also provide o Anticipated challenges and solutions valuable insight into what a rollout at o Data security assurances your institution might look like. o The implementation timeline o The goals and milestones for learners and the institution along the way You’ll also need to involve experts on your team to include information that answer questions, like: o What types of assessments will you need to deliver? For example, are your tests meant to be formative, summative, diagnostic, adaptive? Do you deliver open-response writing questions that require online human scoring? Supporting different types of tests and items can require different platform features.   5 of 13

4 o What are your assessments trying to measure and how will they be linked to the curriculum? o Are there any known feature requirements for the platform? If so, what are they. o Who will be using the system and how will they be supported? o Who will be responsible for creating your assessment content? Use this section to map out the o How will assessment data be collected, secured, and analyzed? questions you will need to answer as a o How will hosting be handled and the IT infrastructure be supported? part of your digital assessment software implementation plan. o Will the assessments be taken remotely or onsite? o How will you ensure student equity? o How will you ensure accessibility? o What is the expected impact on your institution and how will it be measured? o How and when will you evaluate the impact your technology plan implementation has on student performance? o What would be considered a successful rollout for your institution? o What are the budget constraints?   6 of 13

4a Define system requirements and user volumes: As part of your digital rollout plan, it’s crucial to define the system requirements based on the goals of your assessment campaigns, as well as the volume of users that the testing platform is required to support. There are several user metrics you’ll need to anticipate to ensure the scalability of the system, which can also affect costs: o Will your assessments be delivered year-round or during specific testing windows, and how will test-takers be distributed during these campaigns? o Expected number of test-takers and simultaneous test-takers o Expected number of exams o Expected number of authors and back-end users Once you answer the questions surrounding the capabilities your system will need to support, you’ll want to consider several features of an ideal testing solution.   7 of 13

The Hallmarks of an Ideal Testing Solution: Interoperability & Integration “One-size-fits-all” no longer applies to modern EdTech learning tools. Assessment software must be scalable and secure; it must also work with both the mandated and innovative needs of your school, administration, teachers, and students. To meet these requirements, all new software within your stack should support integrations that enable next-generation learning outcomes — from blended learning courses to student accommodations. Interoperable assessment software removes data silos to allow seamless integration and data exchange between educational content, testing platforms, and learning applications – enabling schools to do more with the technology in their stack. Open Standards True interoperability is established by ensuring that all new and existing technology follows the same open standards, like the ones set by the IMS Global Learning Consortium. Question and Test Interoperability (QTI) and Learning Tools Interoperability (LTI) are two IMS open standards to look for in an assessment platform that ensure the exchange of your content, tests, and data between learning tools. QTI LTI The IMS Global QTI specification enables student and assessment The LTI standard serves as the foundation for rich integration data to feed into item and test content — and vice versa. QTI between assessment platforms and learning and content creation enhanced items and Portable Custom Interactions (PCIs) within tools, like a Learning Management System (LMS). The specification the digital assessment provide a more engaging testing and ensures that all external applications are available from a single- instruction experience and more accurately measure 21st- sign-on point in the LMS, and aims to preserve all learning Century Skills. context and user roles between all tools and applications.   8 of 13

Open Source Accessibility Traditionally, when a school district adopts proprietary software for its core infrastructure, it can easily Natively accessible assessment software become locked into one solution or vendor, which greatly reduces choice and flexibility. Since educational guarantees compliance with the latest policies, goals, and strategies change from time to time, flexible open infrastructure can better enable accessibility standards for your testing you to offer effective solutions and maintain control of your ecosystem without getting locked in. Rather programs and helps you avoid many than being limited by the features offered in a proprietary solution, licensed from a traditional vendor, a common technical pitfalls for users. true open source offering allows institutions to freely access and modify the source code to create the More importantly, investing in accessible system they really need; a solution that works well with other EdTech solutions within the overall existing solutions helps promote digital equity ecosystem. by expanding learning opportunities for students of all abilities. An accessible Every district has a unique combination of students, styles of teaching, and operating models. Proprietary assessment format means they can software is usually packaged to fit a generalized need, which naturally leads to a high acquisition cost. receive appropriate accommodations in a non-intrusive, fully integrated way. While open source software is free of licensing fees, there are associated costs including the technical expertise to implement and configure it, as well as deploy and support it. However, for many agencies, Common education accessibility tools, having this level of control along with the ability to change and add to their digital infrastructure as needed like magnification and font adjustments brings a huge benefit, especially when they tap the services offered by the developers of the open source (size, colors, spacing), allow students system. to access and respond to assessment content. Additional tools may include: Scale • Text-to-speech Interruptions in large-scale, high-stakes testing can have significant consequences for schools and • Highlighters students alike. If the assessment system crashes and student data becomes compromised, an entire • Color Contrast institution can risk losing hundreds of thousands of dollars to retesting and damage control. • Screen Readers • Answer Masking An effective assessment system must be able to handle fluctuating volumes of test deliveries and test- • Choice Striker takers without disruption. A cloud-based solution that is able to scale up and down to support hundreds • Calculator with keyboard and to hundreds of thousands of users significantly improves the test-taking experience by eliminating risks of deployment failure. At the same time, the ability to scale the solution infrastructure down outside of your text-to-speech accessibility testing windows can help reduce overall hosting costs. • Notepads   9 of 13

Data Security & Privacy Automated & Manual Scoring A fully integrated EdTech data solution is useless without stringent data safety Different assessment questions can often require different scoring and security protocol, potentially jeopardizing not only student privacy but mechanisms depending on what they are trying to measure. For online data security of all parties operating within the school system. exams that use simple question types like multiple choice or matching, setting up scoring workflows that mark your students’ work automatically Interoperability standards such as the LTI®, based on the IMS Security can help streamline exam management. Not only does this provide a framework, ensure your assessment software investment is optimized for data better experience for students – who can find out how well they did on an security. When the tools within your ecosystem follow the IMS specification, assessment instantly – but it also cuts down on the amount of admin you all messages and data are secured, easily managed, and protected from have to carry out afterwards. When creating open-ended exam questions, vulnerabilities. online manual scoring tools (often called human scoring) simplify the grading process. Here are the functions of an LTI® compliant application: Linked to Instruction & Curriculum • Control the sharing of personal data with an “off” The software you choose must allow you to link your assessments directly function. to the curriculum and instruction. Many modern curricula now require the measurement of 21st Century Skills like problem-solving and collaboration, • Personal data sharing is only allowed when backed by which have been traditionally difficult to capture in assessment. However, safeguards to guarantee continued data privacy. an assessment platform that supports multi-step PCIs is able to collect richer data to provide insight into these student competencies. • The users have the ability to review the privacy controls at any time. By following the data security standard, the assessment software you select will be optimized to operate native tools and cloud-based applications with the highest level of security in data, location, and role.   10 of 13

5 Work with a trusted technology partner 7 Implement, integrate, iterate and innovate Transitioning assessments from paper to digital can be challenging. To mitigate the risks and ensure a smooth transition, it’s crucial to choose the right technology partner for support, Rolling out technology is a continuous including onboarding, training, and account management. Dedicated training and support process that requires ongoing evaluation for users is critical to ensure access to the solution. and must be adapted as your institution’s needs inevitably evolve. 34% of schools say that lack of support on how to use technology for Tips for data analysis: teaching is a moderate or significant challenge.2 o Anticipate new data collection and An open source assessment solution like TAO is highly recommended for its many benefits, know what to do with it. including: o Analyze the data quality to exclude • 100% interoperability for seamless integrations with existing or new bad data – such as practice test technologies. responses – that will pollute the analysis. • The ability to pick and choose the exact components or features you need and package them into you system of choice, reducing the overall cost of o Be aware of the technical aspect of ownership. the data and how to work with the data format • TAO has repeatedly demonstrated and proven its scalability in large deployments worldwide. 11 of 13 • TAO is a cornerstone technology of the FLIP+ eAssessment community. See https://flip-plus.org 6 Develop technology readiness for learning When rolling out assessment technology at both the classroom and national level, school preparedness cannot be overlooked. Understanding the existing limitations of the current systems in place (if any), and what a school’s or student’s home infrastructure will need to handle can eliminate many of the IT challenges that can emerge with the introduction of new software.  

Conclusion About TAO Digital assessment software represents an evolution in how we teach and how our students learn. TAO, from Open Assessment While new technologies undoubtedly unlock opportunities for improvements in teaching, learning, Technologies, is the leading digital and assessment, it’s not enough to expect that simply adding new tools will solve every challenge. assessment solution for education Successfully rolling out new assessment technology in schools hinges on planning, research, and and career advancement. Modular, understanding teacher and student needs. When rolling out new technology, it’s important to define customizable and interoperable by expected impact, create a detailed project plan, and set learning goals, choosing technology that design, TAO empowers you to break free allows you to align learning to the curriculum. from the confines of proprietary silos, eliminate expensive licensing fees, take Is your school or organization ready for a next-generation digital assessment solution? full control of your testing resources The innovative TAO Testing digital assessment authoring and delivery platform may be the simplest, most transformative choice for you. Our tech and best practices are based on IMS Global interoperability standards — ensuring virtually endless integration options and optimal control for users. Schedule a consultation with one of our specialists to learn about how you can leverage TAO to evolve with today’s learners! Contact Us Sources: 12 of 13 1 New global data reveal education technology’s impact on learning Bryant et al. https://www.mckinsey.com/industries/ education/our-insights/new-global-data-reveal-education-technologys-impact-on-learning 2 Gray, L., Lewis, L., & Chapman, C. (2021, November). Use of Educational Technology for Instruction in Public Schools: 2019—20. Retrieved February 4, 2022, from https://nces.ed.gov/pubs2021/2021017.pdf  

Additional Resources Our case studies and white papers are developed with assessment technology experts to demonstrate practical applications of digital assessment and the power of an open solution. PRODUCT ACCELERATING INNOVATION IN DIGITAL ASSESSMENT Three TAO editions, one powerful platform. Download our product sheet to learn more about TAO’s digital assessment platform solutions for assessment authoring, delivery, rostering and reporting, and find the product tier that’s right for your organization. CASE STUDIES Transitioning an Item Bank from Paper to Digital with The Finnish Matriculation Examination Board (MEB) TAO Addresses the Unique Assessment Needs of Students with Cognitive Disabilities Lithuania’s National Agency for Education (NSA) Finds an Effective Path from Paper to Large-Scale Online Testing TAO Helps Ceneval (Mexico’s National Center for Higher Education) Upgrade Their Assessment Program New York City Department of Education (NYCDOE) Adds Technology and Multimedia to Language Proficiency Exams Leveraging Portable Custom Interactions to Foster the Link Between Teaching and Learning in France INVALSI Addresses the Learning Loss Caused by COVID-19 in Italy Elsevier Incorporates TAO with Ease, Gauging Career Readiness in Healthcare Fields WHITE PAPERS Revolutionize Standardized Testing with Open-Source Software Next Generation Science Standards (NGSS): Opening the Possibilities for Deeper Learning and Assessment TAO Accessibility Conformance Report | WCAG 2.1 AA The Importance of IMS Certification and QTI Compliance The Practical Guide to Successful Large-Scale Online Assessment Three Tangible Ideas for Engaging Assessment Content   13 of 13