Extended Reality and Metaverse - Immersive Technology in Times of Crisis

["194 N. Rihani Fig. 13. Assembling and matching two photogrammetric segments of a Corinthian column of the Southern Decumanus (C3, C4) and (C5, C6) by stone mark matching All assembled column segments were grouped and managed using Meshmixer soft- ware to recreate destroyed columns of the Southern Decumanus (Fig. 14). The exact methodology of assembling and grouping reconstructed columns was implemented to create complete column \ufb02anks, which were reconstructed virtually from multiple assembled columns and beams (Fig. 15). The entire composition of this monument was imported in a 3D virtual environment developed using the UE game engine, to be experienced through the Oculus Quest VR headset system discussed in the following section.","An Interactive and Immersive Walk-Through Experience 195 Fig. 14. Process of assembling the column segments to create complete columns composition 4.4 3D Visualisation and Remote VR Experience Phase Experiencing the 3D VH environment requires allowing the visitor to navigate freely, moving, and attaining a level of interaction with the environment and its content from artefacts, monuments, and virtual characters. In this research, the focal objective is to allow the visitors to explore the 3D reconstructed environment of the Southern Decumanus by approaching the monument, walking among the reconstructed parts, and \ufb01nally interacting with reconstructed \ufb02anks and column segments through viewing some simple related interactive virtual information and playing simple games to increase their interaction during the experience. 4.4.1 The VR Experience Setup and Implementation All the assembled digital creations were imported to UE to build the virtual setting and manage all the imported assets, in order to develop the required interactive experience. In the post-processing phase, the researcher approximately halved the photogrammetric models\u2019 polygons, to avoid any possible glitches that may arise during the VR experience. The interaction inside the virtual environment requires setting a collision for each object; this increases realism for the visitor during the experience (Fig. 16).2 2 Without collision, the user would be able to walk through the mesh and if the mesh simulates physics, the mesh will simply fall through the world as soon as the experience began.","196 N. Rihani Fig. 15. Process of assembling multiple 3D photogrammetric Corinthian columns and beams to create the Southern Decumanus \ufb02anks A key part of the process deals with the programming of the visitor movement and circulation inside the virtual environment of the reconstructed Sothern Decumanus, showing virtual data and related animations in the UE game engine. Before starting the virtual tour, visitors should be offered an instruction sheet showing the UI and instructions for using the VR joysticks during the experience, for making actions such as moving,","An Interactive and Immersive Walk-Through Experience 197 Fig. 16. Screenshots from the UE game engine illustrating the process of developing the VR interactive experience interacting, displaying the visual data, and virtual annotations (Fig. 17). Visitors will be requested to wear the VR headset to start viewing the 3D virtual environment of the Southern Decumanus and starting the interactive immersive experience. Fig. 17. VR joystick controller UI for the visitor interaction and the used VR Oculus Quest VR system The control of the experience is essentially available via the VR headset and joysticks controllers\u2019 interaction. However, it is also accessible for those who prefer to live the experience with a minimum range of immersion through the monitor-based display and mouse keyboard interaction. Visitors would \ufb01nd themselves inside a fully interactive","198 N. Rihani immersive VR experience, in which they can freely look around using the 360\u00ba movement with a high range \ufb01eld-of-view (Fig. 18). Fig. 18. Testing the virtual environment of Southern Decumanus using the VR headset The experience aims to provide future virtual visitors with an unprecedented oppor- tunity to visit this monument, which has never been seen in the physical world in its intact condition, and to explore certain virtual information related to its history and architectural and structural components from a real person\u2019s perspective and interaction scheme. For this purpose, the controlling positions were determined by the possibility of free movement by the visitor. Furthermore, for bridging long distances in the 3D virtual environment of the Southern Decumanus, certain teleportation and locomotion \u2018Point and Teleport\u2019 and \u2018Fly-through\u2019 functions are available for the visitor\u2019s navigation (Fig. 19).","An Interactive and Immersive Walk-Through Experience 199 Fig. 19. \u2018Point and Teleport\u2019 and \u2018Fly-through\u2019 teleportation methods of visitor movements inside the virtual environment of Southern Decumanus The highlight of the VR experience is exploring the reconstructed Southern Decumanus, which is vividly and virtually represented by the Oculus Quest VR glasses directly in front of the visitors\u2019 eyes, on the basis of an interactive 3D environment and assets. The visitors\u2019 hands, via joystick controllers, can interact with various virtual environment components to control the 3D photogrammetric model\u2019s representations. Visitor movement, object selection, and virtual information boxes are enabled via a \u2018laser beam\u2019 controlled by the joystick controller (upon visitor request). Different descriptive architecturally and historically con\ufb01rmed virtual information is visibly demonstrated to the visitor in the 3D environment, illustrated by transition animations. Visitors would have the ability through the joystick controller to select any part of the Southern Decumanus and move it by a click and drag feature. They have the option to disassemble the entire column \ufb02ank and bring it back to its original status before construction, in order to view the process of the construction of the Southern Decumanus from its architectural components (columns and beams). Visitors can interact with such movable parts and view all the related information of the 3D photogrammetrically created environment from all sides, using virtual rotation of the VR system (Fig. 20). In addition, visitors can click to hide or unhide any selected object upon their request and can rearrange the column segments on their own way. Fig. 20. An example of visitor interaction with the digitally reconstructed Southern Decumanus \ufb02anks To increase the level of immersion the entire 3D virtual environment of Jerash and the whole setup of the surrounding area was created using the UE game engine. This can put the visitor inside a simulated virtual world similar to that exists in the physical","200 N. Rihani world of the actual site of Jerash. Such an environment contains various layers of 3D low polygonal assets, e.g., contours, trees, rocks, green surfaces, and landscape, in addition to the created photogrammetric models (Fig. 21). Fig. 21. Process of creating a simulated 3D environment for the real archaeological site of Jerash using the UE game engine 4.4.2 The Virtual Puzzle and Increase Visitor\u2019s Interaction Serious games and gami\ufb01cation offer an interactive and experiential relationship with the cultural heritage domain by disseminating new attractive experiences to different users and within different scenarios (Malegiannaki and Daradoumis 2017). In order to increase visitor interaction and engagement within the virtual environment and its assets, it is proposed to import some simple games and puzzles within the experience. Accord- ing to Anderson et al. (2009), although games are still not completely examined for their pedagogical capability by professionals working in heritage studies, implementing such simple games can facilitate knowledge-seeking through interactivity, and encour- age user participation and sustain their contributions. Applying the main principles of gami\ufb01cation and serious games in immersive heritage experiences can attract visitors to proceed with their experiences by offering them a feeling of playing a game, with some related quizzes, puzzles, checkpoints, and speci\ufb01c objectives to accomplish. There are several areas of great potential for utilising computer games in the VH realm, as they are low-cost, attract users from different ages and backgrounds, and could be designed as open-ended learning experiences (Champion 2016). This is commensurate with the necessity to implement new methods for attracting and encouraging visitors to expand their visits and spend more time in such affected heritage environments in the Middle East, particularly in the current time of COVID-19 in\ufb02uences. A simple \u2018virtual puzzle\u2019 game was developed during this research to attract more visitors (particularly younger ones) to accept this technology and facilitate their experi- ences in the future. Younger visitors of the virtual Southern Decumanus would have the opportunity to interact and play with the created column segments, as the game requires them to collect the scattered \ufb02ank segments from the ground and assemble them again virtually, through a sequential order; this game can be played by multiple users (Fig. 22). The virtual puzzle was tested with a number of student testers at University of Liv- erpool who have given this research informal feedback during the development. The purpose for developing such interactive games that can signi\ufb01cantly increase visitors\u2019 engagement and interaction during their virtual-based experiences, as it changes visi- tors\u2019 role from being simple static viewers to be active players, play a part inside the","An Interactive and Immersive Walk-Through Experience 201 Fig. 22. The proposed virtual puzzle to increase visitors\u2019 interactions with the virtual environment of the Southern Decumanus experience. The users of inatercative games in such cases would learn differently for instance the components of the Cornthian columns and Roman architecture than reading a book or watching a \ufb01lm documantry (Fig. 23). 5 Research Pipeline Fig. 23. Entire research pipeline 6 Conclusion This paper proposes structured work\ufb02ow and methodology of \ufb01eldwork-based research that illustrates the pipeline of capturing, reconstructing, and visually assembling fallen, damaged, and neglected columns of what is today an important visitor attraction and archaeological site. It should be noted that during the work\ufb02ow of this study, the process of capturing photos and reconstructing and editing the 3D digital outcome was done in substandard conditions. The required photos were taken directly on the site by a consumer digital camera, and not in ideal situations (in terms of the column segments\u2019","202 N. Rihani positions and the lighting conditions). In Southern Decumanus of Jerash, as in all other similar sites, historical artefacts cannot be moved from their original locations. There are of course heritage preservation considerations as well as a number of practical reasons, such as the massive weight of each segment, which prevent the moving of such artefacts to a studio or other settings that would yield improved results. The researcher proposed a photogrammetric reconstruction pipeline that could rel- atively easily be applied to other sites and locations in Jerash and Jordan. In terms of heritage documentation, the proposed technological process has offered an effective low- cost method for digital documentation that offer 3D photorealistic representations for our lost physical heritage assets. Moreover, the 3D digital assets can be used ef\ufb01ciently for developing new engaging experiences using immersive technology Augmented Reality (AR) and VR applications, for either in-situ or ex-situ installations. Fig. 24. Testing the experience visitor\u2019s interaction using Oculus Quest VR system Notably, many heritage sites in Jordan and the Middle East contain a plethora of abandoned artefacts that desperately need to be preserved and disseminated to the public. Digitising heritage artefacts and creating digital replicas allows bequeathing such sites and their valuable content to future generations, to avoid absolute loss of heritage due to future threats. This project\u2019s work\ufb02ow and the outcome of VR developed experience could offer different heritage institutions and sites in Jerash and other locations in Jordan a solid springboard for creating new interactive experiences and applications. Such experi- ences and applications can potentially enhance visitors\u2019 off-site virtual visits to indoor museums and exhibitions and attract them to explore virtually different scenarios for non-existent monuments in the real-world environment (Fig. 24). This experience could be tested and visualised by the visitors of Jerash Museum in the future by wearing a VR HMD system or using CAVE technology, to enjoy walking around and interacting with a complete photorealistic 3D representation for one of Jerash\u2019s main destroyed mon- uments, the Southern Decumanus, for the \ufb01rst time. Therefore, this work contributes","An Interactive and Immersive Walk-Through Experience 203 a practice-based research contribution for capturing, reconstructing, and assembling a 2000-year-old monument virtually, developed entirely in 3D. The VR experience in this form is unique in Jordan and presents an informative component of a virtual heritage site. The VR installation using the new advanced systems such as Oculus Quest offer the opportunity to see and closely check the photogrammetric models\u2019 quality during the VR visualisation. Moreover, enabling visitors to walk through the virtual environment of the Southern Decumanus, get a remotely engagement, collecting all of the virtual data, and viewing the artefacts in their original state is considered a ground-breaking contribution in terms of virtual tourism implementation using immersive technology and its applications in Jordan, and particularly for the Jerash heritage site. Fig. 25. Proposed on-site experience using mobile AR installation to be implemented on the physical heritage site of Jerash and Southern Decumanus Furthermore, the emerging technologies that bridge the virtual and physical worlds in the AR context provides a proposed proposition for (re-)experiencing heritage, allow- ing visitors of Jerash to view the Southern Decumanus standing again on its real original location, in the heart of the city. It also allows visitors to get a locative engaging, interac- tive experience using any hand-held device or a smartphone for virtual superimposition of the current state with an ancient building state, for instance, viewing the process of reconstructing the Southern Decumanus virtually (Fig. 25). In closing, an excellent opportunity can be seen in this research to highlight the importance on bringing together expertise from different interdisciplinary domains, such as architects, historians, 3D game developers and heritage site managers and involve them in such collaborative practice-based real projects and measure its impact on the ground. The results of this experiment could shed light into developing a proposed structured low-cost pipeline that can assist other researchers in the \ufb01eld of virtual heritage, digital documentation and representation to \ufb01nd new applications for digitally reconstructing other affected heritage sites in Jordan and the Middle East. Furthermore, this work can potentially offer heritage institutions concerned about monument restoration in Jordan a virtual preconceived visual restoration platform that can assist them before starting physical restoration works.","204 N. Rihani Acknowledgements. I would like to acknowledge the debt I owe to my colleagues at CAVA, the School of Architecture, University of Liverpool, for their intellectual support and warm friendship during this research. My gratitude also goes to Professor Richard Koeck and Dr Nick Webb for the opportunities given to me and all the support throughout this process. References Abdelmonem, M.G., Selim, G., Mushatat, S., Almogren, A.: Virtual platforms for heritage preser- vation in the middle east: the case of medieval Cairo. Int. J. Architect. Res. ArchNet-IJAR 11(3), 28\u201341 (2017) Alqahtani, A.S., Daghestani, L.F., Ibrahim, L.F.: Environments and system types of virtual reality technology in STEM: a survey. Int. J. Adv. Comput. Sci. App. (IJACSA) 8(6), 1\u201313 (2017) Anderson, E.F., McLoughlin, L., Liarokapis, F., Peters, C., Petridis, C., de Freitas, S.: Serious games in cultural heritage (2009) Atkinson, K.B.: Close Range Photogrammetry and Machine Vision. Whittles, Scotland (1996) Boboc, R.G., Duguleana\u02d8, M., Voinea, G.D., Postelnicu, C.C., Popovici, D.M., Carrozzino, M.: Mobile augmented reality for cultural heritage: following the footsteps of Ovid among different locations in Europe. Sustainability 11(4), 1167 (2019) Champion, E.: Entertaining the similarities and distinctions between serious games and virtual heritage projects. Entertain. Comput. 14, 67\u201374 (2016) Creed, C., Sivell, J., Sear, J.: Multi-touch tables for exploring heritage content in public spaces. In: Cheng, E., Gaffney, V., Chapman, H. (eds.) Visual Heritage in the Digital Age. SSCC, pp. 67\u201390. Springer, London (2013). https:\/\/doi.org\/10.1007\/978-1-4471-5535-5_5 El-Hakim, S.F., Beraldin, J.A., Picard, M., Godin, G.: Detailed 3D reconstruction of large-scale heritage sites with integrated techniques. IEEE Comput. Graphics Appl. 24(3), 21\u201329 (2004) Flavi\u00e1n, C., Ib\u00e1\u00f1ez-S\u00e1nchez, S., Or\u00fas, C.: The impact of virtual, augmented and mixed reality technologies on the customer experience. J. Bus. Res. 100, 547\u2013560 (2019) Girbacia, F., Butnariu, S., Orman, A.P., Postelnicu, C.C.: Virtual restoration of deteriorated reli- gious heritage objects using augmented reality technologies. Eur. J. Sci. Theol. 9(2), 223\u2013231 (2013) Jones, A.H.M.: Inscriptions from Jerash. J. Roman Stud. 18(2), 144\u2013178 (1928) Kalay, Y., Kvan, T., Af\ufb02eck, J.: New heritage. New Media and Cultural Heritage. Routledge, London; New York (2008) Kysela, J., \u0160torkov\u00e1, P.: Using augmented reality as a medium for teaching history and tourism. Proc. Soc. Behav. Sci. 174, 926\u2013931 (2015) Maiwald, F., Vietze, T., Schneider, D., Henze, F., M\u00fcnster, S., Niebling, F.: Photogrammetric anal- ysis of historical image repositories for virtual reconstruction in the \ufb01eld of digital humanities. Int. Arch. Photogr. Remote Sens. Spatial Inf. Sci. 42, 447 (2017) Malegiannaki, I., Daradoumis, T: Analyzing the educational design, use and effect of spatial games for cultural heritage: a literature review. Comput. Educ. 108, 1\u201310 (2017) Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. 77(12), 1321\u20131329 (1994) Noh, Z., Sunar, M. S., Pan, Z.: A review on augmented reality for virtual heritage system. In: Chang, M., Kuo, R., Kinshuk, Chen, GD., Hirose, M. (eds) Learning by Playing. Game-based Education System Design and Development. Edutainment 2009. LNCS, vol. 5670, pp. 50\u201361. Springer, Heidelberg (2009). https:\/\/doi.org\/10.1007\/978-3-642-03364-3_7 Rebelo, F., Noriega, P., Duarte, E., Soares, M.: Using virtual reality to assess user experience. Hum. Factors 54(6), 964\u2013982 (2012)","An Interactive and Immersive Walk-Through Experience 205 Sherman, W.R., Craig, A.B.: Understanding Virtual Reality. Morgan Kauffman, San Francisco (2003) Tschirschwitz, F., B\u00fcy\u00fcksalih, G., Kersten, T.P., Kan, T., Enc, G., Baskaraca, P.: Virtualising an Ottoman fortress\u2013laser scanning and 3D modelling for the development of an interactive, immersive virtual reality application. Int. Arch. Photogr. Remote Sens. Spatial Inf. Sci. XLLII- 2\/W9, 723\u2013729 (2019) UNESCO: UNESCO World Heritage List (2019). https:\/\/whc.unesco.org\/en\/list\/. Accessed Mar 2022 Wheeler, M.: Roman art and architecture (1964)","A Study on the Application of Historic Building Information Modeling (HBIM) for XR Cultural Heritage Tourism Jongwook Lee(B) and Boram Kim Department of Cultural Heritage Industry, Korea National University of Cultural Heritage, Buyeo-gun, South Korea [email protected] Abstract. We would like to propose a plan to apply historic building information modeling (HBIM) for XR heritage tourism. It is signi\ufb01cant that the XR heritage tourism content expresses not only a visually detailed image but also the value of heritage and its historical information together to convey the authenticity of the heritage to end-users. Such historical information has been managed in inte- grated forms using the HBIM methodology, but there is a few researches on the methodology using it as XR content. Therefore, we de\ufb01ne the detailed stages of the level of historical information and propose an HBIM system to apply them to XR heritage tourism. Since HBIM has the following characteristics, it can contribute to building an XR heritage content. First, it can be used to visualize the complexity of built her- itage in an XR environment. Second, HBIM-based XR tourism content contains contextual information of cultural heritage, which can provide authentic informa- tion that tourists want to obtain while visiting the heritage sites. Third, it helps to understand heritage by providing integrated heritage information. Fourth, contin- uous XR tourism is possible through the management of user-generated content and user information generated in experiencing heritage. This paper will contribute to delivering the value of cultural heritage to end- users through XR heritage tourism content. Keywords: HBIM \u00b7 Digital heritage \u00b7 Heritage tourism \u00b7 XR content 1 Introduction We can largely divide the purposes of using HBIM into three categories. First, HBIM contributes to heritage management through monitoring and repair by 3D visualisation of the complexity of built heritage. HBIM can provide various materials related to the preservation and management of historic architecture. HBIM can show problems of the building in three dimensions, making it easy for repair workers to check the location of the structural issue. In addition, it supports management of records of 2D and 3D built heritage as well as repair reports, monitoring reports, and regular survey reports. To make this possible, 3D modeling technology has made it possible to make it \u00a9 The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 T. Jung et al. (Eds.): XR 2022, SPBE, pp. 206\u2013216, 2023. https:\/\/doi.org\/10.1007\/978-3-031-25390-4_18","A Study on the Application of (HBIM) 207 three-dimensional models in a virtual environment. This allows experts to identify and diagnose problems even if they do not go directly to the \ufb01eld through virtual reality and network connection. Managers refer to the monitoring of built heritage and identify the cause of the problems using HBIM. To make this possible, metadata and database-related technologies were used. Second, HBIM contributes to tourism and education by providing contextual infor- mation related to buildings. HBIM allows users to share historical facts, videos, digital content, and information on visitor\u2019s appreciation related to historic architecture. To this end, users can receive desired information according to their individual age, interests, and prior knowledge. To this end, customized content service technology is used. In addition, mobile experience technology will be used to integrate digital heritage con- tents in museums, historical sites, and homes. In addition, digital twin technology is used to integrate users\u2019 information and share their appreciation with the community and curators. Third, HBIM is used to digitally restore non-existent buildings or destroyed parts. HBIM supports the construction of parametric model libraries that model by period through the design of a component library of built heritage. To make this possible, parametric modeling technology is used. Content creators can easily and quickly create historical cities or building groups to be used in games and movies by utilizing these characteristics of HBIM. 2 Methodology As a result of collecting and analysing the use cases of HBIM, we could classify the application of HBIM into three categories. \u2022 Visualisation of the complexity of built heritage and integration of heritage information for heritage management \u2022 Providing contextual information for tourism and education \u2022 Digital restoration of built heritage for creation of digital content (Fig. 1)","208 J. Lee and B. Kim Fig. 1. Research structure 3 Visualisation of the Complexity of Built Heritage and Integration of Heritage Information for Heritage Management Built heritage is a historical feature composed of various elements, various materials, and traditional techniques to implement them, unlike general cultural heritage such as stone statues and ceramics. Built heritage has different forms in each region due to cultural diversity, and it is dif\ufb01cult to visualize complex structures by applying digital technology. The digitisation of heritage improves the integrity of cultural heritage management, and for this, an integrated management system of the data must be accompanied. HBIM is the application of building information modeling to historical buildings and enables information management for each member unit. In particular, it can be proposed as a solution for cultural heritage information management, and it brings innovation to heritage management by establishing an integrated data set. HBIM not only has advan- tages in terms of managing cultural heritage information, but also plays an important role in helping to express the authenticity of heritage. UNESCO speci\ufb01es \u2018materials and substances\u2019 as the authenticity of cultural heritage, and many scholars have studied the importance of the correlation between cultural heritage and its materials. Management and expression of member units using HBIM has an advantage in the production of XR content for tourism because it can convey the authenticity of the cultural heritage to visitors in a virtual space. In summary, HBIM functions as a quantitative indicator of value evaluation, inte- grated information management, and visualisation of built heritage. The three functions","A Study on the Application of (HBIM) 209 express the authenticity of cultural heritage when producing XR content based on her- itage and enhance user interaction by providing contextual information. Moreover, the managers working in the \ufb01eld of heritage can utilize HBIM to promote tourism through economic value assessment (Fig. 2). Fig. 2. HBIM overview for heritage management (Jordan-Palomar et al. 2018) 4 Providing Contextual Information for Tourism HBIM is used when evaluating the value of heritage. This means not only expressing the authenticity of cultural heritage, but also quantitatively showing the material properties such as spatial information for an objective evaluation. Rodr\u00edguez (2020) applied HBIM for the \ufb01rst time in parallel with the conventional valuation method in the valuation of cultural heritage and quanti\ufb01ed the historical value of the target architectural heritage in terms of materials and area, thereby increasing the rationality of valuation. Also, Argiolas et al. (2020) examined HBIM as a tool for tourism and produced a prototype for the Complex of Santa Croce in Cagliari. They collected spatial data and semantic data and modelled the BIM environment and external tools. Such a project using BIM technology helps people to enjoy and learn heritage regardless of distance and time. Since heritage is a historical evidence, its educational value is important. Therefore, HBIM can contribute to providing content for history education in virtual reality.","210 J. Lee and B. Kim In addition, interaction with cultural heritage makes visitors aware of its identity and helps them to feel realistic tourism. Therefore, HBIM has a strength in expressing the complexity of built heritage, and it is meaningful in creating content that visualizes each member unit and expresses it in detail. This can be classi\ufb01ed step by step into LOD (Level of Detail) indicating the detailed expression of the member and LOI (Level of Information) denoting the stage of providing information on the member itself (see Sect. 6) (Fig. 3). Fig. 3. TransHBIM project implementation work\ufb02ow and virtual tour using Matterport (Heesom et al. 2021) 5 Digital Restoration of Built Heritage for Creation of Digital Content HBIM has strengths in various industries that produce digital content because cultural heritage can be restored in virtual space. Smelik et al (2014) researched that the modeling in virtual worlds can be utilized in movies, games, and simulations. Also, Bagnolo et al (2021) examined the ways to utilize HBIM and suggested work\ufb02ow to create a serious game through 3D models. It was proved that HBIM has a positive impact on cultural and educational problems in COVID-19 situation. HBIM can digitally restore an existing building with high de\ufb01nition, and the data generated from it can be used in movies and drama productions by providing CG graphics","A Study on the Application of (HBIM) 211 that require speci\ufb01c background. We believe that HBIM can be expressed \ufb02exibly in the XR environment, which signi\ufb01es a variety of scalability of HBIM. This means that additional research and information collection is not required when used in other industries as it contains information about building. In the production of dig- ital content, HBIM will help people use their time and resources ef\ufb01ciently and will pro- vide them with modeling that can be changed according to the production environment (Fig. 4). Fig. 4. The point cloud of HBIM can be utilized in games and\/or movies (Bagnolo et al. 2021) 6 Discussion (XR tourism based on HBIM) 6.1 Providing Contextual information XR tourism based on HBIM is different from modeling in common virtual space in that it is possible to provide information in units of members. The standard used to express its de\ufb01nition follows the Level of Detail (LOD). LOD is de\ufb01ned differently depending on the guidelines between countries or industries. In the UK, according to the AEC (Architecture, Engineering & Construction) industry BIM guidelines, LOD is divided into 6 stages (1 to 6), and in South Korea, it is 100 to 500(6 stages). LOD 1 or 2 stages are used in the basic planning stage, 3 or 4 stages are for the design, and 5 or 6 stages are the construction stage and maintenance level. Since cultural heritage is a building that has already been built, it is necessary to show details beyond the construction stage when expressed through modeling. The higher the level of expression of a building in the content, the more the end-user can feel that cultural heritage is interpreted in the virtual space. Therefore, XR content based on HBIM must be built with LOD 5 or higher. Second, LOI refers to the level of information \ufb01lled in BIM. In order to input infor- mation, it is a priority that metadata appropriate to the XR content be suggested. The metadata should be designed around the user who tours the virtual space, and after the integrated metadata is established, it is possible to provide information that re\ufb02ects the needs of users, such as visitors, heritage managers, and information managers. In the case of built heritage, the size, scale, material, substance, origin, wear, repair history, and technique of the member should be described as metadata of the LOI as a way to express the authenticity of the heritage. Such information might be provided differently","212 J. Lee and B. Kim depending on the pro\ufb01le (e.g. age, ethnicity, country, and occupation) of people who experience XR content. The LOD and LOI suggested in this paper should be developed in detail through continuous research and consensus. 6.2 User customized Content Service In order to provide user-customized content services for tourists, we propose user pro\ufb01les and built heritage metadata to describe built heritage information. Content recommen- dation systems based on the proposed metadata format can retrieve and provide built heritage information according to user pro\ufb01le. Existing customized services are limited to suggesting content already stored in the system based on personal information such as age, interest, and prior knowledge. The proposed system will be able to show relevant semantic information through the web according to personal information. For example, foreigners can connect and show\/receive their own heritage information related to their corresponding culture. In addition, the proposed HBIM-based system will show the details of cultural heritage information differently depending on the speci\ufb01c use case, such as tourism, education, and cultural property management. Ultimately, we will propose a system that considers not only personal information, but also time, climate, and density in the context of environmental content (Fig. 5). Fig. 5. Content service based on users\u2019 interest (Silva et al. 2018)","A Study on the Application of (HBIM) 213 6.3 Management of User-Generated Content Digital Twin is a virtual platform that models physical and digital systems in the same structure as the virtual world and enables monitoring, simulation, information visuali- sation, service veri\ufb01cation, and participation services. This is enabled through various sensing devices using IoT, 5G, Cloud, Edge, AI, AR\/VR, etc. Initially, it was developed by General Motors (GM) to speci\ufb01cally check products. Nowadays the Digital Twin technology is used for solving various industrial and social problems. Digital Twin can visualize cultural heritages in a virtual environment, check their status, perform analysis, simulate, and optimize them. The process of establishing a Digital Twin for HBIM-based XR tourism is as follows. First, spatial information to be re\ufb02ected as a Digital Twin is recorded using SLAM and 360-degree image technology and stored in an integrated database. Second, existing com- ponents and restoration parts of the building are stored in an integrated database using 3D scanning and modeling technologies. Contextual information related to architectural heritage is semantically connected through the cultural heritage database and web using architectural heritage ontology. Existing architectural heritage information and newly restored 3D models are re\ufb02ected as a Digital Twin. Digital Twin uses 3D modeling tech- nology to combine virtual architectural heritage with restored information and provide them as AR\/VR services. Fig. 6. Digital-twin based Tourism Platform and Business Model (Ssin et al. 2021) Digital Twin serves as a data sharing platform that can store and share newly gener- ated data in a cloud database. Digital Twin supports storing visitor\u2019s experiences, proof","214 J. Lee and B. Kim shots, and user-generated content regarding tourists\u2019 cultural heritage. Digital Twin rec- ommends content analysing previous users\u2019 data usage and \ufb01gures as well as provides the previous user\u2019s multimedia contents to new tourists through edge computing-based visualisations (Fig. 6). 6.4 Our Progress In order to pursue tourism of cultural heritage in virtual space and to plan for heritage management in units of member, the Cultural Heritage Administration in South Korea is implementing the \u2018A smart H-BIM modeling technology of wooden architecture for the conservation of Historical and Cultural Environment\u2019 project (HBIM project). This project aims at the ef\ufb01cient management of large-capacity modeling data by developing intelligent modeling technology of wooden architectural heritage. This three-year project intends to build 500 data sets consisting of members and manage 600,000 members based on LOD 4. R&D is largely divided into \ufb01ve \ufb01elds: to construct member archives according to traditional architectural style, procedural modeling technology of traditional buildings, semantic transform technology of HBIM buildings, and intelligent architectural model- ing of HBIM building groups and its placement techniques. The task progress consists of 1) designing members for HBIM and archiving traditional architecture, 2) developing procedural modeling tools, 3) developing group building modeling, 4) developing HBIM data management system, and 5) producing a prototype (Fig. 7). Fig. 7. The process of HBIM project The HBIM project can increase the time and \ufb01nancial ef\ufb01ciency in the production of built heritage content and help heritage managers to manage and monitor heritage data in units of members. Parametric modeling is expected to reduce the repetitive work of modeling and diminish the inef\ufb01ciency of redundant data management. In addition, it is possible to \ufb02exibly cope with the change and replacement of the member. Finally, it is expected that the XR content with HBIM technology helps end-users to feel the authenticity and identity of heritage when they tour historical sites on the virtual reality (Fig. 8).","A Study on the Application of (HBIM) 215 Fig. 8. The prototype of HBIM project 7 Conclusion This study has made the following contributions. The proposed technology can be expected to develop customized cultural heritage tourism services by developing a user model for information experience of architectural heritage. This study can contribute to the digital archive study of architectural heritage information by proposing metadata to include contextual information of the aforementioned history. This study will provide XR content to tourists by restoring the architectural rules and architectural styles of architectural heritage and reconstructing historical and cultural architectural complexes using HBIM-based digital twin software. The application of HBIM technology in the \ufb01eld of digital heritage can be activated by showing examples of the application of HBIM technology in the \ufb01eld of cultural heritage tourism. HBIM-based digital twin technology can support the creation of historical virtual backgrounds in historical video production and will reduce costs in game, museum, and educational content production. In the future, we will digitally restore the appearance of the local building groups using the HBIM-based authoring tool. In addition, we will design metadata that collects and integrates related historical information and conveys contextual information related to architectural heritage on the governmental archives and website to tourists. We would like to add a function to recommend appropriate architectural cultural heritage according to the purpose of the tourists. We intend to build a digital twin to integrate tourists\u2019 appreciation information and use it as architectural cultural heritage information. The proposed HBIM-based XR content will be produced to evaluate user satisfaction at the cultural heritage site. Acknowledgements. This study was conducted with support from the A smart H-BIM modeling technology of wooden architecture for the conservation of Historical and Cultural Environment project of the Cultural Heritage Administration (No. 2021A02P01-001).","216 J. Lee and B. Kim References AEC: AEC (UK) BIM Technology Protocol: Practical implementation of BIM for the UK Architectural, Engineering and Construction (AEC) industry (Version 2.1.1), AEC (2015) Argiolas, R., Bagnolo, V., Cera, S., Cuccu, S.: Virtual environments to communicate built cultural heritage: a HBIM based virtual tour. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLVI-5\/W1, 21\u201329 (2022) Bagnolo, V., Argiolas, R., Cuccu, S., Paba, N.: Beyond HBIM: serious games and procedural modelling for heritage dissemination. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLVI-4\/W4, 55\u201360 (2021) Gable, E., Handler, R.: After authenticity at an american heritage site. Am. Anthropol. 98(3), 568\u2013578 (1996) Heesom, D., Boden, P., Hat\ufb01eld, A., Melo, D.L.S., CzarskaChukwurah, F.: Implementing a HBIM approach to manage the translocation of heritage buildings. Eng. Constr. Architect. Manage. Eng. 28(10), 2948\u20132966 (2021) Historic England: BIM for Heritage: Developing a Historic Building Information Model. Historic England (2017) Jones, S., Yarrow, T.: Crafting authenticity: an ethnography of conservation practice. J. Mater. Cult. 18(1), 3\u201326 (2013) Jordan-Palomar, I., Tzortzopoulos, P., Garc\u00edaValldecabres, J., Pellicer, E.: Protocol to man- age heritage-building interventions using heritage building information modelling (HBIM). Sustainability 10(4), 908 (2018) Korea Institute of Civil Engineering and Building Technology. Guidelines for the BIM in Construction Industry, Ministry of Land, Infrastructure and Transport (2020) Larsen, K.E., Marstein, N.: Conservation of Historic Timber Structures: An Ecological Approach. Butterworth-Heinemann, Oxford (2000) Rodr\u00edguez, M.G.M.: Relationship between historic building information modeling and conven- tional valuation approaches for managing cultural heritage sites and its impact on tourism. J. Herit. Tour. 15(4), 381\u2013397 (2020) Silva, C.A., Toasa, R., Guevara, J., Martinez, H.D., Vargas, J.: Mobile application to encourage local tourism with context-aware computing. In: Rocha, \u00c1., Guarda, T. (eds.) Proceedings of the International Conference on Information Technology & Systems (ICITS 2018). AISC, vol. 721, pp. 796\u2013803. Springer, Cham (2018). https:\/\/doi.org\/10.1007\/978-3-319-73450-7_75 Smelik, R.M., Tutenel, T., Bidarra, R., Benes, B.: A Survey on procedural modelling for virtual worlds. Comput. Graph. Forum 33(6), 31\u201350 (2014) Ssin, S., Suh, M., Lee, J., Jung, T., Woo, W.: Science tour and business model using digital twin- based augmented reality. In: tomDieck, M.C., Jung, T.H., Loureiro, S.M.C. (eds.) Augmented Reality and Virtual Reality. PI, pp. 267\u2013276. Springer, Cham (2021). https:\/\/doi.org\/10.1007\/ 978-3-030-68086-2_20 UNESCO: Operational Guidelines for the Implementation of the World Heritage Convention. UNESCO World Heritage Centre (2021)","Development of the Capital City of Taebong Kingdom Metaverse and King Gungye Arti\ufb01cal Intelligence Digital Human Jin Ho Park1 and Hyoung Ki Ahn2(B) 1 Sejong Campus Virtual Smart City Visualization System Convergence Education and Research Cluster, Korea University, Seoul, South Korea [email protected] 2 Korea Institute for Archaeology and Environment, Seoul, South Korea [email protected] Abstract. The Capital City of Taebong Kingdom in Cheorwon was the beginning of the Korean name \u201cKOREA\u201d and the beginning of the opening of the Goryeo Dynasty. At a time when KOREA\u2019s status has recently attracted global attention due to the so-called \u201cKorean Wave,\u201d research on \u201cTaebong kingdom,\u201d which can be called KOREA\u2019s Original, is being re-examined. However, at a time when the long-standing division of the two Koreas makes it dif\ufb01cult to directly investigate the Taebong Kingdom on the DMZ, reproducing the Taebong Kingdom through the Metaverse is expected to be the only alternative to the site approach caused by the division. Although the arti\ufb01cial intelligence digital human being, who is both the advocate and the \ufb01nisher of the Taebong Kingdom, has already died biologically under the environment of the metaverse space, the so-called \u201cKing Gungye\u201d will be revived in digital form in the metaverse space. As such, it is expected that the history of the 1,000th millennium Taebong Kingdom can be digitally re-examined through the metaverse environment of Taebong Kingdom and King Gungye Arti\ufb01cial Intelligence Digital Human. Keywords: Taebong Kingdom \u00b7 King Gungye \u00b7 Metaverse \u00b7 Digital human \u00b7 AI 1 Introduction Today, the Taebong Kingdom in Cheorwon is a capital located in Hongwon-ri, Cheorwon-gun, Gangwon-do, measuring 7.7 km in Naeseong (inner Wall), 12.5 km in Oeseong (out ter Wall), and 1.8 km in Gungseong (Main Palace). It is now called the Capital of Taebong Kingdom. In the past, during the Later Three Kingdoms period, when King Gungye moved the capital from Gaesong to Cheorwon in 905 AD, he served as the capital of later Goguryeo for a short period of 13 years, from King Taejo Wang Geon ousted King Gungye and established Goryeo Kingdom. In an excavation survey conducted from 1940 during the Japanese colonial period, the stone lantern in front of the palace\u2019s Pojeongjeon Hall (Near DMZ) of the capital of Taebong Kingdom was once National Treasure. \u00a9 The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 T. Jung et al. (Eds.): XR 2022, SPBE, pp. 217\u2013227, 2023. https:\/\/doi.org\/10.1007\/978-3-031-25390-4_19","218 J. H. Park and H. K. Ahn Meanwhile, the Taebong Capital Wall is located within the Demilitarized Zone and the inter-Korean military demarcation line crosses the middle of the city, so no proper excavation investigation has been conducted. Due to the confrontation between the two Koreas, the only The Capital City of Taebong in Korean Peninsula, which is dif\ufb01cult to access or excavate, has been partially investigated from the Japanese occupation to the latest, but has not been able to excavate the entire company. It is impossible to grasp the pattern of the ruins unless a full-scale excavation of City Wall occurs in the future. As an alternative to this, We would like to present a solution to this through the \u2018Taebong Metaverse\u2019. In particular, it is not limited to a particular period of time, and it is divided into time and space from the time of The Capital City of Taebong, the time of King Gungye\u2019s activities, the time of The Capital City of Taebong\u2019s abandonment due to the founding of Goryeo Kingdom, the early 20th century, and the 21st century. Through this, we would like to present an experience of Metaverse beyond time and space that can overcome the dif\ufb01culty of accessing the Taebong Kingdom and overcome the temporal limitation of \ufb01nding the appearance of the Taebong Kingdom even if you go to the current historical site. 2 Digital Restoration of Capital City in Taebong Kingdom In order to achieve the Metaverse of the Taebong Kingdom, the related prior research was analyzed. The most basic foundation for forming the Taebong Metaverse is the digital restoration of the City Wall. This is because the digital restoration of the Taebong City Wall can be expanded to the Taebong Metaverse only when it becomes the core. Therefore, we will consider the digital restoration process of Taebong Kingdom, which is the basis of the metaverse. 2.1 Organizing the Primary Data for Digital Restoration After the division of North and South Korea, a study was conducted on the capital of Taebong Kingdom located in the Demilitarized Zone (DMZ). However, in the past, it was practically dif\ufb01cult to grasp speci\ufb01c aspects in the situation of division between the two Koreas. Since then, through a joint survey conducted by Director Lee Jae-in and the Chuncheon National Museum, it has been possible to speci\ufb01cally con\ufb01rm the existence of the inner and outer fortresses of Cheorwon Fortress in the Demilitarized Zone. Mean- while, the National Museum of Korea was published in 2009 by the National Museum of Korea in 2009 by combining Japanese colonial era\u2019s data and the museum\u2019s on-site survey of Taebong Kingdom, which was conducted in 2008 with related organizations. In the meantime, Korean historians and archaeologists have been enthusiastic about the cultural relics of the Taebong National City Wall, but due to the special nature of the division, little research and excavation of the Taebong National City Wall in the Demilitarized Zone (DMZ) has been made. Currently, there is a limit to the progress of archaeological investigations in the situation of division between the two Koreas, so we have no choice but to focus on the Japanese colonial era survey data.","Development of the Capital City of Taebong Kingdom Metaverse 219 Based on ancient data published by representative scholars such as Keikichi Ogawa, Tadashi Sekino, Ryujo Torii, etc. during the Japanese colonial period, it was estimated that there was one stone lantern in the southern part of the fortress and one stone lantern related to Bongseonsa Buddist Temple founded by King Taejo. Although the location estimation of these Taebong City Rock Lanterns can be traced to some extent, it is dif\ufb01cult to grasp the reality of the \ufb01nal Cheorwon City Wall, such as Taebong Kingdom\u2019s Palace, outer wall, and inner wall. 2.2 Aerial Archeology Study of the Capital City of Taebong Kingdom \u2780 Reconstruction of old terrain using aerial archeology. Literature historical materials and old photographic studies have limitations in uncov- ering the reality of The Capital City of Taebong Kingdom. To overcome this, the archaeological community analyzes the shape of old ruins using aerial photos. In the 1920s, aviation archaeology was newly introduced by O. G. S. Crawford (1886\u20131957). After that, a six-stage topographic environment analysis method was devised by Manabu Takahashi of Japan. Currently, in order to overcome the limi- tations of aerial photograph analysis using a stereoscope, a 3D analysis program of aerial photographs has been developed and widely used for historical investigation through aerial archaeology analysis. The \ufb01rst case of application in Korea is an aerial archaeology analysis of the plain area along with the development of Sejong New City Chungcheongnam-do (developed as the center of the Korean government of\ufb01ce) in 2005. In the same way, the reality of the capital city of Taebong Kingdom was con\ufb01rmed through aerial archaeological analysis. Based on this, it can be applied in various ways to examine the overall location and landscape of the Capital city of Taebong Kingdom by analyzing the topography around the Cherwon. \u2781 Analysis of Capital of Taebong Kingdom Using Aerial Archeology. After the Korean War, the Taebong Kingdom was inaccessible for the past 70 years due to the Demilitarized Zone (DMZ). In order to overcome the limitations, an aerial archaeological analysis of Cheorwon, the capital city of the Taebong Kingdom, was attempted. According to recent research results, Cheorwon, the capital city of the Taebong Kingdom, was built on the basis of a \ufb02at hilly area (HEO Ui haeng and YANG Jeong seok 2021). The outer and inner walls seem to have been built around the ridgeline of each hill. Traces of the palace were also observed. Several gates and gates are identi\ufb01ed on this wall that lead to Cheorwon, the capital city of the Taebong Kingdom. The entire capital\u2019s palaces are assumed to be trapezoidal.","220 J. H. Park and H. K. Ahn In addition, traces of drainage channels and annexes were con\ufb01rmed inside the palace. An east-west road was observed. In Cheorwon, the capital city of the Taebong Kingdom, palaces and buildings are arranged according to the topography. However, due to military tensions, it is impossible to directly visit Cheorwon, the capital city of the Taebong Kingdom. However, research on space utilization has been progressed to some extent through aerial archeology analysis. The timing and method of the construction of each wall is not yet certain. However, I was able to understand the city construction using the topography and the use of the internal spatial structure. Based on the results of the most recent research in 2021, basic data for the digital restoration of Cheorwon, the capital city of the Taebong Kingdom, were prepared. 2.3 3D Modeling the Capital City of the Taebong Kingdom For virtual 3D modeling of the capital city of the Taebong Kingdom, the entire modeling section of the city wall is divided based on the high-altitude analysis data. The capital city of the Taebong Kingdom site and gate site revealed through high-altitude analysis are determined, and the arrangement and form are derived based on the case analysis of temples that can be referred to the capital city of the Taebong Kingdom, such as relics from the Joseon Historical Archives such as stone lanterns. In addition, the status and usage of Jeongnimsa Buddist Temple Site are identi\ufb01ed through literature analysis such as the Three Kingdoms History and the preceding studies in each \ufb01eld, and the Buddha statue and majesty \ufb01eld are estimated through comparative analysis of cities in Korea, Japan. The contents of the historical research for the restoration of the original the capital city of the Taebong Kingdom are as follows. Based on the previous study, the capital city of the Taebong Kingdom Highland Analysis \u2192 Site Size \u2192 Estimation of the layout of private houses including the palace in the capital city of the Taebong Kingdom \u2192 Structure method (including the palace) \u2192 Architectural structure, scale, and structure of the capital city of the Taebong Kingdom, etc. Related artifacts should be used as evidence. In addition, it is necessary to compare and analyze the types of temples that can be referred to in Naewon (inner Palace), where the stone lantern was located, the temple located inside and outside the City Wall of Taebong Kingdom, and Hwangnyongsa Temple in the Uni\ufb01ed Silla Period and Goryeo Dynasty. The most important thing in 3D modeling of the capital city of the Taebong Kingdom is Currently, Cheorwon-gun is building a so-called the capital city of the Taebong Kingdom Theme Park. It aims to be completed in June 2023, and a theme park is being built in the theme park with the","Development of the Capital City of Taebong Kingdom Metaverse 221 character of a comprehensive museum that can cultivate the historic the capital city of the Taebong Kingdom on Cheorwon land 1,000 years ago, including a building where the portrait of King Gungye is enshrined and a space to display exhibits related to the capital city of the Taebong Kingdom (Fig. 1). Fig. 1. Results of aerial archaeological analysis of Cheorwon, the capital city of Taebong Kingdom (HEO Ui haeng\u00b7YANG Jeong seok 2021: 43p, 51p, Posted after editing). 3 The Capital City of the Taebong Kingdom AR and MR After 3D Modeling the capital city of the Taebong Kingdom based on topography, AR and MR contents are created based on this data. First, make a miniature of the capital city of the Taebong Kingdom and put a PC-\ufb01xed AR Device here. Augment the model through the AR window (Fig. 2). Mixed reality content using Microsoft\u2019s HoloLens 2 is also possible. Through this, MR contents are produced for stone lanterns in King Gungye the capital city of the Taebong Kingdom. The audience can also experience King Gungye the capital city of the Taebong Kingdom, which has been digitally restored through 3D modeling. Furthermore, it is possible to experience mixed reality contents targeting stone lanterns located in King Gungye the capital city of the Taebong Kingdom (Fig. 3).","222 J. H. Park and H. K. Ahn Fig. 2. Based on the Miniature of the capital city of the Taebong Kingdom AR","Development of the Capital City of Taebong Kingdom Metaverse 223 Fig. 3. Mixed reality content implementation scene based on Microsoft\u2019s HoloLens 2 4 The Capital City of the Taebong Kingdom Metaverse By applying the Metaverse, it can be attempted on the metaverse of the capital city of the Taebong Kingdom Province in Cheorwon, where it exists in a virtual space as it is in reality, and the information and story of the capital city of the Taebong Kingdom Province, and even an Immersive experience. Existing realistic cultural heritage contents have not reached a total experience by focusing on providing information. Beyond simple restoration of cultural heritage, the value of the capital city of the Taebong Kingdom, a capital city and the religious experience that the people of the capital city of the Taebong Kingdom, referred to as the after Gogurye period, learned in that era should be provided. As an appropriate countermeasure to this, \u2018Metaverse\u2019 is suggested. It aims to overcome the limitations of of\ufb02ine exhibitions such as VR, AR, MR, and XR with metaverse contents. Metaverse is not a Virtual Space that exists alone, but a place-type content that moves with each other while interacting with the virtual space and the real world. If previous digital Heritage technologies focused on information provision and digital restoration, the metaverse space can present a total experience space that transcends reality and virtualization. Metaverse, which can be discussed as an experiential 3D virtual world, can be de\ufb01ned as a kind of virtual world that can combine both reality and virtual space due to rapidly developed technology in recent years. As a recent study related to digital heritage and metaverse, \u201cArti\ufb01cial Intelligence XR Metaverse Content Proposal (Park Jin-ho et al. 2021)\u201d is a paper related to metaverse and digital heritage in earnest (Fig. 4).","224 J. H. Park and H. K. Ahn Fig. 4. The capital city of the Taebong Kingdom Virtual Tourism Metaverse Becomes AI-Avatar for Visitors and Becomes Cheorwon City Wall. 5 Development of King Gungye Digital Human 5.1 Arti\ufb01cial Intelligence (AI) Digital Human Overview Digital Human refers to a virtual character created with computer graphics (CG) that can express appearances and actions that are almost similar to real living humans. This digital human technology is a game. It is used to produce CG contents such as movie special effects, 3D animation, and VFX. Recently, digital human technology has been used to produce video content that restores late artists, famous actors, and historical \ufb01gures (Fig. 5). Fig. 5. King Gungye Arti\ufb01cial intelligence Digital Humans AI Technology is also applied by reproducing unfamiliar historical \ufb01gures with 3D digital humans. On the left is an arti\ufb01cial intelligence video recording of survivors of World War II\u2019s Nazi genocide, and on the right is Baekbeom Kim Gu, a digital human AI was applied by reproducing it with. Produced by Bibimble Co., Ltd.","Development of the Capital City of Taebong Kingdom Metaverse 225 By creating characters with digital human technology, it is possible to create facial expressions of objects to be made into virtual humans, realistic expressions of the entire body, and virtual digital human body movements through rendering and reeling of exter- nal skin of virtual humans. In particular, if combined with arti\ufb01cial intelligence (Arti\ufb01cial Intelligence), it is expected to build the ultimate convergence content in the form of \u2018XR (Extended Reality)\u2019 in the future. Digital human technology can reproduce not only the appearance of a person but also the delicate facial expressions and muscle expressions of body movements when producing a dead or alive person. Digital human technology, which has reached the peak of CG technology, will have a very huge impact on the entire cultural industry in the future. Recently, the latest technologies such as digital human and arti\ufb01cial intelligence have been developed, but their use in digital heritage and museum digital exhibitions is quite insigni\ufb01cant. 5.2 Arti\ufb01cial Intelligence Digital Human Production Process 3D Digital Human technology is used to reproduce historical \ufb01gures from the past and AI is installed here. We make human. This is a digital human restoration project that brings back past \ufb01gures who lived in the past but do not exist now. Arti\ufb01cial technology is given to digital human restoration to produce interactive contents of characters that existed only as portraits in the past. As a domestic case, when Digital Heritage and Arti\ufb01cial Intelligence (AI) are com- bined, some form of arti\ufb01cial intelligence content will explain the case. Until now, arti\ufb01cial intelligence technology can be used in a wide variety of ways, but at this point, a practical solution is interactive arti\ufb01cial intelligence content for cultural heritage guid- ance. Latelier, an urban theme park that opened in Dongdaemun in November 2017, for the \ufb01rst time in Korea, is in a pioneering position where it \ufb01rst started \u201cInteractive Arti\ufb01cial Intelligence Content.\u201d 6 King Gungye Arti\ufb01cial Intelligence Digital Human Although arti\ufb01cial intelligence technology has developed rapidly, it is not easy to apply technology related to the \ufb01eld of \u201ccultural heritage\u201d compared to other industries, and the speed of application is slow. Examples of successful applications in the museum \ufb01eld of arti\ufb01cial intelligence technology so far include statistical analysis such as museum content satisfaction survey using arti\ufb01cial intelligence deep learning technology or big data analysis through data mining. Examples of digital human-based arti\ufb01cial intelli- gence contents include the case of arti\ufb01cial intelligence of Seosan Monk produced by the National Museum of Korea. After reproducing it in the form of a 3D object in the form of \u201cDigital Human,\u201d it is produced so that many visitors can experience the content at the same time through a multi-area display system that supports high-de\ufb01nition (4K UHD). Now arti\ufb01cial intelligence technology is \u2018interactive arti\ufb01cial intelligence technology\u2019. Projecting a high-resolution multi-projection of more than 20,000 eyeballs on this type of structure","226 J. H. Park and H. K. Ahn gives visitors a similar experience as if they were the king of Gungye in the space of the capital city of the Taebong Kingdom in Cheorwon. Experience contents produced based on King Gungye digital human characters and projection mapping are virtual fusion experience contents that travel between the past and the present in the space between king Gungye 3D objects and viewers that appear virtually on the screen (Fig. 6). Fig. 6. A scene from King Gungye\u2019s XR-Based Arti\ufb01cial Intelligence digital human production appears and tells the audience the story of the capital city of Taebong, Cheorwon (DMZ) The plan is to create an environment where visitors can directly talk to King Gungye by creating an arti\ufb01cial intelligence content \u201cexperience zone.\u201d This circular projection mapping and King Gungye Digital Human Arti\ufb01cial Intelligence content can further double the immersion of the \u201cCheolwon the capital city of the Taebong Kingdom.\u201d First, visitors can freely experience it in the form of content without wearing 3D glasses or HMD equipment. Content visitors can freely experience realistic content in a virtual environment without worrying about VR motion sickness. There is no hassle or inconvenience of wearing HMD. In addition, from the perspective of the exhibition subject, the dif\ufb01culty of managing the equipment that visitors have to wear individually is reduced, making it easier to maintain the installation content. To complete the arti\ufb01cial intelligence contents of the capital city of the Taebong Kingdom can be built based on thousands of research materials related to the capital city of the Taebong Kingdom in Cheorwon, books, old records, old photographs of Cheorwon-gun old City in the Joseon Dynasty. 7 Conclusion The proposed \u201cThe capital city of the Taebong Kingdom Metaverse\u201d is provided in the form of \u201cXR content\u201d in of\ufb02ine spaces, and services can be provided in the form of \u201cMetaverse Content\u201d in online spaces. The two are expected to replace the current \u201cthe capital city of the Taebong Kingdom,\u201d where on-site viewing is impossible.","Development of the Capital City of Taebong Kingdom Metaverse 227 In addition, at a time when it is not easy to freely and of\ufb02ine tourism due to the coronavirus, cyber virtual tourism is possible through the \u201cThe capital city of the Taebong Kingdom Metaverse\u201d even if you do not visit the capital city of the Taebong Kingdom site in person. This is a type of Metaverse Museum that transcends a kind of time and space that can be experienced even in the past, present, and future of the capital city of the Taebong Kingdom in Cheorwon, which will take place after inter-Korean Uni\ufb01cation in the future. As an alternative to overcoming dif\ufb01culties in access due to this realistic division and limitations in \ufb01eld visits due to the global pandemic situation, the Metaverse of the capital city of the Taebong Kingdom, Cheorwon was proposed. Acknowledgements. This paper was prepared with the support of the 4th Stage BK21 ARMR System and Content Convergence Research Group of Korea University. References Ahn, H.K., Park, J.H.: A study on the contents of the UNESCO World Heritage Troy Ruins layer- based Arti\ufb01cial Intelligence Metaverse. In: Proceedings of the Korean Contents Association Comprehensive Conference, pp. 55\u201356 (2021) Heo, U.H., Yang, J.S.: A study on the paleotopographic and structural analyses of Cherwon castle in Taebong. MUNHWAJAE Korean J. Cult. Herit. Stud. 54(2), 38\u201355 (2021) Park, J.H.,\u00b7Lee, Y.G.,\u00b7Lee, J.S., Ahn, H.K.: Seokguram Digital Contents Prospects for the Proposal of Arti\ufb01cial Intelligence Extended Reality. Korea Society of Image Arts and Media, Serial 19 No. 4 (2021) Park, J.H., Kim, S.: Arti\ufb01cial Intelligence Contents in the Age of Digital Heritage, Developing Interactive Arti\ufb01cial Intelligence Contents for Historical People. In: International Conference on Convergence Content 2020. The Westin Chosun Busan, Korea, November 12 to 14 (2020) Park, J.H., Kim, S.: Augmented Reality and Virtual Reality: Empowering Human, Place and Busi- ness: Digital Representation of Seokguram Temple UNESCO World Heritage Site. Springer (2017) Park, J.H., Muhammad, T., Ahn, J.: The 3D reconstruction and visualization of seokguram temple world heritage site. In: Proceedings 20th International Conference on Virtual Systems and Multimedia. VSMM (2014) Park, J.H.: Digital restoration of Seokguram Grotto: the digital archiving and the exhibition of South Korea\u2019s representative UNESCO world heritage. In: Ubiquitous Virtual Reality (ISUVR), International Symposium on pp. 26\u201329. IEEE (2012)","Health and Wellbeing","Possibilities of Rehabilitation and Telerehabilitation of Patients with Moderate and Severe Course of COVID-19 Disease Using Virtual Reality J\u00e1n Lacko(B) and Eugen Ru\u017eick\u00fd Faculty of Informatics, Pan-European University, Bratislava, Slovakia {jan.lacko,eugen.ruzicky}@paneurouni.com Abstract. In this paper, we present the possibilities of using virtual reality in the rehabilitation of patients after overcoming COVID-19 disease with a moderate or severe course. We have created a virtual environment with various interactive tasks that help patients in the rehabilitation of especially the musculoskeletal system and cognitive abilities. Patients were divided into groups according to the severity of the course of the disease and the possibility of rehabilitation with a therapist in an outpatient clinic or the possibility of tele-rehabilitation in the home environment was used. Based on the results, we showed that virtual reality was a suitable tool for these types of rehabilitation and provided opportunities for comparing the classical approach and modern methods. Virtual reality had a greater impact on patients\u2019 motivation to rehabilitate, especially in cases where patients were using virtual reality for the \ufb01rst time. Keywords: COVID-19 \u00b7 Virtual reality \u00b7 Rehabilitation \u00b7 Tele-rehabilitation 1 Introduction The course of COVID-19 is very different from patient to patient. In some patients, it may lead to hospital treatment through arti\ufb01cial lung ventilation, while in others it may manifest itself in a feeling of fatigue, or it may cause various symptoms which, in the longer term, may lead to partial muscle atrophy. Rehabilitation is an important part of the treatment of patients after overcoming the acute phase of the disease. As with other diseases, the goal of rehabilitation is to restore patients to all the functions of their body, respectively mitigate the consequences of illness or injury. In this research, we focused on the use of virtual reality devices to rehabilitate patients in a controlled outpatient environment and to rehabilitate patients at home. Through virtual reality methods, we tried to activate patients and motivate them to regular rehabilitation. In creating an environment for rehabilitation and in individual tasks for patients in virtual reality, we used the previous experience gained by the team in designing methods for rehabilitation of patients after a stroke (\u0160ramka et al. 2020). The study evaluated people\u2019s movements using virtual reality and arti\ufb01cial intelligence methods to prevent falls on stairs by using \u00a9 The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 T. Jung et al. (Eds.): XR 2022, SPBE, pp. 231\u2013242, 2023. https:\/\/doi.org\/10.1007\/978-3-031-25390-4_20","232 J. Lacko and E. Ru\u017eick\u00fd emotions (Lacko 2020). Rehabilitation through virtual reality may contribute to a more effective rehabilitation process for patients with mild forms of Parkinson\u2019s disease. In a selected group of 8 patients, an improvement in \ufb01ne hand motor skills was found after 12 weeks (Ru\u017eick\u00fd et al. 2022). 2 Previous works The COVID-19 pandemic has not only economic impact but also health consequences, known as Post-Covid Syndrome (\u0160ramka et al. 2021). Long-term neurological and psy- chiatric adverse effects of COVID-19 may include depression, insomnia, decreased cog- nitive ability, and accelerated aging (Tsutsui et al. 2021). For this reason, it is necessary to monitor the state of the population after the end of a pandemic situation. Clinical signs of neurological disease associated with COVID-19 and the effects of coronavirus on the nervous system led to neurological and psychiatric diseases (Varatharaj et al. 2020). Post COVID-19 syndrome requires rehabilitation care for patients due to physical, res- piratory, and neurological symptoms (Mao et al. 2020). Virtual reality in the context of treatment of patients with COVID-19, or after overcoming them, can be realized in the acute phase of the disease, for example through guided meditation, exploration of natural environments, cognitive stimulation games. Participants commented that the use of VR was useful in coping with isolation and could be implemented within the context of clinical care for COVID-19 patients (Kolbe et al. 2021). Part of the use of virtual reality is the possibility of its use in the home conditions of patients. At present, this model is not widely used in the world. In the case of tele-rehabilitation of patients, computer programs are rather used, in some cases based on arti\ufb01cial intelligence. In the United Kingdom, arti\ufb01cial intelligence research was conducted for individual home rehabilitation in patients suffering from neurodegenerative diseases during the COVID- 19 pandemic (Vourganas et al. 2021). Similar recommendations have been made with physiotherapists, rehabilitation physicians, neurologists, psychologists, and psychiatrists (\u0160ramka et al. 2020). An integral part of the tele-rehabilitation process is the possibil- ity of distance communication with the therapist. In Canada, Alberta Health Services has launched a TeleHealth service focused on the rehabilitation needs of people with acute and chronic musculoskeletal, neurological, and other conditions affected by the COVID-19 pandemic. This service allows patients and caregivers to speak directly to rehabilitation physicians and specialists, physiotherapists (Carriere et al. 2021). A range of digital health technologies and a new environment for digitizing clinical trials to cope with a pandemic are in place (Whitelaw et al. 2020). Arti\ufb01cial intelligence techniques, virtual reality and other modern digital technologies have impact of rehabilitation, tele- rehabilitation, and virtual reality on reducing fatigue, stress and improving the patient\u2019s cognitive functions through an individual optimized procedure (Ru\u017eick\u00fd et al. 2020). During the pandemic, post-stroke patients aged 29 to 89 years were randomly selected to receive combined rehabilitation with VR rehabilitation, with 30 patients receiving com- bined full VR rehabilitation and 10 patients receiving only sham VR rehabilitation. The results showed that patients receiving full VR rehabilitation had shorter hospitalisation time compared to conventional treatment and sham VR treatment only (Chatterjee et al. 2022). Designing VR rehabilitation directly in rehabilitation facilities is the basis for","Possibilities of Rehabilitation and Telerehabilitation of Patients 233 motivational exercises and attractive games for patients during the COVID pandemic. Research \ufb01ndings and experience were used to develop exercises including techniques adapted to tele-rehabilitation (Munoz et al. 2022). 3 Methodology After experience with the use of virtual reality methods in the rehabilitation of patients after stroke and patients with neurological diseases, we also focused on the rehabilitation phase of patients after overcoming the COVID-19 disease. In the Slovak Republic, four waves of this disease have been recorded so far, and during the last three waves, there has been a relatively large number of patients in hospitals. Of these patients, we focused on the subsequent rehabilitation of those who had a moderate or severe course of the disease. According to statistics from the National Health Information Centre in the Slovak Republic, the hospital had a total of 98,109 patients with moderate course, who required medical supervision, and 8,643 patients with a severe course, who required intensive care. Histogram of patients admitted to hospital, ICU and pulmonary ventilation in the Slovak Republic is shown in Fig. 1. Fig. 1. Histogram of patients admitted to hospital (light green), ICU admissions (dark green) and pulmonary ventilation (brown) during the different waves of COVID-19 in the Slovak Republic. (Data source: Data: National Health Information Centre) (Color \ufb01gure online) In the group of patients with moderate course, we included patients who were hospi- talized and subsequently, after overcoming the disease, had a problem with the so-called long-term COVID or post-COVID syndrome, or have had neurological, muscular, or breathing and lung problems. In the group of patients with a serious problem, we included patients who were in the hospital and were put to arti\ufb01cial sleep using pulmonary venti- lation. In addition to the previous problems, such patients also developed problems with rapid muscle weakness, which enabled them to undergo subsequent rehabilitation, \ufb01rst in a supine position, later in a sitting position. 3.1 Patient Selection As part of the research, we created a virtual testing environment with interactive tasks that focused on the movements of the upper limbs and the cognitive abilities of patients.","234 J. Lacko and E. Ru\u017eick\u00fd When selecting patients, we focused on two groups. The \ufb01rst group were patients with a moderate course, without the need to put them to arti\ufb01cial sleep and connect to pulmonary ventilation, but who had problems with movement due to increased fatigue. The second group were patients who had to be connected to pulmonary ventilation and spent more than 2 weeks in bed, resulting in muscle atrophy. As part of the selection of patients, it was necessary to exclude from the rehabilitation process patients who have problems with orientation in space, suffer from problems with the vestibular system, or have problems with epilepsy and vision. When selecting patients who will undergo planned rehabilitation using virtual reality, it was necessary to explicitly know their current health status and exclude parameters that could lead to other health problems caused by the use of VR facilities. For this reason, we have prepared a form that potential candidates for rehabilitation completed before completing it. Thanks to pre-form, we were able to exclude high-risk patients who had problems with epilepsy, severe muscle problems, vision problems that prevent stereoscopic vision or blurred vision, problems with the vestibular system, problems with motion sickness. At the same time, we also found out in the questionnaire whether the patients had previous experience with VR, so as to exclude the possible in\ufb02uence of the so-called \u201cWOW\u201d effect on the motivation of patients for regular rehabilitation. 3.2 Hypotheses Based on our previous experience with rehabilitation of patients after stroke with neu- rodegenerative diseases, we formulated two hypotheses, which were veri\ufb01ed on two groups of patients, divided according to the severity of the course of COVID-19. This was an initialization study that will need to be validated with a larger number of patients and a control group. The hypotheses were as follows: H1: Virtual reality accelerates the rehabilitation process based on the fact that the patient is motivated to exercise thanks to the experience of immersion in a virtual environment. H2: It is more advantageous for patients to interact in the scene without the need for controllers, just by scanning the hands with external sensors using avatars of the upper limbs. To verify the hypotheses, we designed testing in the home environment using tele- rehabilitation methods and testing in a controlled environment in a rehabilitation clinic. For testing purposes, we created virtual environments that were adapted for the treatment of upper limbs and respiratory problems. Recommended devices for individual types of patient rehabilitation were designed for the created environment.","Possibilities of Rehabilitation and Telerehabilitation of Patients 235 3.3 Rehabilitation vs. Tele-rehabilitation As part of the research, we implemented two approaches to patient rehabilitation. Patients who used equipment in the home environment and did not need to be under the direct supervision of a therapist used tele-rehabilitation methods. For the needs of tele-rehabilitation, the exercise was designed so that we could use the possibility of per- forming tasks from everyday life in a virtual environment. In this way, we eliminated the need for patients to focus on exercises, but the movements they performed were aimed at rehabilitating the upper limbs in cooperation with the development of cognitive abilities (e.g., moving objects in a given order that the patient had to remember). Patients from the second group used rehabilitation with a therapist in a controlled environment of an outpatient clinic or rehabilitation clinic. Emphasis was placed on the precise execution of exercises in the order speci\ufb01ed by the therapist. The therapist has the opportunity to control the course of the exercise and its dif\ufb01culty. At the same time, it was possible to implement more complex setup of individual devices and sensors for data acquisition. 3.4 Rehabilitation Design A key element in the rehabilitation of patients was the creation of scenarios for the rehabilitation of the upper limbs in connection with cognitive abilities. When designing the individual scenarios, we based on the premise that the patient in virtual reality does not want to feel during the exercise that he is forced to repeatedly perform the same tasks that are aimed at exercising muscles but wants to perform his exercise through natural movement and manipulation of objects in the scene. We have prepared a series of 8 scenarios for the exercise. The scenarios were consulted with rehabilitation experts so that patients gradually rehearsed all muscle groups of the upper limbs. For cognitive abilities, the key was to repeat the sequence of manipulation of objects based on a predetermined order that the patient had to remember. To support the patient\u2019s breathing training, 6 scenarios were created in which the patient had to mirror the inhales and exhalations of the avatar he was watching in the virtual scene. Patients had to breathe at the prescribed rate (number of breaths and exhalations during the prescribed time interval) and hold their breath according to the prescribed schedules. For the manipulation of virtual objects in virtual reality, we designed the use of controllers and the use of natural gestures of the user based on sensing the position and orientation of the hands using the leap motion sensor. We decided to use a leap motion sensor after the \ufb01rst tests, which showed that patients were not able to hold the controls in their hands and manipulate them for a long time due to weak muscles in their hands. It was also easier for patients to use natural gestures to select objects. For training purposes, we used the avatars of the user\u2019s upper limbs, as shown in Fig. 2.","236 J. Lacko and E. Ru\u017eick\u00fd Fig. 2. Avatar of the user\u2019s hands when tracking using the leap motion sensor (left) and to display the HMD controllers HTC VIVE Pro eye (right). When using controllers, it was possible to manipulate objects within the entire space without losing focus on the controller, because it was continuously tracked using outside in systems using HTC VIVE Pro EYE, or inside out system using Oculus quest 2. If using a leap motion sensor, the patient\u2019s hands could only be detected in the border area, and if the patient\u2019s hands were outside the FOV sensor, the system lost focus information and if the user manipulated the object, the manipulation information was interrupted, and the object was classi\ufb01ed as free. Figure 3 shows the FOV for tracking the user\u2019s hands via the leap motion sensor and via the VR headset controllers, showing the loss of focus. Fig. 3. Loss of focus when tracking hands with Leap motion when placing the hand outside the FOV of the sensor cameras (left), maintaining focus for the controllers (right). The results of patient rehabilitation were measured on the basis of information from the system when recording individual actions of the user and on the basis of subjective evaluation of patients using a questionnaire after the rehabilitation process.","Possibilities of Rehabilitation and Telerehabilitation of Patients 237 3.5 Virtual Environment As part of creating virtual environments for virtual reality, we designed 4 environ- ments using a photorealistic environment (corridor, nature, living room and shop) and two environments as non-photorealistic (forest, lake). The created non-photorealistic environment did not require as much rendering performance as the photorealistic envi- ronment, even though the lighting in the scenes was baked into textures and there were no dynamic light sources in the scene. Fig. 4. Photorealistic scene for breathing training with an avatar for mirroring breaths. As part of tele-rehabilitation, since we used the Oculus quest 2 headset, in which, for simplicity, we relied only on the computing power of the device itself, we were forced to look for the optimal ratio of photorealism in creating the environment. From the experience of previous projects, we mostly focused on the use of photore- alistic environments, which were more natural for users and more pleasant in terms of usability. Figure 4 shows a photorealistic scene for a breathing training scenario using an avatar created in the Unreal engine using the Metahuman system. 3.6 Measurements We divided the patients into two groups for measurement purposes. The \ufb01rst group consisted of patients with moderate disease, who used tele-rehabilitation at home, and the second group of patients (severe course), who participated in rehabilitation in a controlled environment at the rehabilitation clinic. There were 10 patients in the \ufb01rst group (28\u201364 years old, average [A]: 54.1 standard deviation [SD]: 10.7). In the second group 6 patients (49\u201369 years old, A: 58.7, SD: 7.6). For patients from the \ufb01rst group, we proposed testing by the method of tele- rehabilitation in the home environment (initially controlled with remote control of the therapist, then performed only by the patient himself). For this scenario, we used devices for VR, which do not require a connection to a computer and are so-called all-in-one solution (Oculus quest 2). To move in space, it was necessary for the devices to have","238 J. Lacko and E. Ru\u017eick\u00fd 6DoF. For this reason, we used devices with inside-out tracking as test devices. The position and orientation of the upper limbs were tracked by controllers. According to the suggestions of neurologists and rehabilitation specialists, we pre- pared a schedule of exercises and games that were individually adapted according to age and personality type. In the initial phase of exercise, shorter 5-min VR exercises were performed more frequently, which were gradually extended in time up to 20 min according to the parameters detected. Patients from the second test group used the possibility of rehabilitation in a con- trolled environment with a therapist directly in the outpatient clinic. Assisted rehabilita- tion was performed using devices (HTC VIVE Pro eye) with wireless transmission and a more powerful computer, while for tracking the upper limbs of patients we did not use controllers directly due to muscle atrophy, but the position and orientation of the upper limbs was tracked using a leap motion tracker. Rehabilitation of both groups of patients was realized only after the patient was cured of COVID-19 and followed by a medical certi\ufb01cate. The duration of rehabilitation was set at 6\u201310 weeks. The moderate course of the disease, the shorter the rehabilitation time required. Each week, the patient had to rehabilitate four in VR and it was supplemented by a classic rehabilitation of 2 \u00d7 40 min a week. The length of rehabilitation was determined based on recommendations for the length of use of virtual reality devices and doctors\u2019 recommendations. 4 Findings In the questionnaire, we monitored ten critical indicators, through which we evaluated the impact of the rehabilitation system in virtual reality on the progress or regression of the disease and also focused on indicators that can monitor the impact of scenar- ios, photorealistic environment or the use of controllers on the rehabilitation process. The individual answers in the questionnaire were evaluated using a Likert scheme of 5 answers. We monitored the following indicators: \u2022 (Q1) Motivation of the patient \u2022 (Q2) \u201eWOW\u201c effect during the rehabilitation process \u2022 (Q3) Subjective comparison of the patient progress\/regress \u2022 (Q4) Quality of scene \u2022 (Q5) In\ufb02uence of photorealism \u2022 (Q6) Using of controllers or hand tracking \u2022 (Q7) In\ufb02uence of motion sickness \u2022 (Q8) In\ufb02uence of body position (standing, sitting, lying) on the VR experience \u2022 (Q9) Comparison of VR and non-VR rehabilitation \u2022 (Q10) Overall patient satisfaction Table 1 shows the average values and standard deviations within the individual focus groups within the indicators Q1-Q10. Figure 5 shows a histogram of group 1 answers for key questions related to established hypotheses. The same responses for group 2 are shown in Fig. 6.","Possibilities of Rehabilitation and Telerehabilitation of Patients 239 Table 1. Average answers and SD for the questions of the questionnaire for groups 1 and 2 for the evaluation of the impact of rehabilitation using VR. Age Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Group 1 A 54,1 4 4,3 4,4 4,2 4,7 4 2,1 2,2 2 4,1 SD 10,7 0,9 0,8 0,5 0,6 0,5 0,7 1,3 0,9 1,2 1 Group 2 A 54,1 4 4,3 4,4 4,2 4,7 4 2,1 2,2 2 4,1 SD 10,7 0,9 0,8 0,5 0,6 0,5 0,7 1,3 0,9 1,2 1 Fig. 5. Histogram of group 1 answers for key questions related to established hypotheses. Based on the results of rehabilitation therapy for patients, it turned out that the use of virtual reality has brought positive results, especially in patient motivation. For patients who used virtual reality for the \ufb01rst time, the so-called \u201cWOW\u201d effect, which had a positive effect on the motivation of patients to continue rehabilitation. The individual designed and implemented scenarios focused on speci\ufb01c problems of patients, especially in movement (primarily of the upper limbs). In virtual reality, we tried to use the positive motivation of patients. However, the same results of patient therapy can be achieved with classical rehabilitation. Investigating the comparison of time differences in the acquisition of original motor skills in patients who have been rehabilitated with VR and in those who have used classical rehabilitation will be the subject of further research.","240 J. Lacko and E. Ru\u017eick\u00fd Fig. 6. Histogram of group 2 answers for key questions related to established hypotheses. 5 Discussion In the research, we focused on verifying two hypotheses. Based on the results of the questionnaire implemented after the rehabilitation and based on interviews with patients in both focus groups, we identi\ufb01ed that for Hypothesis 1 (Virtual reality accelerates the rehabilitation process based on the fact that the patient is motivated to exercise thanks to the experience of immersion in and virtual environment) are key factors: \u2022 In\ufb02uence of wow effect, \u2022 Quality of scene, in\ufb02uence of photorealism of the virtual environment, \u2022 Motion sickness. Motion sickness can be minimized by using devices that track 6DoF. For Hypothesis 2 (It is more advantageous for patients to interact in the scene without the need for controllers, just by scanning the hands with external sensors using avatars of the upper limbs) are key factors: \u2022 Using of controllers or hand tracking, \u2022 In\ufb02uence of body position. When using means of tele-rehabilitation at home, it turned out that patients require a one-click solution that launches the rehabilitation application directly after turning on the device. Patients who had no previous experience with similar devices had problems starting the application correctly at the beginning of therapy. It has also been shown that virtual environments should be as intuitive as possible with a minimum of graphical user interface controls and distractions.","Possibilities of Rehabilitation and Telerehabilitation of Patients 241 The results of our research show that in this area of patient rehabilitation, we should focus on: \u2022 Creation of new comprehensive rehabilitation systems usable in outpatient clinics and in home use. \u2022 Creating environments that will have an activating character in terms of patient motivation but will also evoke a pleasant atmosphere. \u2022 Design of tele-rehabilitation procedures using virtual reality, including feedback and remote evaluation of patient progress or regression. 6 Conclusion In the initial research, we compared rehabilitation and tele-rehabilitation approaches using virtual reality for patients after overcoming COVID-19. The individual conclu- sions show that the method of virtual reality can have great potential. Based on our observations, we designed and implemented a system consisting of complex scenes and within them interactive tasks focused mainly on the development of upper limb mobility and cognitive abilities. In the future work, we plan to focus on creating a comprehensive system for mea- suring patient progress, which will store data on cloud storage and will evaluate them using arti\ufb01cial intelligence methods. We will also focus on the multi-user connection of the patient and the therapist in one virtual environment using a combination of virtual and augmented reality. To verify the results of the initial study, it is necessary to perform a comparative study on a larger number of patients. Acknowledgements. The research was supported by the GAAA grant agency within the project Advanced applications of arti\ufb01cial intelligence and data display in the industrial and medical domain (GA \/ 6\/2021) and the Tatra banka Foundation within the project Virtual reality and rehabilitation of patients after stroke (2021digvs00). References Carriere, J., Sha\ufb01, H., Brehon, K., Manhas, K.P., et al.: Report: utilizing AI and NLP to assist with healthcare and rehabilitation during the COVID-19 pandemic. Front. Artif. Intell. 4, 613637 (2021) Chatterjee, K., Buchanan, A., Cottrell, K., Hughes, S., Day, T.W., John, N.W.: Immersive virtual reality for the cognitive rehabilitation of stroke survivors. IEEE Trans. Neural Syst. Rehabil. Eng. 719\u2013728 (2022). https:\/\/doi.org\/10.1109\/TNSRE.2022.3158731 Kolbe, L., Jaywant, A., Gupta, A., Vanderlind, W.M., Jabbour, G.: Use of virtual reality in the inpatient rehabilitation of COVID-19 patients. Gen. Hosp. Psychiatry 71, 76\u201381 (2021). https:\/\/ doi.org\/10.1016\/j.genhosppsych.2021.04.008 Lacko, J.: Health safety training for industry in virtual reality. Cybern. Inform. 1\u20135 (2020). https:\/\/ doi.org\/10.1109\/KI48306.2020.9039854 Mao, L., Jin, H., Wang, M., Hu, Y., Chen, S., He, Q., et al.: Neurologic manifestations of hospital- ized patients with coronavirus disease 2019 in Wuhan Chine. JAMA Neurol. 77(6), 683\u2013690 (2020)","242 J. Lacko and E. Ru\u017eick\u00fd Munoz, J., Mehrabi, S., Li, Y., et al.: Immersive Virtual Reality exergames for persons living with dementia: user-centered design study as a multistakeholder team during the COVID-19 pandemic. JMIR Ser. Games 10(1), e29987 (2022) Ru\u017eick\u00fd, E., Ma\u0161\u00e1n, J., \u0160ramka, M.: Arti\ufb01cial Intelligence to rehabilitation for post-COVID syndrome. Int. J. Healthcare New Technol. Soc. Work 16(4), 149\u2013158 (2020) Ru\u017eick\u00fd, E., Lacko, J., Ma\u0161\u00e1n, J., Tim\u00e1rov\u00e1, G., Sramka, M.: Virtual reality for rehabilitation of \ufb01ne motor functions of hands in Parkinson\u2019s disease. Rehabilit\u00e1cia, 59(1), 89\u201398 (2022) (Obzor, Bratislava. ISSN 0375-0922) \u0160ramka, M., Lacko, J., Ru\u017eick\u00fd, E., Ma\u0161\u00e1n, J.: Combined methods of rehabilitation of patients after stroke: virtual reality and traditional approach. Neuroendocrinol. Let. 41(3), 101\u2013111 (2020) \u0160ramka, M., Ma\u0161\u00e1n, J., Ru\u017eick\u00fd, E., Prid\u00e1vkov\u00e1, Z.: Post-COVID syndrome and nervous system. Int. J. Healthcare New Technol. Soc. Work 16(3), 101\u2013110 (2021) Tsutsui, M., Gerayeli, F., Sin, D.D.: Pulmonary rehabilitation in a post-Covid-19 world: telereha- bilitation as a new standard in patients with COPD. Int. J. COPD 16, 379\u2013391 (2021). https:\/\/ doi.org\/10.2147\/COPD.S263031 Varatharaj, A., Thomas, N., Ellul, M., et al.: UK-wide surveillance of neurological and neuropsy- chiatric complications of COVID-19 the \ufb01rst 153 patients. SSRN. 3601761 (2020). 10.1016\/ S2215-0366 (20) 30287-X Vourganas, I., Stankovic, V., Stankovic, L.: Individualised responsible arti\ufb01cial intelligence for home-based rehabilitation. Sensors 21(1), 2 (2021). https:\/\/doi.org\/10.3390\/s21010002 Whitelaw, S., et al.: Applications of digital technology in COVID-19 pandemic planning and response. Lancet Dig. Health 2(8), e435\u2013e440 (2020)","Effect of Virtual Reality Based Nursing Skills on the Performance Ability, Performance Con\ufb01dence, and Practice Satisfaction of Nursing College Students Yonghwa Lee1, Minjeong Suh2(B), and Timothy Jung2 1 Department of Nursing, Kunsan College of Nursing, Gunsan, Republic of Korea [email protected] 2 Faculty of Business and Law, Manchester Metropolitan University, Manchester, UK [email protected], [email protected] Abstract. The development and application of remote learning using immersive technology have been accelerated in nursing colleges due to recent technologi- cal advances and the Covid-19 pandemic. Although virtual nursing education is effective in improving nursing competency by improving metacognition, there is limited research to examine the effectiveness of virtual nursing education. There- fore, this study aims to examine the effects of using virtual reality to learn core nursing techniques measured via ratings of performance ability, con\ufb01dence, and practice satisfaction. Thirty nursing students were split into three experimental groups based on practice content utilized: 1) HMD-based VR, 2) Web-based VR, and 3) video practice. Each group was required to perform intradermal and intra- venous injection techniques. Con\ufb01dence and satisfaction of the group using VR content were rated higher than the group with general video learning. This study supports that virtual reality-based nursing education is a useful tool for practical education through metacognition and could serve as a new alternative to nursing education in a pandemic environment. Keywords: Performance ability \u00b7 Performance con\ufb01dence \u00b7 Practice satisfaction \u00b7 VR in nursing education 1 Introduction Nursing is a practical discipline that must cultivate the ability to solve the subject\u2019s health problems in practice based on professional theory (Yoo 2000). To this end, nurs- ing students are in a situation where repetitive learning and training are essential for the success of their respective skills (Paik 2004). The Korean Association Board of Nursing Education (KABONE) developed 20 standardized nursing techniques and re\ufb02ected them in the practical training of nursing universities. By the 4th cycle of nursing education certi\ufb01cation evaluation in 2022, 18 courses\/categories were applied by deleting, chang- ing, and adding some techniques. New nurses with high con\ufb01dence in performing core \u00a9 The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 T. Jung et al. (Eds.): XR 2022, SPBE, pp. 243\u2013252, 2023. https:\/\/doi.org\/10.1007\/978-3-031-25390-4_21","244 Y. Lee et al. nursing were found to also have high performance ability for core nursing after hospital employment and low work stress (Bang & Kim 2014). Therefore, suf\ufb01cient education on core nursing is expected to improve the \ufb01eld adaptability of new nurses after graduation. In order to facilitate this growth, nursing colleges have a system to complete core nursing techniques step by step according to their dif\ufb01culty level. Evaluation and education are applied not only to basic nursing practice, but also to clinical practice subjects; however, this core nursing education is mainly conducted through lectures, in-school practice, and video (Cheng et al. 2014). As a result, it is very dif\ufb01cult for students to acquire skills directly at the hospital practice stage (Oh 2017). In addition, clinical practice for nursing students is becoming increasingly dif\ufb01cult due to the lack of institutions capable of clinical practice compared to the number of nursing students (Medley & Horne 2005). Therefore, it is necessary to develop and apply various practical education methods to improve the core nursing performance of nursing students. Various technical education methods have been proposed and used, such as practice using standardized patients, sim- ulators, and self-practice. However, in the case of practice using standardized patients, there is a limit to \ufb01nding and paying for trained standardized patients, and it is dif\ufb01cult to have consistent results due to differences in response methods for each subject. Fur- thermore, there is a limit to applying these practices to technical education that requires repetitive learning. Practical training using simulators has a limited number of oppor- tunities for repetitive learning because the number of guidance teachers is smaller than that of students in nursing colleges, meaning the explanation, demonstration, and stu- dent observation of these simulations are bottlenecked by the capacity of teachers the institution may have (Song & Kim 2015). In the case of autonomous practice, students\u2019 con\ufb01dence and satisfaction are increased as a method of learning skills on their own without instructors, but attendance checks to improve participation in autonomous prac- tice cause passive attitudes of nursing students and have limitations in terms of practical accuracy (Paik 2004). In addition, it was recognized that the recent COVID-19 pan- demic required the development and application of various virtual learning mediums in comparison to practical face-to-face education nursing colleges. The recent technolog- ical advancement suggests that practical education that combines various information communication technologies is possible. Among them, virtual reality is a technology that stimulates users\u2019 senses and makes them feel real through immersive interaction (Padilha et al. 2018; Verkuyl & Hughes 2019) by using computers to provide virtual space (Aukstakalnis & Blatner 1992) to overcome the limitation of existing learning and practice. Although virtual nursing education is effective in improving nursing compe- tency, there is limited research to examine the effectiveness of virtual nursing education. This study aims to examine the effectiveness of nursing education by utilizing virtual reality and provide empirical evidence. 2 Literature Review 2.1 Self-con\ufb01dence in Nursing Skills Self-con\ufb01dence is one of the key components for an effective and safe clinical per- formance. Student nurses must manage clinical situations during training to build and","Effect of Virtual Reality Based Nursing Skills 245 strengthen self-con\ufb01dence; it is important to be able to cope with complex clinical sit- uations and build con\ufb01dence. High \ufb01delity simulation (HFS) has been shown to be very useful in developing and increasing con\ufb01dence in nursing students. Simulation training is a novel tool that holds a lot of potential in the \ufb01eld of health. Improving con\ufb01dence can result in improvements in patient care and safe clinical performances (Bailey & Emory 2022). Students who undergo more simulation training report a sig- ni\ufb01cant increase in their professional con\ufb01dence immediately following training. The effect is students reporting twice as much con\ufb01dence after using technology as compared to using non-technical technology. Simulation training has a signi\ufb01cant positive effect on the professional con\ufb01dence of nursing students and it also seems to have a positive effect on the acquisition of nursing knowledge. In other words, improved con\ufb01dence can lead to improved patient care and safe clinical outcomes (Fuglsang et al. 2022). Therefore, simulation training can also contribute to fundamental pedagogical strategies when training health professionals and is characterized by high satisfaction and safety, which are two main goals of nursing (Mesquita et al. 2019). Through role modelling, nursing students can build con\ufb01dence and enjoy a smooth transition into the nursing workforce (Rosli et al. 2022). 2.2 Satisfaction with Nursing Skill Education Student satisfactions results from an individual\u2019s assessment of the educational experi- ences, as well as provided services and facilities. The constructivist learning theory states that an individual gains more satisfaction in learning by understanding abstract concepts, thinking critically, and learning new concepts. This theory is an innovative education framework that adapts to students\u2019 needs through practical experiences. In particular, clinical experiences are very important for students studying medicine and nursing, and videos are used in class to enhance students\u2019 experiences. After asking students ques- tions before and after class, it was found that 68% of students felt satis\ufb01ed with the class, and the average score of satisfaction increased by 10% (Nurse-Clarke et al. 2022). The fundamental way for professional nurses in medical institutions to manage safe, accurate, and high-quality patient care is to gain and maintain suf\ufb01cient self-con\ufb01dence. Professional and student nurses exposed to high \ufb01delity simulation (HFS) can improve self-con\ufb01dence by interacting in realistic scenarios provided by simulation methods. This con\ufb01dence represents the successful learning experience of nursing students and leads to high satisfaction levels. Simulation programs improve the satisfaction of nurs- ing education despite unstable and irrational changes in patients\u2019 clinical conditions. This means that nursing students can use VR technology to increase patient safety and self-con\ufb01dence (Guerrero et al. 2022). 2.3 VR in Nursing Education There are several previous studies that examined the applications of virtual reality in nursing education. The study by Behmadi et al. (2022) revealed that virtual educa- tion provides a practical medical education in a safe, cost-effective, reproducible and \ufb02exible learning environment that can be practiced repeatedly. Additionally, immersive VR training improves nursing competency by improving metacognition (Rim and Shin"]