TeleSTAR
TeleStar: Telepresence for Surgical Assistance and Training using Augmented Reality
Duration: January 2020 - December 2020
Goals and Motivation
Telepresence for Surgical Assistance and Training using Augmented Reality, uses a full digital surgical microscope and augmented/mixed reality to provide new tools that permit teaching and intraoperative assistance. The system offers new ways to visualize relevant surgical information, allowing surgeons to improve, and also explain, their decisions. Surgical decisions can be easily visualized during course units, on high-resolution screens or a head-mounted display, in any local or remote location. The system guarantees the exact same surgeon’s field-of-view for all attending healthcare professionals. “Simple” standard procedures, as well as rare and complicated procedures, can be visually annotated and augmented – and supplemented with audio comments – to make every step of the surgery transparent. A bidirectional communication platform allows all attendees to address questions to the surgeon at the right time, which creates a dynamic teaching environment despite local distance. The teaching concept can be scaled to other surgical domains and existing degree programs.
Current Problem
There is currently a lack of pedagogical and technical education and training measures for surgical interventions to reduce the associated digital illiteracy. The intraoperative surgeon’s view can only be conveyed to third parties (senior/chief physicians, biomedical engineering students) to a limited extent during advanced trainings and surgical workshops. This specifically concerns the surgical field-of-view as well as the lack of image-based annotation and commentary options that are directly linked to the anatomical scene. Within the framework of EIT-Health, two projects are currently being funded that try to address the above-mentioned problems with VR. But, there are currently no new training concepts for the promising approaches based on augmented reality (AR). Furthermore, there is a lack of networked IP-based solutions that are able to transfer intraoperative assistance and teaching systems with additional AR information to remote operating rooms or lecture halls. Especially in Germany, there is currently no program that is awarded with CME.
Solution
TeleSTAR will develop a training concept for healthcare professionals namely senior and chief physicians in ENT surgery using augmented reality (AR) in combination with computer vision algorithms. The healthcare professional networks of TU Delft, Charité – Universitätsmedizin Berlin and ARRI Medical will be brought together. ENT procedures will be performed using the full digital surgical microscope “ARRISCOPE”. ENT chief and senior physicians from the German network will attend the operation in Berlin. The procedures will be transmitted to remote lecture rooms in Delft. The recorded annotated video material will then be made available to all participants for discussion and teaching purposes. The training effect will be monitored with a digital questionnaire. Because of limited me for physicians, we will organize intensive 1-2 days sessions 0. Technical Demo, 1. Description of patient’s history, 2. Preoperative examinations and imaging will be presented to the learners, 3. Participants can suggest therapeutic strategy, which is discussed with the expert surgeon, 4. We will use teledialog tools to ensure interactivity and make live transmissions in 3D, 5. During live transmission, relevant surgical information will be highlighted with AR tools, 6. Finally, a 2nd expert will create an ad hoc questionnaire based on the performed intervention and sent to remote learners. The learning outcome will be monitored based on the correct answers and will be awarded with CME points.
Project Partners
· Fraunhofer HHI
· ARRI Medical
· Charité –Universitätsmedizin Berlin, Klinik für HNO
· TU Delft
Publications
E. L. Wisotzky, J.-C. Rosenthal, P. Eisert, A. Hilsmann, F. Schmid, M. Bauer, A. Schneider, F. C. Uecker,
Interactive and Multimodal-based Augmented Reality for Remote Assistance using a Digital Surgical Microscope,
IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 2019
A. Schneider, M. Lanski, M. Bauer, E. L. Wisotzky, J.-C. Rosenthal,
An AR-Solution for Education and Consultation during Microscopic Surgery,
Proc. Computer Assisted Radiology and Surgery (CARS), Rennes, France, 2019.