Motivation
Intraoperative imaging is currently performed using white light illumination and surgical decisions are taken without additional information of further visual assisting tools resulting in disadvantages as not all available information have been exploited. The project MultiARC targets closing this information gap and reducing surgical duration as well as complications and revision surgeries.
Goals and Approaches
The focus of MultiARC is to combine hyperspectral imaging, 3D-measurement and a true-to-scale augmentation of clinical images for ear, nose and throat (ENT)-surgery in microscopy and endoscopy. To achieve this goal, we develop an intraoperative system for a collaborative and immersive decision-making chain. This system includes a novel illumination unit with suitable evaluation algorithms for tissue differentiation and a stereo-image-based measurement solution. The intraoperative images are updated according to the captured information and augmented with the additional information (tissue type and measurements). To achieve a true-to-scale 3D visualization, a robust calibration setting of stereo-zoom optics will be developed. The evaluation and a transfer into other application areas is done in the AR/VR-Living-Lab.
Innovation and Outlook
The MultiARC system allows intuitive and interactive handling of AR-data for multilateral telepresence. The multimodal data is visualized in the binocular of the microscope, on AR/VR-glasses and 3D-displays. The augmentation of virtual prostheses allows an exact decision-making of the implant sizes and will reduces revisions.
Find more information and updates on the project website.
Fraunhofer HHI is responsible for the development of the novel hyperspectral tissue differentiation algorithms as well as the stereo-image-based measurement solution for microscopic and endoscopic applications.
Project Consortium
- pripares GmbH
- ARRI Medical
- Fraunhofer HHI
- Ubimax GmbH
- Charité – Universitätsmedizin Berlin, Klink für HNO
- Schölly Fiberoptic GmbH (associated partner)
Related Research Topics
Award
Jean-Claude Rosenthal, Zhenglei Hu, Niklas Gard, Peter Eisert
Endoscopic Vision Challenge - Stereo Correspondence and Reconstruction of Endoscopic Data (SCARED), organized by Intuitive Surgical, 22nd International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), Shenzhen, China, October 2019. Runners-Up Award
Publications
E. L. Wisotzky, P. Arens, S. Dommerich, A. Hilsmann, P. Eisert, F. C. Uecker,
Determination of optical properties of cholesteatoma in the spectral range of 250 to 800 nm,
Biomedical Optics Express, 11(3), 2020.
E. L. Wisotzky, B. Kossack, F. C. Uecker, P. Arens, S. Dommerich, A. Hilsmann, P. Eisert,
Validation of two techniques for intraoperative hyperspectral human tissue determination,
Proceedings of SPIE, 10951:109511Z, 2019.
E. L. Wisotzky, F. C. Uecker, S. Dommerich, A. Hilsmann, P. Eisert, P. Arens,
Determination of optical properties of human tissues obtained from parotidectomy in the spectral range of 250 to 800 nm, Journal of Biomedical Optics, 24(12):125001, 2019.
E. L. Wisotzky, J.-C. Rosenthal, P. Eisert, A. Hilsmann, F. Schmid, M. Bauer, A. Schneider, F. C. Uecker,
Interactive and Multimodal-based Augmented Reality for Remote Assistance using a Digital Surgical Microscope, IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 2019.
N. Gard, J.-C. Rosenthal, S. Jurk, A. Schneider, P. Eisert,
Image-based measurement by instrument tip tracking for tympanoplasty using digital surgical microscopy, Proc. of SPIE, 10951:1095119, 2019.
J.-C. Rosenthal, P. Eisert
Remote Visual Inspection, A Real-Time Image-Based Stereo Video Processing Chain for 3D Endoscopy, inspect - World of Vision, international, Wiley, no. 2, pp. 44-47, September 2019