Exploration of temporal bone anatomy using mixed reality (HoloLens): development of a mixed reality anatomy teaching resource prototype

Pavithran Maniam; Philipp Schnell; Lilly Dan; Rony Portelli; Caroline Erolin; Rodney Mountain; Tracey Wilkinson

doi:10.1080/17453054.2019.1671813

Exploration of temporal bone anatomy using mixed reality (HoloLens): development of a mixed reality anatomy teaching resource prototype

Pavithran Maniam^*, Philipp Schnell, Lilly Dan, Rony Portelli, Caroline Erolin, Rodney Mountain, Tracey Wilkinson

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

23 Citations (Scopus)

Abstract

Mixed reality (MR), a technology which supplements the real world with virtual objects, is increasingly becoming available as a teaching tool in medical education. The Microsoft HoloLens device allows operators to experience MR using a head-mounted device without interfering with their physical reality, stimulating a realistic learning experience using virtual objects. This project aimed to develop a MR anatomy teaching application with HoloLens for exploring the anatomy of the temporal bone. The educational application was developed from a multidisciplinary collaboration between undergraduate and postgraduate students across several academic disciplines with Medtronic, a medical technology company. 3D anatomical models were built using ZBrush and Blender, while the HoloLens1 application was developed using Windows 10, Visual Studio 2017, Unity and Mixed Reality Toolkit (MRTK). Modules developed within the application included a basic HoloLens tutorial, a virtual temporal bone with surgical anatomy landmarks and free drilling of the temporal bone. The basic tutorial allows the operator to adapt to the MR environment prior to exploring the anatomical landmarks of the 3D temporal bone. The free drilling of the temporal bone using vertex displacement and texture stretching replicates a real-time bone drilling experience and allows the operator to explore the anatomical relationship between different otological structures.

Original language	English
Pages (from-to)	17-26
Number of pages	10
Journal	Journal of Visual Communication in Medicine
Volume	43
Issue number	1
DOIs	https://doi.org/10.1080/17453054.2019.1671813
Publication status	Published - 2 Jan 2020

Bibliographical note

Funding Information:
The authors would like to acknowledge Stefanie Schultz, Naomi Culpin and Charlie Ritchie for the assistance they provided during development of the mixed reality application. Further thanks to Danny Verkissen, Nicolas Denervaud and Agnieszka Strzembosz from Medtronic for funding the project and providing technical support and relevant resources. Lastly, many thanks to Gareth James from COMBlox Ltd for providing technical advice and support on the Microsoft HoloLens software. Materials in this manuscript were orally presented in the Anatomical Society Winter Meeting in 2017.

Funding Information:
Medtronic Public Limited Company (to P.M., P.S., R.P. and L.D.) to develop a mixed reality educational tool using HoloLens. The authors would like to acknowledge Stefanie Schultz, Naomi Culpin and Charlie Ritchie for the assistance they provided during development of the mixed reality application. Further thanks to Danny Verkissen, Nicolas Denervaud and Agnieszka Strzembosz from Medtronic for funding the project and providing technical support and relevant resources. Lastly, many thanks to Gareth James from COMBlox Ltd for providing technical advice and support on the Microsoft HoloLens software. Materials in this manuscript were orally presented in the Anatomical Society Winter Meeting in 2017.

Publisher Copyright:
© 2019, © 2019 The Institute of Medical Illustrators.

Keywords

educational tool
ENT
HoloLens
mixed reality
surgical
Temporal bone

ASJC Scopus subject areas

Health Professions (miscellaneous)
Visual Arts and Performing Arts

Access to Document

10.1080/17453054.2019.1671813

Cite this

@article{399652763e6a45158dc6147510b288f9,

title = "Exploration of temporal bone anatomy using mixed reality (HoloLens): development of a mixed reality anatomy teaching resource prototype",

abstract = "Mixed reality (MR), a technology which supplements the real world with virtual objects, is increasingly becoming available as a teaching tool in medical education. The Microsoft HoloLens device allows operators to experience MR using a head-mounted device without interfering with their physical reality, stimulating a realistic learning experience using virtual objects. This project aimed to develop a MR anatomy teaching application with HoloLens for exploring the anatomy of the temporal bone. The educational application was developed from a multidisciplinary collaboration between undergraduate and postgraduate students across several academic disciplines with Medtronic, a medical technology company. 3D anatomical models were built using ZBrush and Blender, while the HoloLens1 application was developed using Windows 10, Visual Studio 2017, Unity and Mixed Reality Toolkit (MRTK). Modules developed within the application included a basic HoloLens tutorial, a virtual temporal bone with surgical anatomy landmarks and free drilling of the temporal bone. The basic tutorial allows the operator to adapt to the MR environment prior to exploring the anatomical landmarks of the 3D temporal bone. The free drilling of the temporal bone using vertex displacement and texture stretching replicates a real-time bone drilling experience and allows the operator to explore the anatomical relationship between different otological structures.",

keywords = "educational tool, ENT, HoloLens, mixed reality, surgical, Temporal bone",

author = "Pavithran Maniam and Philipp Schnell and Lilly Dan and Rony Portelli and Caroline Erolin and Rodney Mountain and Tracey Wilkinson",

note = "Funding Information: The authors would like to acknowledge Stefanie Schultz, Naomi Culpin and Charlie Ritchie for the assistance they provided during development of the mixed reality application. Further thanks to Danny Verkissen, Nicolas Denervaud and Agnieszka Strzembosz from Medtronic for funding the project and providing technical support and relevant resources. Lastly, many thanks to Gareth James from COMBlox Ltd for providing technical advice and support on the Microsoft HoloLens software. Materials in this manuscript were orally presented in the Anatomical Society Winter Meeting in 2017. Funding Information: Medtronic Public Limited Company (to P.M., P.S., R.P. and L.D.) to develop a mixed reality educational tool using HoloLens. The authors would like to acknowledge Stefanie Schultz, Naomi Culpin and Charlie Ritchie for the assistance they provided during development of the mixed reality application. Further thanks to Danny Verkissen, Nicolas Denervaud and Agnieszka Strzembosz from Medtronic for funding the project and providing technical support and relevant resources. Lastly, many thanks to Gareth James from COMBlox Ltd for providing technical advice and support on the Microsoft HoloLens software. Materials in this manuscript were orally presented in the Anatomical Society Winter Meeting in 2017. Publisher Copyright: {\textcopyright} 2019, {\textcopyright} 2019 The Institute of Medical Illustrators.",

year = "2020",

month = jan,

day = "2",

doi = "10.1080/17453054.2019.1671813",

language = "English",

volume = "43",

pages = "17--26",

journal = "Journal of Visual Communication in Medicine",

issn = "1745-3054",

publisher = "Taylor & Francis",

number = "1",

}

TY - JOUR

T1 - Exploration of temporal bone anatomy using mixed reality (HoloLens)

T2 - development of a mixed reality anatomy teaching resource prototype

AU - Maniam, Pavithran

AU - Schnell, Philipp

AU - Dan, Lilly

AU - Portelli, Rony

AU - Erolin, Caroline

AU - Mountain, Rodney

AU - Wilkinson, Tracey

N1 - Funding Information: The authors would like to acknowledge Stefanie Schultz, Naomi Culpin and Charlie Ritchie for the assistance they provided during development of the mixed reality application. Further thanks to Danny Verkissen, Nicolas Denervaud and Agnieszka Strzembosz from Medtronic for funding the project and providing technical support and relevant resources. Lastly, many thanks to Gareth James from COMBlox Ltd for providing technical advice and support on the Microsoft HoloLens software. Materials in this manuscript were orally presented in the Anatomical Society Winter Meeting in 2017. Funding Information: Medtronic Public Limited Company (to P.M., P.S., R.P. and L.D.) to develop a mixed reality educational tool using HoloLens. The authors would like to acknowledge Stefanie Schultz, Naomi Culpin and Charlie Ritchie for the assistance they provided during development of the mixed reality application. Further thanks to Danny Verkissen, Nicolas Denervaud and Agnieszka Strzembosz from Medtronic for funding the project and providing technical support and relevant resources. Lastly, many thanks to Gareth James from COMBlox Ltd for providing technical advice and support on the Microsoft HoloLens software. Materials in this manuscript were orally presented in the Anatomical Society Winter Meeting in 2017. Publisher Copyright: © 2019, © 2019 The Institute of Medical Illustrators.

PY - 2020/1/2

Y1 - 2020/1/2

N2 - Mixed reality (MR), a technology which supplements the real world with virtual objects, is increasingly becoming available as a teaching tool in medical education. The Microsoft HoloLens device allows operators to experience MR using a head-mounted device without interfering with their physical reality, stimulating a realistic learning experience using virtual objects. This project aimed to develop a MR anatomy teaching application with HoloLens for exploring the anatomy of the temporal bone. The educational application was developed from a multidisciplinary collaboration between undergraduate and postgraduate students across several academic disciplines with Medtronic, a medical technology company. 3D anatomical models were built using ZBrush and Blender, while the HoloLens1 application was developed using Windows 10, Visual Studio 2017, Unity and Mixed Reality Toolkit (MRTK). Modules developed within the application included a basic HoloLens tutorial, a virtual temporal bone with surgical anatomy landmarks and free drilling of the temporal bone. The basic tutorial allows the operator to adapt to the MR environment prior to exploring the anatomical landmarks of the 3D temporal bone. The free drilling of the temporal bone using vertex displacement and texture stretching replicates a real-time bone drilling experience and allows the operator to explore the anatomical relationship between different otological structures.

AB - Mixed reality (MR), a technology which supplements the real world with virtual objects, is increasingly becoming available as a teaching tool in medical education. The Microsoft HoloLens device allows operators to experience MR using a head-mounted device without interfering with their physical reality, stimulating a realistic learning experience using virtual objects. This project aimed to develop a MR anatomy teaching application with HoloLens for exploring the anatomy of the temporal bone. The educational application was developed from a multidisciplinary collaboration between undergraduate and postgraduate students across several academic disciplines with Medtronic, a medical technology company. 3D anatomical models were built using ZBrush and Blender, while the HoloLens1 application was developed using Windows 10, Visual Studio 2017, Unity and Mixed Reality Toolkit (MRTK). Modules developed within the application included a basic HoloLens tutorial, a virtual temporal bone with surgical anatomy landmarks and free drilling of the temporal bone. The basic tutorial allows the operator to adapt to the MR environment prior to exploring the anatomical landmarks of the 3D temporal bone. The free drilling of the temporal bone using vertex displacement and texture stretching replicates a real-time bone drilling experience and allows the operator to explore the anatomical relationship between different otological structures.

KW - educational tool

KW - ENT

KW - HoloLens

KW - mixed reality

KW - surgical

KW - Temporal bone

UR - http://www.scopus.com/inward/record.url?scp=85076463528&partnerID=8YFLogxK

U2 - 10.1080/17453054.2019.1671813

DO - 10.1080/17453054.2019.1671813

M3 - Article

C2 - 31645155

AN - SCOPUS:85076463528

SN - 1745-3054

VL - 43

SP - 17

EP - 26

JO - Journal of Visual Communication in Medicine

JF - Journal of Visual Communication in Medicine

IS - 1

ER -

Exploration of temporal bone anatomy using mixed reality (HoloLens): development of a mixed reality anatomy teaching resource prototype

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this