Medicine Meets Virtual Reality 18: NextMed - Volume 163 Studies in Health Technology and Informatics

MEDICINE MEETS VIRTUAL REALITY 18 Studies in Health Technology and Informatics This book series was started in 1990 to...

Author: J.D. Westwood | S.W. Westwood | L. Fellander-Tsai | R.S. Haluck | H.M. Hoffman | R.A. Robb | S. Senger | K.G. Vosburgh

16 downloads 1932 Views 16MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

MEDICINE MEETS VIRTUAL REALITY 18

Studies in Health Technology and Informatics This book series was started in 1990 to promote research conducted under the auspices of the EC programmes’ Advanced Informatics in Medicine (AIM) and Biomedical and Health Research (BHR) bioengineering branch. A driving aspect of international health informatics is that telecommunication technology, rehabilitative technology, intelligent home technology and many other components are moving together and form one integrated world of information and communication media. The complete series has been accepted in Medline. Volumes from 2005 onwards are available online. Series Editors: Dr. O. Bodenreider, Dr. J.P. Christensen, Prof. G. de Moor, Prof. A. Famili, Dr. U. Fors, Prof. A. Hasman, Prof. E.J.S. Hovenga, Prof. L. Hunter, Dr. I. Iakovidis, Dr. Z. Kolitsi, Mr. O. Le Dour, Dr. A. Lymberis, Prof. J. Mantas, Prof. M.A. Musen, Prof. P.F. Niederer, Prof. A. Pedotti, Prof. O. Rienhoff, Prof. F.H. Roger France, Dr. N. Rossing, Prof. N. Saranummi, Dr. E.R. Siegel, Prof. T. Solomonides and Dr. P. Wilson

Volume 163 Recently published in this series Vol. 162. E. Wingender (Ed.), Biological Petri Nets Vol. 161. A.C. Smith and A.J. Maeder (Eds.), Global Telehealth – Selected Papers from Global Telehealth 2010 (GT2010) – 15th International Conference of the International Society for Telemedicine and eHealth and 1st National Conference of the Australasian Telehealth Society Vol. 160. C. Safran, S. Reti and H.F. Marin (Eds.), MEDINFO 2010 – Proceedings of the 13th World Congress on Medical Informatics Vol. 159. T. Solomonides, I. Blanquer, V. Breton, T. Glatard and Y. Legré (Eds.), Healthgrid Applications and Core Technologies – Proceedings of HealthGrid 2010 Vol. 158. C.-E. Aubin, I.A.F. Stokes, H. Labelle and A. Moreau (Eds.), Research into Spinal Deformities 7 Vol. 157. C. Nøhr and J. Aarts (Eds.), Information Technology in Health Care: Socio-Technical Approaches 2010 – From Safe Systems to Patient Safety Vol. 156. L. Bos, B. Blobel, S. Benton and D. Carroll (Eds.), Medical and Care Compunetics 6 Vol. 155. B. Blobel, E.Þ. Hvannberg and V. Gunnarsdóttir (Eds.), Seamless Care – Safe Care – The Challenges of Interoperability and Patient Safety in Health Care – Proceedings of the EFMI Special Topic Conference, June 2–4, 2010, Reykjavik, Iceland Vol. 154. B.K. Wiederhold, G. Riva and S.I. Kim (Eds.), Annual Review of Cybertherapy and Telemedicine 2010 – Advanced Technologies in Behavioral, Social and Neurosciences Vol. 153. W.B. Rouse and D.A. Cortese (Eds.), Engineering the System of Healthcare Delivery ISSN 0926-9630 (print) ISSN 1879-8365 (online)

Medicine Meets Virtual Reality 18 NextMed

Edited by

James D. Westwood Susan W. Westwood MA Li Felländer-Tsai MD PhD Randy S. Haluck MD FACS Helene M. Hoffman PhD Richard A. Robb PhD Steven Senger PhD and

Kirby G. Vosburgh PhD

Amsterdam • Berlin • Tokyo • Washington, DC

© 2011 The authors. All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without prior written permission from the publisher. ISBN 978-1-60750-705-5 (print) ISBN 978-1-60750-706-2 (online) Library of Congress Control Number: 2011920396 Publisher IOS Press BV Nieuwe Hemweg 6B 1013 BG Amsterdam Netherlands fax: +31 20 687 0019 e-mail: [email protected] Distributor in the USA and Canada IOS Press, Inc. 4502 Rachael Manor Drive Fairfax, VA 22032 USA fax: +1 703 323 3668 e-mail: [email protected]

LEGAL NOTICE The publisher is not responsible for the use which might be made of the following information. PRINTED IN THE NETHERLANDS

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved.

v

Preface James D. WESTWOOD Aligned Management Associates, Inc. ENIAC, the first electronic universal digital computer, was born on Valentine’s Day 1946—a lifetime ago. It and its emerging peers were elephantine contraptions, but they evolved rapidly, increasing in speed and shrinking in size, adopting efficiencies of scale in reproduction and mutating continuously. Who are their offspring today? Five billion mobile phones and similarly ubiquitous personal and business computers in countless variations. What was once a costly academic and military project is now an everyday tool. When Medicine Meets Virtual Reality launched in 1992, computers were already popular in most of the industrialized world, although relatively expensive and clunky. (Remember the dot-matrix printer?) The Internet was about to make its commercial debut, providing a means to link all these solitary devices into a communicating, sharing, interactive meta-forum. More so than print, the computer was image-friendly. Unlike television and cinema, the computer-plus-Internet was multi-directional—users could create and share a moving image. Cinema and TV were meeting their eventual heir as “virtual reality” arrived on the scene. At MMVR, virtual reality becomes a theater for medicine, where multiple senses are engaged—sight, sound, and touch—and language and image fuse. (Taste and smell are still under-utilized, alas.) Simulation lets actors rehearse in any number of ways, interrupting and reconfiguring the plot to create the most compelling finale. Visualization alters costumes to clarify relationships, and shifts sets and lighting to sharpen focus or obscure a background. Impromptu lines are recorded for possible adoption into the standard repertoire. Audience members, who need not be physically present, may chat with the actors mid-performance or take on a role themselves. Critics can instantly share their opinions. Whether the actors and audience are physicians, patients, teachers, students, industry, military, or others with a role in contemporary healthcare, the theater of virtual reality provides a singular tool for understanding relationships. Medical information can be presented in ways not possible in books, journals, or video. That information can be manipulated, refined, recontextualized, and reconsidered. Experience finds a wider audience than would fit in a surgical suite or classroom. Therapeutic outcomes can be reverse engineered. Precisely because the theater is unreal, the risks of experimentation and failure vanish, while the opportunity to understand remains. The availability and veracity of this educational virtual theater are improving due to steady technological improvement: this is the purpose of MMVR. Most of the industrialized world is currently undergoing an economic correction whose end result is far from clear. The happier news is that many emerging economies continue to flourish during the downturn. Furthermore, knowledge resources that were once the privilege of wealthier countries are now more easily shared, via computers and the Internet, with those who are catching up. Children (and adults) are being

vi

trained on inexpensive and interconnected devices, acquiring literacy and a better chance at higher education. Healthcare is an important part of this worldwide dissemination of expertise enabled by the virtual theater of learning. As developing regions progress, their most creative minds can take part in the quest for what’s next in medicine. The vision of a better educated, more productive, and healthier global population is clarified. Someone born in 1992, as was MMVR, could be attending a university now. She or he might be working on research that is shared at this conference. We who organize MMVR would like to thank the many researchers who, for a generation, have come from around the world to meet here with the aim of making very real improvements in medicine.

vii

MMVR18 Proceedings Editors James D. Westwood MMVR18 Conference Organizer Aligned Management Associates, Inc. Susan W. Westwood MA MMVR18 Proceedings Coordinator Aligned Management Associates, Inc. Li Felländer-Tsai MD PhD Professor, Department of Orthopedics Director, Center for Advanced Medical Simulation and Training Chair, Department of Clinical Science, Intervention and Technology Karolinska University Hospital Karolinska Institutet Randy S. Haluck MD FACS Professor of Surgery Chief, Minimally Invasive Surgery and Bariatrics Vice Chair for Technology and Innovation Penn State, Hershey Medical Center Helene M. Hoffman PhD Assistant Dean, Educational Computing Adjunct Professor of Medicine Division of Medical Education School of Medicine University of California, San Diego Richard A. Robb PhD Scheller Professor in Medical Research Professor of Biophysics & Computer Science Director, Biomedical Imaging Research Laboratory Mayo Clinic College of Medicine Steven Senger PhD Professor and Chair, Department of Computer Science Professor, Department of Mathematics University of Wisconsin – La Crosse Kirby G. Vosburgh PhD Assistant Professor of Radiology Brigham & Women’s Hospital Harvard Medical School

viii

MMVR18 Organizing Committee Michael J. Ackerman PhD National Library of Medicine Kóan Jeff Baysa MD Vera List Center for Art and Politics; The New School Steve Charles MD MicroDexterity Systems; University of Tennessee Patrick C. Cregan FRACS Nepean Hospital, Sydney West Area Health Service Li Felländer-Tsai MD PhD Karolinska University Hospital; Karolinska Institutet Cali M. Fidopiastis PhD University of Alabama at Birmingham Henry Fuchs PhD University of North Carolina Walter J. Greenleaf PhD Greenleaf Medical Systems; InWorld Solutions; Virtually Better Randy S. Haluck MD FACS Penn State, Hershey Medical Center David M. Hananel CAE Healthcare Wm. LeRoy Heinrichs MD PhD Stanford University School of Medicine Helene M. Hoffman PhD University of California, San Diego Kanav Kahol PhD Arizona State University Mounir Laroussi PhD Old Dominion University Heinz U. Lemke PhD Technical University Berlin Alan Liu PhD Uniformed Services University

ix

Bertalan Meskó MD University of Debrecen; Webicina.com Greg T. Mogel MD Kaiser Permanente Kevin N. Montgomery PhD Stanford University Makoto Nonaka MD PhD Foundation for International Scientific Advancement Roger Phillips PhD CEng FBCS CIPT University of Hull; Vertual, Ltd. Carla M. Pugh MD PhD Northwestern University Giuseppe Riva PhD Università Cattolica del Sacro Cuore di Milano Albert A. Rizzo PhD University of Southern California Richard A. Robb PhD Mayo Clinic College of Medicine Jannick P. Rolland PhD University of Rochester; University of Central Florida Anand P. Santhanam PhD University of California, Los Angeles Richard M. Satava MD FACS University of Washington Steven Senger PhD University of Wisconsin – La Crosse Ramin Shahidi PhD Stanford University School of Medicine Yunhe Shen PhD University of Minnesota Marshall Smith MD PhD Banner Good Samaritan Medical Center Thomas Sangild Sørensen PhD University of Aarhus

x

Don Stredney Ohio Supercomputer Center; The Ohio State University Julie A. Swain MD U.S. Food and Drug Administration Robert M. Sweet MD University of Minnesota Kirby G. Vosburgh PhD Brigham & Women’s Hospital; Harvard Medical School Dave Warner MD PhD MindTel LLC; Institute for Interventional Informatics Suzanne J. Weghorst MA MS University of Washington Brenda K. Wiederhold PhD MBA BCIA Virtual Reality Medical Institute Mark Wiederhold MD PhD Virtual Reality Medical Center Ozlem Yardimci PhD Baxter Healthcare Corporation

xi

Contents Preface James D. Westwood Conference Organization Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy Kamyar Abhari, Sandrine de Ribaupierre, Terry Peters and Roy Eagleson Sleep Dysfunctions Influence Decision Making in Undemented Parkinson’s Disease Patients: A Study in a Virtual Supermarket Giovanni Albani, Simona Raspelli, Laura Carelli, Lorenzo Priano, Riccardo Pignatti, Francesca Morganti, Andrea Gaggioli, Patrice L. Weiss, Rachel Kizony, Noomi Katz, Alessandro Mauro and Giuseppe Riva Visual Tracking of Laparoscopic Instruments in Standard Training Environments Brian F. Allen, Florian Kasper, Gabriele Nataneli, Erik Dutson and Petros Faloutsos On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy Joseph B. Anstey, Erin J. Smith, Brian Rasquinha, John F. Rudan and Randy E. Ellis Classification of Pulmonary System Diseases Patterns Using Flow-Volume Curve Hossein Arabalibeik, Samaneh Jafari and Khosro Agin Cost-Efficient Suturing Simulation with Pre-Computed Models Venkata Sreekanth Arikatla, Ganesh Sankaranarayanan and Suvranu De Anesthesia Residents’ Preference for Learning Interscalene Brachial Plexus Block (ISBPB): Traditional Winnie’s Technique vs. Ultrasound-Guided Technique Imad T. Awad, Colin Sinclair, Ewen W. Chen, Colin J.L. McCartney, Jeffrey J.H. Cheung and Adam Dubrowski Fuzzy Control of a Hand Rehabilitation Robot to Optimize the Exercise Speed in Passive Working Mode Mina Arab Baniasad, Mohammad Akbar, Aria Alasty and Farzam Farahmand Engaging Media for Mental Health Applications: The EMMA Project R. Baños, C. Botella, S. Quero, A. García-Palacios and M. Alcañiz NeuroSim – The Prototype of a Neurosurgical Training Simulator Florian Beier, Stephan Diederich, Kirsten Schmieder and Reinhard Männer Low-Cost, Take-Home, Beating Heart Simulator for Health-Care Education Devin R. Berg, Andrew Carlson, William K. Durfee, Robert M. Sweet and Troy Reihsen An Adaptive Signal-Processing Approach to Online Adaptive Tutoring Bryan Bergeron and Andrew Cline Comparison of a Disposable Bougie Versus a Newly Designed Malleable Bougie in the Intubation of a Difficult Manikin Airway Ben H. Boedeker, Mary Bernhagen, David J. Miller and W. Bosseau Murray Improving Fiberoptic Intubation with a Novel Tongue Retraction Device Ben H. Boedeker, Mary Bernhagen, David J. Miller, Thomas A. Nicholas IV, Andrew Linnaus and W.B. Murray

v vii 1

8

11

18

25 31

36

39

44 51 57

60

65 68

xii

Combined Intubation Training (Simulated and Human) for 4th Year Medical Students: The Center for Advanced Technology and Telemedicine Airway Training Program Ben H. Boedeker, Mary Bernhagen, Thomas A. Nicholas IV and W. Bosseau Murray Battlefield Tracheal Intubation Training Using Virtual Simulation: A Multi Center Operational Assessment of Video Laryngoscope Technology Ben H. Boedeker, Kirsten A. Boedeker, Mary A. Bernhagen, David J. Miller and Timothy Lacy Intubation Success Rates and Perceived User Satisfaction Using the Video Laryngoscope to Train Deploying Far Forward Combat Medical Personnel Ben H. Boedeker, Mary A. Barak-Bernhagen, Kirsten A. Boedeker and W. Bosseau Murray Field Use of the STORZ C-MAC™ Video Laryngoscope in Intubation Training with the Nebraska National Air Guard Ben H. Boedeker, Mary A. Bernhagen, David J. Miller, Nikola Miljkovic, Gail M. Kuper and W. Bosseau Murray The Combined Use of Skype™ and the STORZ CMAC™ Video Laryngoscope in Field Intubation Training with the Nebraska National Air Guard Ben H. Boedeker, Mary Bernhagen, David J. Miller, Nikola Miljkovic, Gail M. Kuper and W. Bosseau Murray Online Predictive Tools for Intervention in Mental Illness: The OPTIMI Project Cristina Botella, Inés Moragrega, R. Baños and Azucena García-Palacios An Integrated Surgical Communication Network – SurgON Richard D. Bucholz, Keith A. Laycock, Leslie L. McDurmont and William R. MacNeil Web-Accessible Interactive Software of 3D Anatomy Representing Pathophysiological Conditions to Enhance the Patient-Consent Process for Procedures D. Burke, X. Zhou, V. Rotty, V. Konchada, Y. Shen, B. Konety and R. Sweet Fast Adaptation of Pre-Operative Patient Specific Models to Real-Time Intra-Operative Volumetric Data Streams Bruce M. Cameron, Maryam E. Rettmann, David R. Holmes III and Richard A. Robb Realistic Visualization of Living Brain Tissue Llyr ap Cenydd, Annette Walter, Nigel W. John, Marina Bloj and Nicholas Phillips A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy Sonny Chan, Peter Li, Dong Hoon Lee, J. Kenneth Salisbury and Nikolas H. Blevins Acquisition of Technical Skills in Ultrasound-Guided Regional Anesthesia Using a High-Fidelity Simulator Jeffrey J.H. Cheung, Ewen W. Chen, Yaseen Al-Allaq, Nasim Nikravan, Colin J.L. McCartney, Adam Dubrowski and Imad T. Awad MeRiTS: Simulation-Based Training for Healthcare Professionals David Chodos, Eleni Stroulia and Sharla King A Framework for Treatment of Autism Using Affective Computing Seong Youb Chung and Hyun Joong Yoon

71

74

77

80

83

86 93

96

99

105

112

119

125 132

xiii

Modification of Commercial Force Feedback Hardware for Needle Insertion Simulation Timothy R. Coles, Nigel W. John, Giuseppe Sofia, Derek A. Gould and Darwin G. Caldwell Visualization of Pelvic Floor Reflex and Voluntary Contractions Christos E. Constantinou, Daniel Korenblum and Bertha Chen Mixed Virtual Reality Simulation – Taking Endoscopic Simulation One Step Further O. Courteille, L. Felländer-Tsai, L. Hedman, A. Kjellin, L. Enochsson, G. Lindgren and U. Fors A Serious Game for Off-Pump Coronary Artery Bypass Surgery Procedure Training Brent Cowan, Hamed Sabri, Bill Kapralos, Fuad Moussa, Sayra Cristancho and Adam Dubrowski Progressive Simulation-Based Program for Training Cardiac Surgery-Related Skills Sayra Cristancho, Fuad Moussa, Alex Monclou, Camilo Moncayo, Claudia Rueda and Adam Dubrowski MSMS Software for VR Simulations of Neural Prostheses and Patient Training and Rehabilitation Rahman Davoodi and Gerald E. Loeb Virtual Reality System in Conjunction with Neurorobotics and Neuroprosthetics for Rehabilitation of Motor Disorders Alessandro De Mauro, Eduardo Carrasco, David Oyarzun, Aitor Ardanza, Anselmo Frizera Neto, Diego Torricelli, José Luis Pons, Angel Gil and Julian Florez Modeling the Thermal Effect of the Bipolar Electrocautery for Neurosurgery Simulation Sébastien Delorme, Anne Cabral, Fábio Ayres and Di Jiang CliniSpace™: A Multiperson 3D Online Immersive Training Environment Accessible Through a Browser Parvati Dev, W. LeRoy Heinrichs and Patricia Youngblood Medical Education Through Virtual Worlds: The HLTHSIM Project Roy Eagleson, Sandrine de Ribaupierre, Sharla King and Eleni Stroulia Ubiquitous Health in Practice: The Interreality Paradigm Andrea Gaggioli, Simona Raspelli, Alessandra Grassi, Federica Pallavicini, Pietro Cipresso, Brenda K. Wiederhold and Giuseppe Riva Bench Model Surgical Skill Training Improves Novice Ability to Multitask: A Randomized Controlled Study Lawrence Grierson, Megan Melnyk, Nathan Jowlett, David Backstein and Adam Dubrowski A Design of Hardware Haptic Interface for Gastrointestinal Endoscopy Simulation Yunjin Gu and Doo Yong Lee Open Surgery Simulation of Inguinal Hernia Repair Niels Hald, Sudip K. Sarker, Paul Ziprin, Pierre-Frederic Villard and Fernando Bello SML: SoFMIS Meta Language for Surgical Simulation Tansel Halic and Suvranu De

135

138

144

147

150

156

163

166

173 180 185

192

199 202

209

xiv

A Software Framework for Multimodal Interactive Simulations (SoFMIS) Tansel Halic, Sreekanth A. Venkata, Ganesh Sankaranarayanan, Zhonghua Lu, Woojin Ahn and Suvranu De Simulation of Vaginal Wall Biomechanical Properties from Pelvic Floor Closure Forces Map Shin Hasegawa, Yuki Yoshida, Daming Wei, Sadao Omata and Christos E. Constantinou A Generalized Haptic Feedback Approach for Arbitrarily Shaped Objects Rui Hu, Kenneth E. Barner and Karl V. Steiner Piezoelectric Driven Non-Toxic Injector for Automated Cell Manipulation H.B. Huang, Hao Su, H.Y. Chen and J.K. Mills Virtual Arthroscopy Trainer for Minimally Invasive Surgery Vassilios Hurmusiadis, Kawal Rhode, Tobias Schaeffter and Kevin Sherman Design for Functional Occlusal Surface of CAD/CAM Crown Using VR Articulator Tomoko Ikawa, Takumi Ogawa, Yuko Shigeta, Shintaro Kasama, Rio Hirabayashi, Shunji Fukushima, Asaki Hattori and Naoki Suzuki Biopsym: A Learning Environment for Trans-Rectal Ultrasound Guided Prostate Biopsies Thomas Janssoone, Grégoire Chevreau, Lucile Vadcard, Pierre Mozer and Jocelyne Troccaz Comparison of Reaching Kinematics During Mirror and Parallel Robot Assisted Movements Zahra Kadivar, Cynthia Sung, Zachary Thompson, Marcia O’Malley, Michael Liebschner and Zhigang Deng Serious Games in the Classroom: Gauging Student Perceptions Bill Kapralos, Sayra Cristancho, Mark Porte, David Backstein, Alex Monclou and Adam Dubrowski Influence of Metal Artifacts on the Creation of Individual 3D Cranio-Mandibular Models Shintaro Kasama, Takumi Ogawa, Tomoko Ikawa, Yuko Shigeta, Shinya Hirai, Shunji Fukushima, Asaki Hattori and Naoki Suzuki Web-Based Stereoscopic Visualization for the Global Anatomy Classroom Mathias Kaspar, Fred Dech, Nigel M. Parsad and Jonathan C. Silverstein Expanding the Use of Simulators as Assessment Tools: The New Pop Quiz Abby R. Kaye, Lawrence H. Salud, Zachary B. Domont, Katherine Blossfield Iannitelli and Carla M. Pugh Validation of Robotic Surgery Simulator (RoSS) Thenkurussi Kesavadas, Andrew Stegemann, Gughan Sathyaseelan, Ashirwad Chowriappa, Govindarajan Srimathveeravalli, Stéfanie Seixas-Mikelus, Rameella Chandrasekhar, Gregory Wilding and Khurshid Guru Practical Methods for Designing Medical Training Simulators Thomas Knott, Sebastian Ullrich and Torsten Kuhlen The Minnesota Pelvic Trainer: A Hybrid VR/Physical Pelvis for Providing Virtual Mentorship Vamsi Konchada, Yunhe Shen, Dan Burke, Omer B. Argun, Anthony Weinhaus, Arthur G. Erdman and Robert M. Sweet

213

218

224 231 236

239

242

247

254

261

264 271

274

277

280

xv

Registration Stability of Physical Templates in Hip Surgery Manuela Kunz, John F. Rudan, Gavin C.A. Wood and Randy E. Ellis Real-Time 3D Avatars for Tele-Rehabilitation in Virtual Reality Gregorij Kurillo, Tomaz Koritnik, Tadej Bajd and Ruzena Bajcsy Fundamentals of Gas Phase Plasmas for Treatment of Human Tissue Mark J. Kushner and Natalia Yu. Babaeva VR-Based Training and Assessment in Ultrasound-Guided Regional Anesthesia: From Error Analysis to System Design Erik Lövquist, Owen O’Sullivan, Donnchadh Oh’Ainle, Graham Baitson, George Shorten and Nick Avis Real-Time Electrocautery Simulation for Laparoscopic Surgical Environments Zhonghua Lu, Venkata Sreekanth Arikatla, Dingfang Chen and Suvranu De Guidewire and Catheter Behavioural Simulation Vincent Luboz, Jianhua Zhai, Tolu Odetoyinbo, Peter Littler, Derek Gould, Thien How and Fernando Bello Design and Implementation of a Visual and Haptic Simulator in a Platform for a TEL System in Percutaneuos Orthopedic Surgery Vanda Luengo, Aurelie Larcher and Jérôme Tonetti Computational Modeling of Human Head Electromagnetics for Source Localization of Milliscale Brain Dynamics Allen D. Malony, Adnan Salman, Sergei Turovets, Don Tucker, Vasily Volkov, Kai Li, Jung Eun Song, Scott Biersdorff, Colin Davey, Chris Hoge and David Hammond Simulation and Modeling of Metamorphopsia with a Deformable Amsler Grid Anabel Martin-Gonzalez, Ines Lanzl, Ramin Khoramnia and Nassir Navab Development of a Customizable Software Application for Medical Imaging Analysis and Visualization Marisol Martinez-Escobar, Catherine Peloquin, Bethany Juhnke, Joanna Peddicord, Sonia Jose, Christian Noon, Jung Leng Foo and Eliot Winer Pneumoperitoneum Technique Simulation in Laparoscopic Surgery on Lamb Liver Samples and 3D Reconstruction F. Martínez-Martínez, M.J. Rupérez, M.A. Lago, F. López-Mir, C. Monserrat and M. Alcañíz Technology Transfer at the University of Nebraska Medical Center Kulia Matsuo, Henry J. Runge, David J. Miller, Mary A. Barak-Bernhagen and Ben H. Boedeker CvhSlicer: An Interactive Cross-Sectional Anatomy Navigation System Based on High-Resolution Chinese Visible Human Data Q. Meng, Y.P. Chui, J. Qin, W.H. Kwok, M. Karmakar and P.A. Heng Generation of Connectivity-Preserving Surface Models of Multiple Sclerosis Lesions Oscar Meruvia-Pastor, Mei Xiao, Jung Soh and Christoph W. Sensen A Comparison of Videolaryngoscopic Technologies David J. Miller, Nikola Miljkovic, Chad Chiesa, Nathan Schulte, John B. Callahan Jr. and Ben H. Boedeker Telemedicine Using Free Voice over Internet Protocol (VoIP) Technology David J. Miller, Nikola Miljkovic, Chad Chiesa, John B. Callahan Jr., Brad Webb and Ben H. Boedeker

283 290 297

304

311 317

324

329

336

343

348

351

354

359 366

369

xvi

iMedic: A Two-Handed Immersive Medical Environment for Distributed Interactive Consultation Paul Mlyniec, Jason Jerald, Arun Yoganandan, F. Jacob Seagull, Fernando Toledo and Udo Schultheis Patient Specific Surgical Simulator for the Evaluation of the Movability of Bimanual Robotic Arms Andrea Moglia, Giuseppe Turini, Vincenzo Ferrari, Mauro Ferrari and Franco Mosca CyberMedVPS: Visual Programming for Development of Simulators Aline M. Morais and Liliane S. Machado A Bloodstream Simulation Based on Particle Method Masashi Nakagawa, Nobuhiko Mukai, Kiyomi Niki and Shuichiro Takanashi Laser Induced Shockwaves on Flexible Polymers for Treatment of Bacterial Biofilms Artemio Navarro, Zachary D. Taylor, David Beenhouwer, David A. Haake, Vijay Gupta, Warren S. Grundfest Virtual Reality Haptic Human Dissection Caroline Needham, Caroline Wilkinson and Roger Soames The Tool Positioning Tutor: A Target-Pose Tracking and Display System for Learning Correct Placement of a Medical Device Douglas A. Nelson and Joseph T. Samosky A Cost Effective Simulator for Education of Ultrasound Image Interpretation and Probe Manipulation S.A. Nicolau, A. Vemuri, H.S. Wu, M.H. Huang, Y. Ho, A. Charnoz, A. Hostettler, C. Forest, L. Soler and J. Marescaux A Portable Palpation Training Platform with Virtual Human Patient Tyler Niles, D. Scott Lind and Kyle Johnsen A Development of Surgical Simulator for Training of Operative Skills Using Patient-Specific Data Masato Ogata, Manabu Nagasaka, Toru Inuiya, Kazuhide Makiyama and Yoshinobu Kubota Virtual Reality Image Applications for Treatment Planning in Prosthodontic Dentistry Takumi Ogawa, Tomoko Ikawa, Yuko Shigeta, Shintaro Kasama, Eriko Ando, Shunji Fukushima, Asaki Hattori and Naoki Suzuki The Initiation of a Preoperative and Postoperative Telemedicine Urology Clinic Eugene S. Park, Ben H. Boedeker, Jennifer L. Hemstreet and George P. Hemstreet Modeling Surgical Skill Learning with Cognitive Simulation Shi-Hyun Park, Irene H. Suh, Jung-hung Chien, Jaehyon Paik, Frank E. Ritter, Dmitry Oleynikov, Ka-Chun Siu Virtual Reality Stroop Task for Neurocognitive Assessment Thomas D. Parsons, Christopher G. Courtney, Brian Arizmendi and Michael Dawson Implementation of Virtual Online Patient Simulation V. Patel, R. Aggarwal, D. Taylor and A. Darzi Patient-Specific Cases for an Ultrasound Training Simulator Kresimir Petrinec, Eric Savitsky and Cheryl Hein

372

379

386 389

394

397

400

403

408

415

422

425

428

433

440 447

xvii

Stereo Image-Based Arm Tracking for In Vivo Surgical Robotics Eric Psota, Kyle Strabala, Jason Dumpert, Lance C. Pérez, Shane Farritor and Dmitry Oleynikov A Simulation Framework for Wound Closure by Suture for the Endo Stitch Suturing Instrument Sukitti Punak and Sergei Kurenov Simplified Cosserat Rod for Interactive Suture Modeling Sukitti Punak and Sergei Kurenov A Design for Simulating and Validating the Nuss Procedure for the Minimally Invasive Correction of Pectus Excavatum Krzysztof J. Rechowicz, Robert Kelly, Michael Goretsky, Frazier W. Frantz, Stephen B. Knisley, Donald Nuss and Frederic D. McKenzie AISLE: An Automatic Volumetric Segmentation Method for the Study of Lung Allometry Hongliang Ren and Peter Kazanzides Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery Hongliang Ren, Denis Rank, Martin Merdes, Jan Stallkamp and Peter Kazanzides Visualization of Probabilistic Fiber Tracts in Virtual Reality Tobias Rick, Anette von Kapri, Svenja Caspers, Katrin Amunts, Karl Zilles and Torsten Kuhlen NeuroVR 2 – A Free Virtual Reality Platform for the Assessment and Treatment in Behavioral Health Care Giuseppe Riva, Andrea Gaggioli, Alessandra Grassi, Simona Raspelli, Pietro Cipresso, Federica Pallavicini, Cinzia Vigna, Andrea Gagliati, Stefano Gasco and Giuseppe Donvito Personal Health Systems for Mental Health: The European Projects Giuseppe Riva, Rosa Banos, Cristina Botella, Andrea Gaggioli and Brenda K. Wiederhold An Intelligent Virtual Human System for Providing Healthcare Information and Support Albert A. Rizzo, Belinda Lange, John G. Buckwalter, Eric Forbell, Julia Kim, Kenji Sagae, Josh Williams, Barbara O. Rothbaum, JoAnn Difede, Greg Reger, Thomas Parsons and Patrick Kenny Virtual Reality Applications for Addressing the Needs of Those Aging with Disability Albert Rizzo, Phil Requejo, Carolee J. Winstein, Belinda Lange, Gisele Ragusa, Alma Merians, James Patton, Pat Banerjee and Mindy Aisen The Validation of an Instrumented Simulator for the Assessment of Performance and Outcome of Knot Tying Skill: A Pilot Study David Rojas, Sayra Cristancho, Claudia Rueda, Lawrence Grierson, Alex Monclou and Adam Dubrowski Manual Accuracy in Comparison with a Miniature Master Slave Device – Preclinical Evaluation for Ear Surgery A. Runge, M. Hofer, E. Dittrich, T. Neumuth, R. Haase, M. Strauss, A. Dietz, T. Lüth and G. Strauss

454

461 466

473

476 479

486

493

496

503

510

517

524

xviii

Are Commercially Available Simulators Durable Enough for Classroom Use? Jonathan C. Salud, Katherine Blossfield Iannitelli, Lawrence H. Salud and Carla M. Pugh Toward a Simulation and Assessment Method for the Practice of Camera-Guided Rigid Bronchoscopy Lawrence H. Salud, Alec R. Peniche, Jonathan C. Salud, Alberto L. de Hoyos and Carla M. Pugh Use of Sensor Technology to Explore the Science of Touch Lawrence H. Salud and Carla M. Pugh Real-Time “X-Ray Vision” for Healthcare Simulation: An Interactive Projective Overlay System to Enhance Intubation Training and Other Procedural Training Joseph T. Samosky, Emma Baillargeon, Russell Bregman, Andrew Brown, Amy Chaya, Leah Enders, Douglas A. Nelson, Evan Robinson, Alison L. Sukits and Robert A. Weaver Toward a Comprehensive Hybrid Physical-Virtual Reality Simulator of Peripheral Anesthesia with Ultrasound and Neurostimulator Guidance Joseph T. Samosky, Pete Allen, Steve Boronyak, Barton Branstetter, Steven Hein, Mark Juhas, Douglas A. Nelson, Steven Orebaugh, Rohan Pinto, Adam Smelko, Mitch Thompson and Robert A. Weaver A Fixed Point Proximity Method for Extended Contact Manipulation of Deformable Bodies with Pivoted Tools in Multimodal Virtual Environments Ganesh Sankaranarayanan, Zhonghua Lu and Suvranu De Collision and Containment Detection Between Biomechanically Based Eye Muscle Volumes Graciela Santana Sosa and Thomas Kaltofen Visualization of 3D Volumetric Lung Dynamics for Real-Time External Beam Lung Radiotherapy Anand P. Santhanam, Harini Neelakkantan, Yugang Min, Nicolene Papp, Akash Bhargava, Kevin Erhart, Xiang Long, Rebecca Mitchell, Eduardo Divo, Alain Kassab, Olusegun Ilegbusi, Bari H. Ruddy, Jannick P. Rolland, Sanford L. Meeks and Patrick A. Kupelian Laser Surgery Simulation Platform: Toward Full-Procedure Training and Rehearsal for Benign Prostatic Hyperplasia (BPH) Therapy Yunhe Shen, Vamsi Konchada, Nan Zhang, Saurabh Jain, Xiangmin Zhou, Daniel Burke, Carson Wong, Culley Carson, Claus Roehrborn and Robert Sweet 3D Tracking of Surgical Instruments Using a Single Camera for Laparoscopic Surgery Simulation Sangkyun Shin, Youngjun Kim, Hyunsoo Kwak, Deukhee Lee and Sehyung Park Perceptual Metrics: Towards Better Methods for Assessing Realism in Laparoscopic Simulators Ravikiran B. Singapogu, Christopher C. Pagano, Timothy C. Burg and Karen J.K.L. Burg Role of Haptic Feedback in a Basic Laparoscopic Task Requiring Hand-Eye Coordination Ravikiran B. Singapogu, Christopher C. Pagano, Timothy C. Burg, Karen J.K.L. Burg and Varun V. Prabhu

531

535

542

549

552

555

560

567

574

581

588

591

xix

A Model for Flexible Tools Used in Minimally Invasive Medical Virtual Environments Francisco Soler, M. Victoria Luzon, Serban R. Pop, Chris J. Hughes, Nigel W. John and Juan Carlos Torres Segmentation of 3D Vasculatures for Interventional Radiology Simulation Yi Song, Vincent Luboz, Nizar Din, Daniel King, Derek Gould, Fernando Bello and Andy Bulpitt EEG-Based “Serious” Games and Monitoring Tools for Pain Management Olga Sourina, Qiang Wang and Minh Khoa Nguyen A New Part Task Trainer for Teaching and Learning Confirmation of Endotracheal Intubation Cyle Sprick, Harry Owen, Cindy Hein and Brigid Brown Mobile Three Dimensional Gaze Tracking Josef Stoll, Stefan Kohlbecher, Svenja Marx, Erich Schneider and Wolfgang Einhäuser High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions Hao Su, Alex Camilo, Gregory A. Cole, Nobuhiko Hata, Clare M. Tempany and Gregory S. Fischer Electromyographic Correlates of Learning During Robotic Surgical Training in Virtual Reality Irene H. Suh, Mukul Mukherjee, Ryan Schrack, Shi-Hyun Park, Jung-hung Chien, Dmitry Oleynikov and Ka-Chun Siu Web-Based Interactive Volume Rendering Stefan Suwelack, Sebastian Maier, Roland Unterhinninghofen and Rüdiger Dillmann A Method of Synchronization for Haptic Collaborative Virtual Environments in Multipoint and Multi-Level Computer Performance Systems Kazuyoshi Tagawa, Tatsuro Bito and Hiromi T. Tanaka A Hybrid Dynamic Deformation Model for Surgery Simulation Kazuyoshi Tagawa and Hiromi T. Tanaka Single and Multi-User Virtual Patient Design in the Virtual World D. Taylor, V. Patel, D. Cohen, R. Aggarwal, K. Kerr, N. Sevdalis, N. Batrick and A. Darzi Terahertz Imaging of Biological Tissues Priyamvada Tewari, Zachary D. Taylor, David Bennett, Rahul S. Singh, Martin O. Culjat, Colin P. Kealey, Jean Pierre Hubschman, Shane White, Alistair Cochran, Elliott R. Brown and Warren S. Grundfest Quantifying Surgeons’ Vigilance During Laparoscopic Operations Using Eyegaze Tracking Geoffrey Tien, Bin Zheng and M. Stella Atkins Modeling of Interaction Between a Three-Fingered Surgical Grasper and Human Spleen Mojdeh Tirehdast, Alireza Mirbagheri, Mohsen Asghari and Farzam Farahmand Quantizing the Void: Extending Web3D for Space-Filling Haptic Meshes Sebastian Ullrich, Torsten Kuhlen, Nicholas F. Polys, Daniel Evestedt, Michael Aratow and Nigel W. John Dissecting in Silico: Towards a Taxonomy for Medical Simulators Sebastian Ullrich, Thomas Knott and Torsten Kuhlen

594

599

606

611 616

623

630

635

638 645 650

653

658

663

670

677

xx

Computed Tomography as Ground Truth for Stereo Vision Measurements of Skin Amy M. Vanberlo, Aaron R. Campbell and Randy E. Ellis Towards the Visualization of Spiking Neurons in Virtual Reality Anette von Kapri, Tobias Rick, Tobias C. Potjans, Markus Diesmann and Torsten Kuhlen The Use of Virtual Training to Support Insertion of Advanced Technology at Remote Military Locations Madison I. Walker, Robert B. Walker, Jeffrey S. Morgan, Mary Bernhagen, Nicholas Markin and Ben H. Boedeker Three Dimensional Projection Environment for Molecular Design and Surgical Simulation Eric Wickstrom, Chang-Po Chen, Devakumar Devadhas, Matthew Wampole, Yuan-Yuan Jin, Jeffrey M. Sanders, John C. Kairys, Martha L. Ankeny, Rui Hu, Kenneth E. Barner, Karl V. Steiner and Mathew L. Thakur Reality Graded Exposure Therapy with Physiological Monitoring for the Treatment of Combat Related Post Traumatic Stress Disorder: A Pilot Study Dennis Patrick Wood, Jennifer Webb-Murphy, Robert N. McLay, Brenda K. Wiederhold, James L. Spira, Scott Johnston, Robert L. Koffman, Mark D. Wiederhold and Jeff Pyne Applications of Tactile Feedback in Medicine Christopher Wottawa, Richard Fan, James W. Bisley, Erik P. Dutson, Martin O. Culjat and Warren S. Grundfest Needle Insertion Simulation by Arbitrary Lagrangian-Eulerian Method Satoshi Yamaguchi, Koji Satake, Shigehiro Morikawa, Yoshiaki Shirai and Hiromi T. Tanaka Clinical Performance of Dental Fiberscope Image Guided System for Endodontic Treatment Yasushi Yamazaki, Takumi Ogawa, Yuko Shigeta, Tomoko Ikawa, Shintaro Kasama, Asaki Hattori, Naoki Suzuki, Takatsugu Yamamoto, Toshiko Ozawa and Takashi Arai A Novel Virtual Reality Environment for Preoperative Planning and Simulation of Image Guided Intracardiac Surgeries with Robotic Manipulators Erol Yeniaras, Zhigang Deng, Mushabbar A. Syed, Mark G. Davies and Nikolaos V. Tsekos Enabling Surgeons to Create Simulation-Based Teaching Modules Young In Yeo, Saleh Dindar, George Sarosi and Jörg Peters Using a Virtual Integration Environment in Treating Phantom Limb Pain Michael J. Zeher, Robert S. Armiger, James M. Burck, Courtney Moran, Janid Blanco Kiely, Sharon R. Weeks, Jack W. Tsao, Paul F. Pasquina, R. Davoodi and G. Loeb Validation of a Virtual Preoperative Evaluation Clinic: A Pilot Study Corey V. Zetterman, Bobbie J. Sweitzer, Brad Webb, Mary A. Barak-Bernhagen and Ben H. Boedeker Multifunction Robotic Platform for Natural Orifice Surgery Xiaoli Zhang, Wei Jian Chin, Chi Min Seow, Akiko Nakamura, Michael Head, Shane Farritor, Dmitry Oleynikov and Carl Nelson Maintaining Forward View of the Surgical Site for Best Endoscopic Practice Bin Zheng, Maria A. Cassera, Lee L. Swanström, Adam Meneghetti, Neely O.N. Panton and Karim A. Qayumi

680 685

688

691

696

703

710

713

716

723 730

737

740

743

xxi

Phenomenological Model of Laser-Tissue Interaction with Application to Benign Prostatic Hyperplasia (BPH) Simulation Xiangmin Zhou, Nan Zhang, Yunhe Shen, Dan Burke, Vamsi Konchada and Robert Sweet Subject Index Author Index

749

757 763

This page intentionally left blank

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-1

1

Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy Kamyar ABHARI a,b , Sandrine de RIBAUPIERRE c , Terry PETERS a,b and Roy EAGLESON a,d a Imaging Research Laboratories, Robarts Research Institute b Biomedical Engineering Program, The University of Western Ontario Ontario c Department of Clinical Neurological Sciences, The University of Western Ontario, London Health Sciences Centre d Department of Electrical and Computer Engineering, Faculty of Engineering, The University of Western Ontario, London, Ontario, Canada Abstract. Endoscopic third ventriculostomy is a minimally invasive technique to treat hydrocephalus, which is a condition in which the patient is retaining excessive amount of cerebrospinal fluid in the head. While this surgical procedure is fairly routine, it carries some risks, mainly associated with the lack of depth perception, since monocular endoscopes provide only 2D views. We studied the advantages given by a 3D stereoendoscope over a 2D monocular endoscope, first by assessing the variability of stereoacuity in each subject, then in analyzing their overall correct response rate in differentiating between heights of two different images with 2D and 3D vision. Keywords. Hydrocephalus, Endoscopic Third Ventriculostomy, Stereo-endoscopy

Introduction Hydrocephalus is an abnormal accumulation of cerebreospinal fluid (CSF) within the brain, and is one the most common source of developmental disability among children as it affects one in every 500-1000 live births [1]. Obstructive hydrocephalus can be treated either with a shunt, draining fluid away from the head, or with an Endoscopic Third Ventriculostomy (ETV) which involves making a hole in the ventricular system to bypass the obstruction. In the last decade, ETV gradually has become the procedure of choice for obstructive hydrocephalus. The technique involves making a small perforation on the floor of the third ventricle to allow extra CSF to drain into the interpeduncular cistern. The ETV operation involves using an endoscope to navigate within the ventriclar system. There are different types of endoscope used but they all produce 2D images. Although ETV is an effective approach, it is not without risk to the patient, and the speed and accuracy of the intervention is dependent on visualization of the floor of the third ventricle and basilar artery. The basilar artery is located a few millimeters behind the clivus. Accurate localization of the basilar artery and its two most important branches

2

K. Abhari et al. / Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy

(posterior cerebral arteries) is essential to avoid injuring them during the procedure. Injuring the basilar artery might lead to a major stroke or be lethal. Although the floor of the third ventricle can be transparent, in some cases it is thickened by an old infection, hemorrhage, or tumoral cells, and therefore it is impossible to see through and visualize all the structures. In these cases, the task of localization of the basilar artery is extremely difficult and in some cases impossible. Unfortunately, there are no reliable visible textures on the membrane of the third ventricle that can be helpful to locate and avoid basilar artery. However, there are different structures which deform the floor of the third ventricle and provide some relief. In hydrocephalus patients, the pressure of CSF inside the ventricular system gradually reshapes the structure of the ventricles. As a result, the surface of third ventricle is also pushed down. With the pressure, the floor of the third ventricle will then adopt the shape of the underlying structures. This mechanism creates a bump on the floor of the third ventricle above the basilar artery. We believe that this curvature can be used in order to locate and avoid the basilar artery if it can be detected within the stereoendoscopic view. Without providing 3D information, however, surgeons are not able to observe this landmark and differentiate it from the other parts of the third ventricle. These structures may not be visualized with monocular endoscopes where observers suffer from the lack of depth perception. Stereoscopic endoscopes, on the other hand, can provide necessary depth information required to properly locate and visualize these structures. Two clinical centres have evaluated the use of stereoscopic neuroendoscopes in a few patients [2], but the technique has not been fully investigated, and the difference between the 2D and 3D never been studied objectively. The objective of this study is to establish a methodology to determine whether 3Dendoscopy can improve accuracy of the ETV in cases where impaired depth perception can be problematic or even catastrophic during surgery. Using a stereo-endoscopic camera, the physical structure of the brain can be observed in 3D which offers the observer appropriate depth perception of the brain’s anatomy. In this paper, we evaluate the feasibility of this approach using experiments which involve comparing the established 2D method and the proposed 3D technique in terms of its sensitivity to depth discrimination. Our hypothesis is that this method will significantly improve the localization and avoidance of basilar artery with the goal of having safer and faster ETV interventions.

Materials and Methods 1. Materials 1.1. Virtual Environment Stereoacuity, similar to visual acuity, is a measure of the perceptual capacity to detect small differences in depth using stereo vision. Although there are commercially available tests for stereoacuity such as Random Dot Stereograms [8], they usually vary the point positions in depth, and not the size and shape of the perceived stimulus. In clinical settings, it is important to appreciate not only the relative distance between structures, but also the curvature of the surface. In addition, there are some monocular cues that can be present at the area of operation, such as small blood vessels, different opacity of the membrane, etc. Building our own stereoacuity test, allowed us to control these


3

Figure 1. Model of the third ventricle

factors, and therefore we were able to correlate our results with results obtained in a clinical setting. It was therefore necessary to design a specialized visualization tool. The system described here extends a 3D biomedical visualization platform developed within our VASST lab (AtamaiViewer, (c) Atamai Inc., London, ON). The system makes use of a FakeSpace TM display with a high-resolution stereoscopic projection system that offers a large slanted table-top display for use in immersive visualization experiments. To begin with, a set of scale models were fabricated based on the real anatomy and workspace geometry of the third ventricle (Figure 1). First, the models were specified using a computer-aided design program. A texture map, acquired endoscopically, of the floor of the third ventricle is mapped onto the surface of our virtual models. The selected texture has no distinguishable monocular cues. Each model may include a bump on the surface similar to what the basilar artery induces in hydrocephalus patients. In live ETV surgery scenarios, this bump may have a range of prominence. The height of the bump on our models ranges from zero (no bump) to 3mm with step value of 0.1mm (i.e. 31 models in total). 1.2. VisionSense Camera The VisionSense stereoendoscope camera (VisionSense Ltd., Isreal) is a compact (3.8mm – 4.9mm) FDA-approved device, which makes it a good candidate for neurosurgical procedures. Previously designed stereoendoscopes, were not suitable for minimally invasive neurosurgeries as they are significantly larger than commonly used endoscopes. Several studies have demonstrated the practicality of the VisionSense camera and its overall advantage over the monocular endoscopes [3] [4] [5]. 1.3. Physical Environment: Preliminary Prototype Using stereolithographic rapid-prototyping technology, seven different phantoms (ranging from 0mm to 3mm with 0.5mm step value) were created based on our computergenerated models as seen in Figure 1. A number of experiments were conducted using these models in order to determine some of key variables required for our final prototype (Refer to section 2.3. for details). 1.4. Physical Environment: Final Prototype In order to collect preliminary data, two rapid-prototyped phantoms were placed under the VisionSense camera in each trial. This set-up brought us some undesirable effects including a gradient around the edges and a glare due to the reflection of the endoscope’s light. Although these effects were not pronounced, they could potentially be used as

4


monocular cues. For this reason, new series of phantoms were made out of two parts silicone to provide smooth surface with the accuracy of 0.1mm in any preferable colour.

2. Method In this section, we outline a methodology for the design and evaluation of a stereoendoscopic method to facilitate ETV, in comparison with the current method involving monocular endoscopy. Our main goal was to examine the use of stereopsis to identify the location of the basilar artery by detecting the bump in the tissue. In order to test this, we are consequently testing the role of stereo in the task of differentiating between two different surfaces in terms of their depth. 2.1. Virtual Environment Experiment The stereoacuity test involves number of trials and a simple discrimination task. In each trial, subjects are presented with a pair of models side-by-side from the top view angle in stereo. They are asked to sit comfortably viewing the FakeSpace TM screen while wearing LCD shutter glasses. The task involves selecting the model with bigger bump by pressing the corresponding keys on the keyboard. The methodology employed is based on a standard psychophysical ’staircase’ design for establishing stereoacuity [6]. 2.2. VisionSense Experiment: Preliminary The second stage of the experiments involves using the VisionSense stereoendoscope to compare 3D and 2D visualization in terms of the accuracy of completing a task. In this stage, the experiments involved using a set of plastic phantoms and the VisionSense endoscope to make a comparison between 2D and 3D ETV interventions. Subjects’ ability to discriminate bump heights was compared in two conditions: (i) using the VisionSense camera with Stereo and Monocular cues present, and (ii) with similar views but with no stereo. Each trial involves placing two different phantoms side by side on a plexiglass holder and asking the subjects to select the one with taller bump. Using this set-up, users could observe the phantoms on the dedicated display similar to the way in which neurosurgeons do in the operating room. The task consisted of locating and selecting the target (the bump in this case) which was most prominent in 3D. The experiments are conducted once in monocular view and then later using stereo. In order to include the subjects’ stereo sensitivity profile in our analysis, the virtual stereo environment and the VisionSense stereoendoscope are required to provide the same stereo effect or disparity. To fulfill this requirement, we calculated and varied the inter-ocular distance as well as the focal point of the virtual cameras. The distance between the blocks and the lens is also kept the same for both real and virtual environments. 2.3. VisionSense Experiment: Final In this stage of our study, the task is identical to the previous one with the difference of using the silicone phantom. For any psychophysical experiment, it is necessary to deter-


5

mine the following variables: Start Point, Step Size, Stop Point, and Modification of step sizes [9]. Collected data from the previous phase of study provide required information to identify these variables: 2.3.1. Initializing the sequence of trials In order to lower the total number of trials for each subject, we produce stimulus values based on an estimate of their stereoacuity threshold. By using our preliminary data, the mean of overall correct responses reaches 85% when the height difference is approximately 1.25mm for both the VisionSense and FakeSpace. Therefore, 1.25mm was selected as the first stimulus in our series. 2.3.2. Step size In our series of stimulus displays, the step size can be defined as the minimum difference between the bump height values as we move from one trial to the next. Ideally, the proper step size value should be selected as the intensity at which subjects perform discriminations halfway between chance guessing and perfect responses [9]. Since in our preliminary experiments, subjects’ performance reached almost 75% when the step size was 0.5mm, we lowered the minimum height difference by half (0.25 mm) in order to achieve the proper step size value. 2.3.3. Where to stop Finding the stop point is challenging since it can be a compromise between a large series of stimuli for higher accuracy and a small number of trials for economy in time and therefore minimizing the effect of fatigue. To be fair and efficient, it is necessary to find the trial where the correct hit value reaches its plateau as our stop point. To fulfill this condition, the number of trials was increased to 44 from 38 as the mean value of correct responses reached 96% in our preliminary experiments. 2.3.4. Modification of step sizes As a common approach in psychophysical experiments [9], steps were set to be larger in the beginning of the series and gradually get smaller when the final threshold value has been reached.

Results The data recorded during the first and second phase (section 2.1. and 2.2. respectively) were the basis for our final experimental design (Refer to section 2.3. for details). The overall quantitative result from the series of experiments in the final phase is shown in Table 1 and illustrated in Figure 2. We choose a threshold of 90% of correct answers to analyze what height difference in the bumps subjects would be able to see. As seen in the psychometric graph, all subjects perform above the threshold (90%), for a height difference between the two bumps of 0.75mm in stereo. The same pool of subjects did not achieve the same threshold value for the height difference of less than 2.5mm. Since the stereo and mono conditions were run using the same basic heights for the bumps, a paired t-test was used to analyze the data. The result of the t-test indicates that

6


Table 1. Results: Average Correct Response Rate (CR) vs. Height Difference (HD) Mode [n] [M] [SD] [STDerr]

HD: 0mm, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2, and 2.5mm

Stereo [10] [83.8] [23.06] [7.29]

CR: 45%, 45, 65, 90, 95, 98.2, 100, 100, 100, and 100%

Mono [10] [73.6] [14.68] [4.643]

CR: 45%, 60, 60, 70, 75, 79, 85, 85, 87, and 90%

Figure 2. Correct Response Rate vs Height Difference

stereo-endoscopes, compare to the monocular ones, will significantly improve the localization and avoidance of basilar artery by increasing our ability to detect small differences in depth. (t=2.93, p=0.01, with the CI of 95%).

Discussion and Conclusion Our data show that if the basilar artery is impinging on the membrane, deforming it by at least 0.75mm, this stereo cue can allow the surgeon to avoid that area with 90% confidence. Other monocular cues may be present, and consequently can be used in addition to these cues which are purely stereoscopic. This paper advocates for the stereoendoscopic extension of the monocular stereoscopic approach, since with minimal incremental cost, this can dramatically improve the performance of locating the basilar artery. The low value of the threshold demonstrates the ability of our subjects in order to differentiate about the location of the basilar artery and the rest of the third ventricle. The results obtained using the VisionSense camera showed that subjects’ performance when making use of monocular cues and stereo cues is subject-dependent. Note that it is impossible to eliminate completely the monocular cues when surfaces are presented to an observer using stereovision. Our data indicate that subjects have the ability to make use of one cue or the other, preferentially, according to personal choice or perceptual capacity. In all cases, however, the subjects were never worse when using Stereo and Mono cues, as compared with Monocular vision alone; and in several cases, their acuity thresholds


7

were improved significantly for the Stereo and Mono presentation cases. The next phase of this series of experiments will be to determining the accuracy to which the subjects can localize a target (the place where they would make the endoscopic hole), again using the VisionSense camera, and comparing the 2D and 3D cue conditions. We will report on our data collection covering data for the stereoacuity in more subjects, and also acquiring data on proper localization of the target and differences between 2D and 3D images. We also plan to study the feasibility of overlapping the 3D stereoendoscopy with an ultrasonic doppler image of the basilar artery in order to increase the accuracy. A further step would be to be able to map both of those images into a preoperative imaging and use them in order to update the neuronavigation system in real-time. Some teams have tried to use endoscopic ultrasound to increase the accuracy of their operations [8], but were limited by the short penetration depth and the inability to scan anteriorly. Incorporating different technologies (Doppler US, neuronavigation) with stereoendoscopy, should lead to a more accurate way of localizing the target, and therefore to safer operations. In addition, our methodology can then be applied to more complicated neuroendoscopic procedures (ie multiple cysts, tumors etc). Overall, our results show that although there seem to be some inter-subject variability in the stereoacuity, stereoendoscopy facilitates neuroendoscopic performance, especially when the anatomical cues are poor. Acknowledgements The authors would like to thank J. Moore, C. Wedlake, and E. Cheng for valuable discussions and technical support. This project was supported by the Canadian Institutes for Health Research (Grant MOP 74626), the National Science and Engineering Research Council of Canada (Grants R314GA01 and A2680A02), the Ontario Research and Development Challenge Fund, the Canadian Foundation for Innovation and Ontario Innovation Trust. Graduate student funding for K. Abhari was provided by scholarships from the National Science and Engineering Research Council of Canada and by the University of Western Ontario. References [1] [2] [3] [4] [5]

[6] [7] [8] [9]

National Institute of Neurological Disorders and Stroke, http://www.ninds.nih.gov. Chen, J.C., Levy, M.L., Corber, Z., Assi, M.M., Concurrent three dimensional neuroendoscopy: initial descriptions of application to clinical practice, Minim Invasive Neurosurg., 6(4) (1999). Fraser, J.F., Allen, B., Anand, V.K., Schwartz, T.H., Three-dimensional neurostereoendoscopy: subjective and objective comparison to 2D, Neurosurgical focus, 52 (1) (2009) 25-31. Tabaee, A., Anand, V.K., Fraser, J.F., Brown, S., Singh, A., Schwartz, T.H., Three-dimensional endoscopic pituitary surgery, Neurosurgery, 65 (2009) 288-295. Roth, J., Singh, A., Nyquist, G., Fraser, J., Bernardo, A, Anand, V.K., Schwartz, T.H., ThreeDimensional and 2-Dimensional Endoscopic Exposure of Midline Cranial Base Targets Using Expanded Endonasal and Transcranial Approache, Neuro-surgery, 65(6) (2009) 1116-1130 Andrews, T., Glennerster, A., Parker, A.: Stereoacuity Thresholds in the Presence of a Reference Surface, J. of Vision Research, 41 (2001) 3051-3061 Resch, K.D., Transendoscopic ultrasound in ventricular lesions, Surgical neurology, 69(4) (2008) 375382 Julesz B., Foundations of Cyclopean Perception, The University of Chicago Press ISBN 0-226-41527-9. Cornsweet T. N., The Staircase Method in Psychophysics, The American Journal of Psychology,Â˘a75(3) (1962) 485–491

8


Sleep Dysfunctions Influence Decision Making in Undemented Parkinson’s Disease Patients: A Study in a Virtual Supermarket Giovanni ALBANI 1, Simona RASPELLI 2, Laura CARELLI 3 Lorenzo PRIANO 1, Riccardo PIGNATTI 1, Francesca MORGANTI 3 Andrea GAGGIOLI 2-4, Patrice L. WEISS 5, Rachel KIZONY 5-6, Noomi KATZ 6 Alessandro MAURO 1, Giuseppe RIVA 2-4 1

Department of Neurosciences and Neurorehabilitation, Istituto Auxologico Italiano, IRCCS, Piancavallo-Verbania, Italy 2 Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, IRCCS, Milan, Italy 3 Department of Human Sciences, University of Bergamo, Bergamo, Italy 4 Psychology Department, Catholic University of Milan, Italy 5 Department of Occupational Therapy, University of Haifa, Haifa, Israel 6 Research Institute for the Health & Medical Professions, Ono Academic College, Kiryat Ono, Israel

Abstract. In the early-middle stages of Parkinson’s disease (PD), polysomnographic studies show early alterations of the structure of the sleep, which may explain frequent symptoms reported by patients, such as daytime drowsiness, loss of attention and concentration, feeling of tiredness. The aim of this study was to verify if there is a correlation between the sleep dysfunction and decision making ability. We used a Virtual Reality version of the Multiple Errand Test (VMET), developed using the NeuroVR free software (http://www.neurovr2.org), to evaluate decision-making ability in 12 PD notdemented patients and 14 controls. Five of our not-demented 12 PD patients showed abnormalities in the polysomnographic recordings associated to significant differences in the VMET performance. Keywords: Virtual Reality, Assessment, Parkinson’s disease, NeuroVR, VMET

1. Introduction In the early-middle stages of Parkinson’s disease (PD), polysomnographic studies show early alterations of the structure of the sleep, which may explain frequent symptoms reported by patients, such as daytime drowsiness, loss of attention and concentration, feeling of tiredness. Apparently these symptoms may involve a deficit in the executive functions, so the goal of this study was to verify the existence of a correlation between the sleep dysfunction and decision making ability in PD not-demented patients.

G. Albani et al. / Sleep Dysfunctions Influence Decision Making

9

Specifically, polysomnographic data were associated with the performance obtained by the PD patients in the virtual version of a neuropsychological test, the Multiple Errand Test (MET). The MET is an assessment of executive functions in daily life originally developed by Shallice and Burgess [1] specifically for high functioning patients and adapted into the simple version and the hospital version. It consists of three tasks that abide by certain rules and is performed in a mall-like setting or shopping centre.

2. Methods We evaluated 12 PD not-demented patients and 14 controls. In particular, patients who had a severe cognitive impairment (MMSE < 19), a severe motor impairment, auditory language comprehension difficulties (score at the Token Test < 26,5), object recognition impairments (score at the Street Completion Test < 2,25), spatial hemiinattention and neglect, excessive state and trait anxiety (score at the State and Trait Anxiety Index > 40) and excessive depression state (score at the Beck Depression Inventory > 16) were excluded from the study. A neuropsychological evaluation was conducted on the patients selected according to the above criteria, with the aim to obtain an accurate overview of patients’ cognitive functioning. More, the decision making ability was assessed using a virtual version of MET (VMET), which was presented within a virtual supermarket [2-3]. In particular, subjects were invited to buy some items following a defined shopping list and to obtain some information (e.g., the closing time of the supermarket) following specific rules (e.g., you are not allowed to go into the same aisle more than once). While completing the MET procedure, the time of execution, total errors, inefficiencies, rule breaks, strategies, interpretation failures and partial tasks failures (e.g., maintained task objective to completion; maintained sequence of the task; divided attention between components of task and components of other VMET tasks and no evidence of perseveration) were measured. All patients and controls performed a videopolysonnographyc study within a week after the VMET evaluation.

3. Results In normal subjects, neuropsychological tests correlated with the findings of VMET. In PD patients, on the other hand, while traditional neuropsychological test were normal, VMET scores showed significant differences between patients and controls (Table 1). More, five (group A) of our not-demented 12 PD patients of this study showed abnormalities in the videopolysomnographic recordings, such as insomnia, sleep fragmentation and REM behaviour disorders. Concerning VMET analysis, group A in comparison with those patients with normal polysomnographic data (group B), showed significant differences in time of execution (mean p= 0.05) and errors (p = 0.05).

10

G. Albani et al. / Sleep Dysfunctions Influence Decision Making

4. Conclusions VMET gave us important additional data concerning the cognitive status of PD patients, telling us that also PD not-demented patients may present an underlying unknown cognitive dysfunction. Moreover, this study also suggested a correlation between dysexecutive syndrome and sleep abnormalities in PD: five of our not-demented 12 PD patients showed abnormalities in the polysomnographic recordings associated to significant differences in the VMET performance. Table 1. Differences between groups in the VMET performance

Errors Searched item in the correct area Maintained task objective to completion Maintained sequence of the task Divided attention Organized materials appropriately throughout task Self corrected upon errors made during the task No evidence of perseveration Sustained attention through the sequence of the task Buying a chocolate bar Buying toilet paper Buying a sponge Buying two products from refrigerated products aisle Going to the beverage aisle and asking about what to buy Rule breaks Strategies

Group Healthy subjects Patiets Healthy subjects Patiets Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects

N 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12

Mean 17,64 25,08 8,86 11,92 8,86 11,83 8,93 12,08 9,29 12,25 9,50 12,25 9,86 12,50 8,50 11,92 9,43 12,17 9,29 13,25 9,07 13,33 9,07 13,33 9,64 12,83

Std. Deviation 3,895 4,757 1,512 2,314 1,351 2,368 1,328 2,234 1,437 2,379 1,990 2,454 1,834 1,931 1,160 2,429 1,342 2,082 2,555 3,888 2,165 3,939 2,556 3,939 2,590 3,326

14

10,50

2,312

Patients Healthy subjects Patients Healthy subjects Patients

12 14 12 14 12

15,17 28,50 24,92 37,36 47,33

1,992 2,378 3,423 8,608 3,339

5. References [1] [2]

[3]

Shallice, T., & Burgess, P. W. (1991). Deficits in strategy application following frontal lobe damage in man. Brain 114, 727-741. S. Raspelli, L. Carelli, F. Morganti, B. Poletti, B. Corra, V. Silani, and G. Riva, Implementation of the multiple errands test in a NeuroVR-supermarket: a possible approach, Studies in Health Technology and Informatics 154, 115-119. G. Albani, S. Raspelli, L. Carelli, F. Morganti, P.L. Weiss, R. Kizony, N. Katz, A. Mauro, and G. Riva, Executive functions in a virtual world: a study in Parkinson's disease, Studies in Health Technology and Informatics 154, 92-96.


11

Visual Tracking of Laparoscopic Instruments in Standard Training Environments Brian F. ALLEN a Florian KASPER a Gabriele NATANELI a Erik DUTSON b Petros FALOUTSOS a a Department of Computer Science, University of California, Los Angeles b Department of Surgery, University of California, Los Angeles Abstract. We propose a method for accurately tracking the spatial motion of standard laparoscopic instruments from video. By exploiting the geometric and photometric invariants common to standard FLS training boxes, the method provides robust and accurate tracking of instruments from video. The proposed method requires no modifications to the standard FLS training box, camera or instruments. Keywords. Laparoscopic Surgery, Surgery Training, Machine Vision

Introduction Laparoscopic surgery is the most common and widely available minimally invasive surgical technique employed today. With a fiber-optic camera and specialized instruments, entire procedures can be accomplished through keyhole incisions. In comparison to open surgery, laparoscopic procedures are less invasive, require shorter periods of hospitalization and entail faster recovery time and less pain for the patient. However, such benefits do not come without costs. In the case of a laparoscopic surgery, perhaps the primary trade-off is the difficulty of the operation and the need for a specialized repertoire of motor skills. To address the difficulty of training and evaluating the skill of surgeons, theSociety of American Gastrointestinal and Endoscopic Surgeons (SAGES) adopted the Fundamentals of Laparoscopic Surgery (FLS) as a standardized toolset for certification and assessment. FLS is a set of experimentally validated training tasks and equipment [6], providing a standardized means to assess the motor skills specific to laparoscopy. Such objective measure of skill is particularly important in light of studies that show that training surgeons have little ability to self-assess [5]. FLS assessment gauges manual skills entirely on two features of task performance: movement efficiency (measured by the time taken to complete the task) and a precision measure specific to the task. Precision measures include transient, observed actions, such as dropping a block in the peg transfer task, as well as after-the-fact measures, such as divergence from the target circle in the cutting task, or security of a suture knot. Improvement in the accuracy of assessment has been demonstrated by considering more information than FLS records. In

12

B.F. Allen et al. / Visual Tracking of Laparoscopic Instruments in Standard Training Environments

Camera

FLS Training Box FLS Task

Analog Signal Video Digitizer Frames Instrument Edges 2D Tracking

Estimate Trocar Position

Instrument Tip Position (2D) 3D Tracking Instrument Tip Position (3D)

(a) Overview of the process showing data flow.

(b) A standard FLS box trainer.

Figure 1.

particular, tracking the full spatial motion of the instruments during the course of the task performance provided significant gains by considering metrics such as the path length instrument tips travelled [8]. Unfortunately, the equipment needed to acquire detailed spatial tracking data is expensive and specialized. The researchers have predominately employed either (1) precise magnetic tracking [1], (2) mechanical linkages attached to the instruments [7], or (3) virtual reality (VR) simulators with joysticks replacing laparoscopic instruments [11]. Note that (1) and (2) require physical attachments to instruments, while VR simulators typically rely on joysticks that simulate actual laparoscopic instruments. Notably, and most comparable to our work, Tonet et al. [9] considered tracking actual instruments using computer vision. However, that method requires modifying the instruments by affixing a ring of Lambertian material at a know position. In addition, machine vision techniques for laparoscopy have been proposed to control robotic camera holders [10], and for visual-servoing of laparoscopic robots [4]. In this work, we make use of several methods employed by other authors. Voros et al. proposed the use of a probabilistic Hough transform [10] for tracking instruments to automate control of a laparoscope. Doignon et al. [3] describe a least-squares fit of the instrument positions across a series of images to estimate the trocar position. The main contribution of this work is the synthesis of a complete system for tracking tools in FLS training boxes, including the accurate detection of the instrument shafts within the image, the estimation of tool-tip position along the shaft, the automatic registration of the trocar’s position, and the geometric computation of the camera-space position. This method, summarized in figure 1(a), and is specifically tailored to tracking laparoscopic instruments in standard FLS trainer boxes. Our goal is purposefully less ambitious than attempts to track instruments in general settings, such as in vivo. We happily exchange generality for reliability and accuracy in this particularly useful setting.


13

1. Methods and Materials The primary equipment of FLS is a “box trainer,” pictured in figure 1(b), with several ports and a fixed camera. Our system accepts the video recorded by the camera included in the standard FLS training box. 1.1. Image-Space Position (2D) Distinct photometric features of the FLS toolset allow us to robustly track the 2D position of instrument tool-tips within each frame of video. Our algorithm has three phases: (1) color space analysis and extraction of the instrument contours, (2) line fitting to estimate the direction of each instrument shaft , (3) linear search to identify the most probable position of the tool-tip along each instrument. In the standard FLS setup, both the pegs and instrument shafts have a distinct black color. A simple thresholding operation provides a binary probability map of both the pegs and the shafts (shown in figure 2(b)), which we then filter with the application of the erosion and dilation morphological operators. By carefully picking the number of iterations for these operators, we isolate the contours of the two instruments in one step, as shown in figure 2(c). The number of iterations is determined automatically, as described in Section 1.1.2. Automated tuning greatly improves the robustness of this step. By applying the Hough transform on the isolated instrument maps, we extract the lateral contours of each shaft (shown in figure 2(d)). Considering that the instruments are always posed diagonally in the frame, we use the inclination of the lateral contours to group them as belonging to the left and right instruments. We fit a line by least-squares that corresponds to the major axis of each instrument, to each group. The forward direction (from the image borders to the center) of each axis defines a line along which we are going to search for the instrument tool-tips. Figure 2(e) shows the best-fit lines for the example frame. 1.1.1. Searching for the Instrument Tool-Tips with a Confidence Estimate The demarcation point between the instrument and the tool-tip is clearly defined by the abrupt transition between the black color of the shaft and the lighter color of the tool-tip metal body. For added robustness, we search for this point along the direction of each instrument in two functional domains: (1) color space, and (2) gradient space. If we call α the angle between the instrument axis and the the Y-axis, the directional gradient of the image along this angle is given by convolving the image with a rotated forward differencing kernel: ⎡

⎤ cos(α) − sin(α) cos(α) cos(α) + sin(α) ⎣ ⎦ −sin(α) 0 sin(α) −cos(α) − sin(α) −cos(α) −cos(α) + sin(α) The point TG found in the gradient domain is consistently more accurate than TC found in the color space. Therefore, we always use TG for tracking the position of the tool-tip. On the hand, we use TC to produce an estimate of the confidence we have in TG . We found experimentally that the accuracy of tracking is greatly affected by a shift in the color space characteristics of the instrument region, due to the tool-tips getting out of focus. Hence, by estimating the discrepancy

14


(a) Unmodified frame from (b) Binary probability map of (c) Binary mask with isolated the FLS camera during a black regions. instruments. training task.

(d) Extracted lateral contours (e) Instrument direction estiof instruments. mated using line-fitting.

(f) Tracked position in 2D.

Figure 2.

between TC and TG , which are two measurements of the same quantity, we obtain a rather reliable estimate of the confidence of TG . We express this notion as C P (TG ) = 1 − TG −T where β is a normalization constant. β The linear search for TG assumes that there is a single sharp peak in the gradient. However, this assumption is often violated by the presence of specular highlights along the instrument shaft. Noting that such highlight are typically clustered in the region of the shaft closer to the edge of the screen, we mitigate their effect by starting the search from the middle of the instrument axis as opposed to the beginning. 1.1.2. Automated Tuning One parameter that greatly affects the robustness of our approach is the number of iterations for the erosion operator: too many steps remove the instrument regions completely, while too few leave additional noise in the initial binary mask computed in the first step of the algorithm. To address this problem, we consider the raw binary probability map of the pegs and instruments, use a heuristic to remove the instrument contours, and determine the minimum number of erosion steps required to remove all the noise. We repeat this approach for a window of frames to find the best value for the given video sequence. 1.2. Camera-Space Position (3D) The key idea that allows locating the 3D position of the tool-tip from a single frame of video is recognizing that the vanishing point of the edges of the instrument’s image provides the 3D direction of the instrument d [2]. That is, the vector from the camera (i.e., the origin of the camera frame using the pin-hole model) to the vanishing point is equal to the direction of the instrument itself. Figure 2(g) illustrates this property, with R representing the fixed point through which the instrument passes. Likewise, the diagram illustrates that the 3D position of the


15

(g) The geometry of the image formation of (h) Calculation of the depth λ of the trothe instrument. car. The shown plane contains the major axis of the projected ellipse of the trocar (x0 x1 ) and the camera.

Figure 2. Continued.

tool-tip is the intersection of two lines: the line that passes through the trocar R in the direction of the instrument d , and the pre-image of the tool-tip (i.e., the line passing through both the camera point and the image of the tool-tip). This approach assumes that the camera-space position of the trocar R is known. Unfortunately, it is not possible to locate R from a single frame. 1.2.1. Edges’ Vanishing Point and Direction of the Instrument Once the images of the framing edges of the instrument (eu , el ) are found, the vanishing point is V = eu × el , assuming lines eu , el and point V are in 2D homogeneous coordinates. Thus, all lines in the scene that are parallel to the direction of the instrument d will have images that pass through V . Now consider the line that passes through the camera point C and is parallel to d: C + td. The image of this line must also pass through V , as V is the vanishing point for the ¯ is equivalent to C + td. Since the world frame direction d . Therefore, the line CV is simply the camera frame, the direction of the instrument is simply d = V||V−0 || . 1.2.2. Position of Tool-Tip The tool-tip point T is the point on the instrument that corresponds to the distal end of the instrument (see figure 2(g)). The tool-tip is some unknown distance k from R in the direction of the tool, T = R + sd. But note that T is also located on the pre-image of point T , i.e., on the line C + t(T − C) = t(T − 0) = tb with b ≡ (T − 0). The procedure for locating T in the image will be considered in the next section. Ideally, T is simply at the intersection of lines L1 (s) = R + sd and L2 (t) = tb, however such precision is unlikely. Instead, consider the points on each line s) and L2 (t˜). The segment L1 (˜ s)L2 (t˜) is uniquely closest to the other line, L1 (˜ perpendicular to both L1 (s) and L2 (t).

16


d, bb, (R − 0) − b, bd, (R − 0) d, db, b − (d, b)2

(1)

d, db, (R − 0) − d, bd, (R − 0) t˜ = d, db, b − (d, b)2

(2)

s˜ =

s)L2 (t˜) as the estimate of T gives Taking the midpoint of L1 (˜

T =

(R + s˜d) + (t˜b) . 2

(3)

1.2.3. Locating the Trocar from a Sequence of Images So far we have assumed that the position of the trocar R (the center of the region of space through which all instruments pass) is known. To determine R , the framing edge pairs (eiu , eil ) for each frame i are collected. If there were no errors in the (eiu , eil ), the image of the trocar would be the locus of points on the image plane between the edges for all frames. Due to inevitable noise, the actual image of the trocar is smaller than the observed locus. To more robustly find the trocar’s image, an image point R is found as the point closest to all of the (eiu , eil ), that is, for E = {eiu , eil }, ∀i and v i ⊥ li for all l ∈ E, R = arg max p∈I

projv (l0i − p) .

(4)

i

With the center of the image of the trocar R determined, the ellipse centered at R with one axis of (R − 0) that best matches the set of E is found. Define x0 ≡ ||R − [w/2, h/2]T || and x1 ≡ x0 + m, where w, h are the width and height of the image, and 2m is the length of the major axis of the ellipse. The geometry of the trocar’s projection is shown in figure 1(a), in the plane containing the ellipse’s major axis and the camera. Defining a, b, r , d as in figure 1(a), the depth of R, λ is determined by r = d tan(b) x0 d = sin(a)

b = tan−1 (x1 ) − tan−1 (x0 ) r λ = d . r

With both λ and the image of the trocar R , the 3D position of the trocar is known. 2. Results For our experiments, we captured several video sequences of the FLS peg transfer task with the standard camera included in the box trainer and a completely unaltered setup. The illumination is provided by an array of LED lights included in the box. One group of tasks was performed by an expert surgeon featuring controlled smooth motions, while a second group was performed by a novice and


17

is affected by jittery non-smooth motions. In both cases, we recorded robust 2D tracking of the instrument tips that were validated visually. Figure 2(f) shows the tracked position (in yellow) of the two instrument tips from the unmodified FLS video, shown in figure 2(a). The accompanying video shows the performance of our tracker for a short clip with thumbnails of the intermediate steps. The measure of confidence of the tracked position allows us to automatically disable tracking of an instrument tip when it is no longer visible in the scene. The tracker is unable to track the position of the instrument tip accurately when the instrument is too close to the camera and thus very blurry. However, in such cases, the measure of confidence is very low, as expected. 3. Conclusion In this paper we presented a complete system for tracking the 3D position of the instrument tips of a standard FLS box trainer. Our approach is robust, does not require any physical alteration of the toolset, and works with the standard camera included in the kit. In the future, we would like to combine our existing tracking capabilities with a more thorough analysis of the entire scene as a means to produce a more accurate assessment of FLS tasks. References [1]

[2] [3]

[4]

[5]

[6]

[7]

[8] [9]

[10]

[11]

B. Allen, V. Nistor, E. Dutson, G. Carman, C. Lewis, and P. Faloutsos. Support vector machines improve the accuracy of evaluation for the performance of laparoscopic training tasks. Surgical endoscopy, 24(1):170–178, 2010. A. Cano, P. Lamata, F. Gay´ a, and E. G´ omez. New Methods for Video-Based Tracking of Laparoscopic Tools. Biomedical Simulation, pages 142–149, 2006. C. Doignon, F. Nageotte, and M. de Mathelin. The role of insertion points in the detection and positioning of instruments in laparoscopy for robotic tasks. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2006, pages 527–534, 2006. A. Krupa, C. Doignon, J. Gangloff, and M. de Mathelin. Combined image-based and depth visual servoing applied to robotized laparoscopic surgery. In Proc. of the 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2002. VA Pandey, JHN Wolfe, SA Black, M. Cairols, CD Liapis, and D. Bergqvist. Selfassessment of technical skill in surgery: the need for expert feedback. Annals of The Royal College of Surgeons of England, 90(4):286, 2008. J. Peters, G.M. Fried, L.L. Swanstrom, N.J. Soper, L.F. Sillin, B. Schirmer, K. Hoffman, et al. Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery, 135(1):21–27, 2004. J. Rosen, J.D. Brown, L. Chang, M. Barreca, M. Sinanan, and B. Hannaford. The Blue Dragon-a system for measuring the kinematics and the dynamics of minimally invasive surgical tools in-vivo. In Proceedings- IEEE International Conference on Robotics and Automation, volume 2, pages 1876–1881. Citeseer, 2002. C.D. Smith, T.M. Farrell, S.S. McNatt, and R.E. Metreveli. Assessing laparoscopic manipulative skills. The American Journal of Surgery, 181(6):547–550, 2001. O. Tonet, R.U. Thoranaghatte, G. Megali, and P. Dario. Tracking endoscopic instruments without a localizer: A shape-analysis-based approach. Computer Aided Surgery, 12(1):35– 42, 2007. S. Voros, J.A. Long, and P. Cinquin. Automatic detection of instruments in laparoscopic images: A first step towards high-level command of robotic endoscopic holders. The International Journal of Robotics Research, 26(11-12):1173, 2007. JD Westwood, HM Hoffman, D. Stredney, and SJ Weghorst. Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: Results from randomised controlled studies using the MIST VR laparoscopic simulator. Medicine Meets Virtual Reality: art, science, technology: healthcare and evolution, page 124, 1998.

18


On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy Joseph B ANSTEYa, Erin J SMITHb, Brian RASQUINHAb, John F RUDANc, and Randy E ELLISa,b,c,1 a School of Computing, Queen’s University, Kingston, Ontario, Canada K7L3N6 b Department of Mechanical and Materials Engineering, Queen’s University c Department of Surgery,Queen’s University

Abstract. There is a growing body of evidence to suggest the arthritic hip is an irregularly-shaped, aspherical joint, especially in severely pathological cases. Current methods used to study the shape and motion of the hip in-vivo, are invasive and impractical. This study aimed to assess whether a plastic model of the hip joint can be accurately made from a pelvic CT scan. A cadaver hemi-pelvis was CT imaged and segmented from which a 3D plastic model of the proximal femur and hemi-pelvis were fabricated using rapid-prototyping. Both the plastic model and the cadaver were then imaged using a high-resolution laser scanner. A three-way shape analysis was performed to compare the goodness-of-fit between the cadaver, image segmentation, and the plastic model. Overall, we obtained submillimeter fit accuracy between all three hip representations. Shape fit was least favorable in areas where the boundary between cartilage and bone is difficult to distinguish. We submit that rapid-prototyping is an accurate and efficient mechanism for obtaining 3D specimens as a means to further study the irregular geometry of the hip. Keywords. Hip, Anatomy, Arthritis, Computed Tomography, Stereolithography

Introduction Detailed physical study of population variations in anatomy, including bones, is limited by the availability of specimens but computed tomography (CT) scans of patients are more abundant. This raises the question: how accurate is the reverse engineering of anatomy from a medical image? To take the hip joint as an example, the currently accepted belief is that the hip is a ball-and-socket joint with spherical congruent joint surfaces of the femoral head and acetabulum [1]. However, there is an emerging body of evidence to suggest the contrary – that the arthritic hip is, in fact, aspherical in nature [2]. This is especially true in pathologies such as femoroacetabular impingement, osteoarthritis, and developmental hip dysplasia. Thus it is important to accurately understand the shape and movement of this irregular joint in order to devise appropriate treatments for disease.

1

Corresponding Author: Randy E Ellis, School of Computing, Queen’s University, Kingston, ON, Canada K7L 3N6; E-mail: [email protected]

J.B. Anstey et al. / On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy

19

Because of the unique shape and anatomical location of the pelvis, it is difficult to study the motion of this joint in vivo using optoelectrically tracked skin markers. Recent efforts have been made to study the motion of the hip using computer navigation [3]; however, these methods are invasive and thus not practical for largescale in vivo studies. Moreover, cadaver specimens with specific pathologies are difficult to source, expensive, and if un-embalmed (e.g., frozen) short-lived. By comparison, medical images of pathological hips, such as CT scans, are readily available as they are often standard practice in pre-operative care. The purpose of the current investigation was to assess the accuracy of replicating the shape of the hip joint using 3D rapid prototyping. If this process is sufficiently accurate, then plastic models derived from patient CT images could potentially be used as a means to study the kinematics of an irregularly shaped or pathological hip joint.

1. Methods & Materials A formaldehyde-fixed hemi-pelvis was imaged with all soft tissues intact using a 16slice CT scanner (Lightspeed+ XCR , General Electric, Milwaukee, USA) with a slice thickness of 0.625mm. The images were saved and later imported into commercially available Mimics software (Materialise, Leuven, Belgium). The anatomy was systematically segmented into 3D digital models using a step-wise process that ensured the production of precise representations of the imaged anatomy. The process began by applying a threshold to highlight the surface of the bony anatomy with a mask. This mask was then manually edited until satisfactory segmentation of the hip in all three orthogonal planes (coronal, sagittal, and axial) was achieved. The masks for the hip bone and proximal femur were rendered into digital 3D models. The models were then visually examined for unusual bumps or pits that may be considered atypical of the anatomy. If an unusual surface feature was observed, the area was compared to the raw CT images of that location. If the unusual feature was found to accurately represent the CT data no action was taken, otherwise the area was edited to accurately reflect the images in the CT data. When satisfied with the outcome of the digital 3D models, they were saved as a Stereo-Lithography (.STL) file and sent to a rapid-prototyping machine (Dimension sst 1200es, Stratasys, Eden Prairie, USA) for fabrication. Upon printing, the model was again visually examined for any unusual surface features not seen in the CT data and their articulation with one another was examined to ensure that they did indeed articulate (since we knew the cadaver anatomy articulated, it was important to ensure that the modeled anatomy also articulated). To ensure that the articulations were typical of a human joint, a senior orthopedic surgeon (JFR) was consulted to evaluate the articulation. 1.1. Cadaver Preparation The bones comprising the cadaver hip were retrieved by removing all soft tissues using typical dissection techniques with a scalpel, forceps, and a blunt probe. The labrum and fat pad were also removed from the acetabulum, and attention was given to the fovea on the head of the femur to remove the remnants of the ligamentum teres. The bones were scraped clean using a scalpel and a 5% solution of hydrogen peroxide to loosen tissue from the non-articulating bone surfaces. Our goal was to compare the 3D

20


models to the actual anatomy with the articular cartilage intact so extra care was taken to not damage the cartilage. 1.1.1. Shape Analysis The head and acetabular components of both the cadaver and plastic models were scanned using a laser scanner (Central V.5.7.0, ShapeGrabber, Ottawa, Canada) to obtain point-cloud representation of their surfaces. Because of the complex 3D geometry of the components, the scans were acquired in small patches that were within the plane of the laser; the specimens were rotated between each scan to collect data from the entire surface. A three-way analysis (Figure 1) was performed to determine the goodness-of-fit between: i) the cadaver and derived CT segmentation, ii) the CT segmentation and subsequent plastic model, and iii) the cadaver and the plastic model. The laser-scanned point cloud data was used to generate a STL tessellation for each surface patch. These were imported into the Mimics environment along with the 3D segmentation model. Mimics was used to perform a preliminary registration of each surface patch to the 3D segmentation model. This was accomplished both manually (visually) as well as with the local and global Mimics registration tools. The registered surface patches, now in a common global coordinate frame of the 3D segmentation model, were exported as new STL files. These files were imported into MATLAB (MathWorks, Natick, MA) for all subsequent data analysis.

Figure 1: 3-way shape analysis

In the MATLAB environment, a refined registration was performed using an iterative closest point (ICP) algorithm [4] to register each patch to the segmentation model. Subsequently, the 3D segmentation model was triangulated using a Delaunay triangulation [5], and the closest point to each triangle, as well as its corresponding distance (residual), was located on each patch using an iterative closest point algorithm. For the set of point matches, the root-mean-square error (standard deviation, σ) of the residual was computed and used to filter the extreme ~5% outliers (1.96σ). Subsequently, a second refined registration was performed for each patch and new statistical measures computed: residual distance at each point, average and maximum deviations, and root-mean square errors.


21

2. Results A summary of match results for the proximal femur and acetabulum are shown in Tables 1 and 2, respectively. Signed distances were computed to determine whether the matches were inside or outside the target, with positive numbers being outside the target (larger) and negative numbers being inside (smaller). Overall, we obtained sub-millimeter shape accuracy between the shape of the cadaver hip region and both the resulting CT segmentation and 3D plastic model. In both cases, the cadaver was slightly smaller than the CT segmentation. Similarly, the model was slightly smaller than the CT segmentation from which it was derived. As was expected from these findings, the cadaver and model were a close match, with the cadaver being slightly smaller than the model. The residual distances computed from the matched object to the target object were plotted in order to visualize areas of good and poor fit. Figures 2 and 3 depict matches outside ± 1σ for the three-way match. By comparing cadaver specimens to these residual plots, it was noted that mismatches tended to occur in specific regions. These included areas where there was residual soft tissue on the cadaver specimen that was detected by laser scanning, but not in the CT segmentation or consequently the model (these are positive distances, indicating that they are external). Both osteophytic regions on the femur and along the acetabular rim also showed greater mismatch, likely because osteophytes are difficult to segment due to their semi-cartilaginous compositions. For the same reasons, areas of cartilage deterioration on the surface of the femoral head also showed a higher degree of mismatch.

3. Discussion There were potential sources of error in our data collection and analytical process, which were consequences of the time it took to acquire data in the study. The cadaver was first imaged with CT, then dissected, and laser-scanned at a later date. After CT imaging, the cadaver specimen was wrapped in a cloth soaked with a moistening solution (water, glycerin, Potassium Acetate, and Dettol) and stored in a heavy duty plastic bag at room temperature. Post-dissection, the remaining bones were stored using the same method until the completion of our study. It is unknown how this storage process may have affected the size and shape of the bone. Because the bones had to be exposed to the room’s environmental conditions during the laser scanning for extended periods of time (up to 2.5hrs) on multiple occasions, there may have been changes due to dehydration, especially of the articular cartilage. In particular, we noticed that dehydration of the specimen over time led to tiny “hairs” of periosteal tissue to appear on surface of the cadaver specimens. These “hairs” may have affected the quality of registration, and hence the quality of analysis of cadaver-based comparisons. This is further supported by the better matches observed between the smooth plastic model and the CT segmentation.

22


Figure 2: Residual distance maps for proximal femur matches. Poorly-matched areas outside of one standard deviation are shown as black (smaller than the match target) or white (larger than the match target). Areas within one standard deviation are uncolored (gray).

Table 1: Results of proximal femur matches. Root-mean-square error, average and maximum deviation were computed for the residual distance at each point. Signed distances were computed to determine whether the matches were inside or outside the target, with positive numbers being outside the target (larger) and negative numbers being inside (smaller).

RMSE (σ) Average Deviation (unsigned) Average Deviation (signed) Max Deviation (unsigned)

CADAVER-toSEGMENTATION 0.61 mm

MODEL-toSEGMENTATION 0.49 mm

CADAVER-toMODEL 0.48 mm

0.58 mm

0.47 mm

0.42 mm

-0.49 mm

- 0.46 mm

-0.32 mm

1.62 mm

0.94 mm

1. 58 mm

Our results also suggest that there may have been some over-segmentation of the CT scans, mainly in regions containing osteophytes (such as the femoral head-neck junction and the acetabular rim) and along the articular surface, especially in areas of cartilage deterioration. In these regions it was particularly difficult to distinguish a definitive boundary between bone and cartilage on the CT images, even with careful segmentation. Over-segmentation would cause the segmentation and resulting model to be slightly larger than the cadaver, which is implied in our results. We also noted that the plastic model was slightly smaller than the CT segmentation from which it was derived. However, mismatches appeared to be much more uniform over the entire surface, rather than in specific concentrations as we saw with the cadaver-to-CT match. We also observed a tendency for mismatches to follow the directions of material deposition. There are several potential explanations for these observations including the resolution of the 3D printer (approximately ±0.1mm), anisometric plastic hardening following deposition, and thermal fluctuations at the time of laser-scanning that may have an effect on the volume of the plastic model.


23

Figure 3: Residual distance maps for acetabulum matches. Poorly-matched areas outside of one standard deviation are shown as black (smaller than the match target) or white (larger than the match target). Areas within one standard deviation are uncolored (gray).

Table 2: Results of acetabulum matches. Root-mean-square error, average and maximum deviation were computed for the residual distance at each point. Signed distances were computed to determine whether the matches were inside or outside the target, with positive numbers being outside the target (larger) and negative numbers being inside (smaller).

RMSE (σ) Average Deviation (unsigned) Average Deviation (signed) Max Deviation (unsigned)

CADAVER-toSEGMENTATION 0.81 mm

MODEL-toSEGMENTATION 0.58 mm

CADAVER-toMODEL 0.54 mm

0.72 mm

0.55 mm

0.47 mm

-0.58 mm

-0.55 mm

-0.43mm

2.86 mm

1.91 mm

1.94 mm

Additionally, both the CT imaging and laser scanning processes have inherent inaccuracies that may have been propagated through the analytical pipeline. Although we obtained high-quality CT images, our segmentation remained limited to the resolution of the CT scans (0.625mm). Moreover, image quality may have been reduced or distorted through the image processing pipeline, as images were changed from one format to another. For instance, CT slice pixels were combined to form 3D voxels which were then triangulated (without smoothing) to form a surface for subsequent analysis. This study was limited to a single specimen as a proof of the concept of using a rapid prototyping process to reconstruct bony anatomy. Future work could include expanding the number of specimens, using fresh-frozen cadaveric material (or immediately post-mortem animal tissue), and comparing various pathologies to determine whether diseased bone can be accurately reconstructed. On the basis of this work, we are encouraged at the prospect for the use of rapid prototyping as a novel tool in the anatomical sciences. For example, this representation was used to analyze the morphology of the joint by fitting an ellipsoid to the articular surfaces (as in [2]) which quantitatively demonstrated asphericity of the femoral head. If a larger sample size is found to support our current findings we may also begin replicating patient hip joints with no known pathologies and determine whether those hip joints are also aspherical.

24


The results of which have the potential of changing the way the geometry of the hip joint is viewed in mechanical and scientific disciplines.

4. Conclusion Three-dimensional rapid prototyping derived from high-quality CT image segmentations can accurately represent the true shape of the hip joint with submillimeter accuracy. The outcome, however, is clearly dependent on the accuracy of the image segmentation from which the model is derived. Therefore, care must be taken to accurately define the cartilage boundary especially along articular surfaces and in osteophytic regions. Although we can claim that plastic models can accurately depict the shape of the hip joint, more work is needed to draw conclusions concerning use of these models to accurately represent the motion of this joint.

Acknowledgements This work was supported in part by the Canada Foundation for Innovation, the Canadian Institutes for Health Research, Kingston General Hospital, and the Natural Sciences and Engineering Research Council of Canada.

References [1]

Cailliet, R.: The Illustrated Guide to Functional Anatomy of the Musculoskeletal System. American Medical Association, 2004.

[2]

Ellis, R., Rasquinha, B., Wood, G., Rudan, J.: 3D Shape Analysis of Arthritic Hips: A Preliminary Study. Int J Comp Assist Radiol Surg, S137–S142, 2010.

[3]

Thornberry, R. L.: The Combined Use of Simulation and Navigation to Demonstrate Hip Kinematics. J Bone Joint Surg(Am) 91:144-152, 2009.

[4]

Besl, P., McKay, N.: A Method for Registraion of 3-D Shapes. IEEE Trans Pattern Anal Machine Intell 4(2),:239-256, 1992.

[5]

Barber, C., Dobkin, D., Huhdanpaa, H.: The Quick-hull algorithm for convex hulls. ACM Trans Math Software 22(4):469-483, 1996.


25

Classification of Pulmonary System Diseases Patterns Using Flow-Volume Curve a

Hossein ARABALIBEIKa,1, Samaneh JAFARIa and Khosro AGIN b Research Center for Science and Technology in Medicine (RCSTIM), Tehran University of Medical Sciences, Tehran, Iran b Shahid Beheshti University of Medical Sciences, Tehran, Iran

Abstract. Spirometry is the most common pulmonary function test. It provides useful information for early detection of respiratory system abnormalities. While decision support systems use normally calculated parameters such as FEV1, FVC, and FEV1% to diagnose the pattern of respiratory system diseases, expert physicians pay close attention to the pattern of the flow-volume curve as well. Fisher discriminant analysis shows that coefficients of a simple polynomial function fitted to the curve, can capture the information about the disease patterns much better than the familiar single point parameters. A neural network then can classify the abnormality pattern as restrictive, obstructive, mixed, or normal. Using the data from 205 adult volunteers, total accuracy, sensitivity and specificity for four categories are 97.6%, 97.5% and 98.8% respectively. Keywords. pulmonary function test, respirometry, flow-volume curve, artificial neural networks

Introduction Early detection of respiratory system abnormalities raises the chances of successful treatments and drops related costs. Pulmonary function tests (PFTs) measure the efficiency of lungs function. Spirometry is the most widely used PFT. It records the amount of air breathed in and out and the rate at which this process takes place [1]. The preliminary output of the spirometry test is the flow-volume curve. This curve is constructed by calculating the flow and volume of the inhaled and exhaled air during an inspiration and expiration cycle performed with maximum effort (Figure 1a). Normally, Vital Capacity (VC), Forced Vital Capacity (FVC), Forced Expiratory Volume at 1st second (FEV1), ratio of FEV1 to FVC (FEV1%), Peak Expiratory Flow (PEF) and Forced Expiratory Flow at 25 to 75% (FEF 25-75) are extracted from this curve and used as a basis for diagnosis. Age, height, sex and ethnic of the patient influence expected normal values of the measured parameters which in turn affect interpretation of the spirometry results [2].

1

Corresponding Author: Research Center for Science and Technology in Medicine (RCSTIM), Imam Khomeini Hospital, Keshavarz Blvd, Tehran, Iran, Tel: +98 21 66581505, Fax: +98 21 66581533, E-mail: [email protected].

26

H. Arabalibeik et al. / Classification of Pulmonary System Diseases Patterns

Various respiratory diseases generate different flow-volume curve patterns. Restrictive lung diseases (e.g. change in lung parenchyma, disease of the pleura, chest wall or neuromuscular apparatus) are identified by reduced lung volume leading to a shrunk version of the normal flow-volume curve [3]. This pattern is characterized by low FVC and comparatively high expiratory flow (Figure 1b). The obstructive pattern is characterized by a decreased flow and FEV1, usually along with normal or increased volume (Figure 1c). This pattern is a consequence of progressive airflow obstruction in the peripheral airways, associated with lung inflammation, emphysema and mucus hyper secretion [4]. Examples of obstructive airway diseases are asthma, chronic bronchitis, chronic obstructive pulmonary disease (COPD) and emphysema.

(a)

(b)

(c)

(d)

Figure 1. Flow–volume curve of (a) Normal, (b) Restrictive, (c) Obstructive and (d) Mixed subjects


27

In mixed pattern, respiratory system suffers from both obstructive and restrictive abnormalities. Normally volume reduces more than flow (Figure 1d). So this pattern is characterized by reduced FEV1 and FVC values and increased FEV1%. Automated diagnosis systems generally use the extracted parameters from the curve. Some recent publications have suggested the use of different intelligent systems as decision support systems to help the physicians in diagnosis [2-7]. All of these methods just use the mentioned parameters, while expert physicians use the morphology and pattern of the flow volume curve as well. Are these parameters sufficient to capture the precious data stored behind the obtained curve? In this research we will show that some simple and computationally inexpensive parameters can better capture the pattern of the curves and contribute more in diagnosing the diseases. In this work MLP neural networks are used as classifier to discriminate between four patterns of pulmonary system operation namely normal, obstructive, restrictive and mixed.

1. Methods and Materials Flow-volume data of 205 adult volunteers consisting of 90 normal, 30 restrictive, 32 obstructive and 53 mixed pattern cases are obtained using a Spirojet spirometer (Ganshorn Company: www.ganshorn.de). The data is then divided to 155 training and 50 test samples. Predicted values of FVC, FEV1, FEV1% and PEF are obtained using age, gender, height and race of patients. The standard protocol of a breath cycle in spirometry according to the recommendation of the American Thoracic Society (ATS) consists of an inhaling to total lung capacity and then exhaling as hard and completely as possible. Diseases such as goiter change the inspiration part of the flow-volume curve, while the expiration part is affected by obstructive, restrictive and mixed abnormality patterns. Curve fitting is a parametric model estimation algorithm. According to a cost function, the algorithm tries to find the optimal values of the predefined smooth function coefficients. The cost function is a measure of the error between the real data and their approximation by the fitted curve. Polynomial models given by (1)

are used in this study to extract some simple features regarding the curve patterns, where n is the order of the polynomial Artificial Neural Networks (ANNs) are computational models consisting of simple processing units connected in a layered structure. They provide promising tools for complex systems modeling, function approximation and decision making in nonlinear multi-criteria domains by learning from examples. A Multilayer Perceptron (MLP) neural network stores the extracted knowledge in layer weights. Learning takes place by adapting weights to minimize the output error between the network’s output and the desired values. Various MLP networks, with different hidden layers and diverse number of neurons in each hidden layer are used to classify four respiratory diseases patterns.

28


Figure 2. A sample of flow-volume curve. Dashed line: original curve; solid line: fitted curve

2. Results Polynomials of orders 5 and 6 are used which leads to R-squared values of more than 0.99. This shows a good fitting of the curves to the measured data which preserve most of the details needed for the diagnosis (Figure 2). MLPs with 1 and 2 hidden layers and diverse number of neurons in each hidden layer are used as classifier. Tangent-sigmoid transfer function is used for hidden layers and linear transfer function for the 4 neurons of the output layer. Each output neuron, specify one of the patterns. The coefficients (pi) of fitted curves as well as the predicted values of FVC, FEV1 and FEV1% were used as inputs of the neural network. The network is trained for a mean squared error of less than 10e-5, using Levenberg– Marquardt (LM) algorithm during 300 epochs. We use early stopping to avoid decrease in the generalization ability of the network caused by over fitting. To compare the discrimination power of each extracted feature, we used Fisher Discriminant Ratio (FDR) which considers both within class and between class scatterings [8]. Figure 3 shows that polynomial coefficients have considerably higher FDR values than parameters usually used for classification. Accuracy, sensitivity and specificity results for different networks are presented in Table 1. For comparison purposes, the best results of using FEV1, FEV1%, FVC, and their corresponding predicted values as decision parameters are also presented (ANN15).

Figure 3. Fisher Discriminant Ratio for different features


29

Table 1. Comparison of different MLP structures No 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Architecture 10-7-4 10-20-4 10-25-4 10-30-4 10-40-4 10-45-4 10-13-10-4 10-13-18-4 10-13-20-4 10-13-30-4 10-13-40-4 10-13-50-4 9-20-4 9-30-4 6-10-10-4

Accuracy 92.5 97.56 95.12 95 96.25 92.7 95.12 95 95.12 95 95.12 90.25 95.12 92.69 87.8

Sensitivity 92.5 97.5 95 95.12 95.12 92.5 92.5 95.12 95 95.12 95 90 95 92.5 82.5

Specificity 98 98.8 98.1 97.61 97.48 97.4 98 99 99 99 98 97 98.1 96.93 95.97

Table 2. Comparison of the results No 1 2 3 4 5

Classifier [2] [5] [6] [7] This work

Accuracy 90 92 92.3 93 97.6

Sensitivity 91.6 92.3 92.6 93 97.5

Specificity 87.5 91.6 91 98 98.8

3. Discussion The 10-20-4 configuration has the best result in classifying respiratory patterns within three layer networks. Most of the networks in the four layer structure present very close results. According to Table 1, ANN2 with a 10-20-4 structure also presents the best results in general. Excessive number of neurons in the hidden layer does not raise the classification performance. In fact, unnecessary modeling power of the network causes over fitting which in turn initiates early stopping. The results of ANN7 to ANN12 show that assuming one more hidden layer results in the same problem. Comparing ANN2 and ANN13, as well as ANN4 and ANN14, show that the polynomial of order 6 outperforms the order 5 polynomial. Although higher polynomial orders preserve more details of the flow-volume curve, but simulations show that polynomials of orders greater than 6 provide unnecessary inputs to the ANN. This makes the MLP more complex without increasing its classification performance. At the other hand, lower order polynomials do not capture the considered necessary details of the curves for appropriate diagnosis and classification. In another word, using the proper order of the polynomial preserves necessary information for the classification and filters out the unnecessary details and noises that not only do not contribute to diagnosing ability but also weaken it. Comparison of accuracy, sensitivity and specificity results of this study and previous works (Table 2) shows that using a set of simple computational features that capture the morphology of the flow-volume curve, results in an improved classification of respiratory disease patterns.

30


ANN15 shows that using a simple MLP neural network of the comparable size with normally used parameters of FEV1, FEV1%, FVC, and their predicted values does not tend to good results. It means that better performance of ANN1 to ANN14 could be attributed to the selected features.

4. Conclusions Spirometry is a common and helpful test in evaluating the functionality of pulmonary system. Normally some parameters like FEV1 and FEV1% which are extracted from the flow-volume curve are used for classification of respiratory system disease patterns. These parameters have rather single point characteristics. They do not represent the shape of the curve sufficiently. The results of this research show that the curve contains more precious information than just what these parameters capture. Using some simple parameters such as fitted curve coefficients, one can extract the information behind spirometry output curve more precisely as an expert physician does.

References [1] [2] [3] [4] [5]

[6] [7]

[8]

http://www.thoracic.org (last accessed: 2010/01/01). M. Veezhinathan and S. Ramakrishnan, Detection of obstructive respiratory abnormality using flow– volume spirometry and radial basis function neural networks, J. Med. Syst. 31 (2007), 461–465. C.R. Sweeney, Equine restrictive lung disease Part 1: Overview, in P. Lekeux (Ed.), Equine Respiratory Diseases, International Veterinary Information Service, Ithaca, New York, USA, 2004. A. Husain and S. Habib, Pattern identification of obstructive and restrictive ventilatory, Pak. J. Physiol. 4 (2008), 30–34. V. Mahesh and S. Ramakrishnan, Assessment and classification of normal and restrictive respiratory conditions through pulmonary function test and neural network, J. Med. Eng. Techno. 31 (2007), 300– 304. M.J. Baemani, A. Monadjemi and P. Moallem, Detection of respiratory abnormalities using artificial neural networks, Journal of Computer Science 4 (2008), 663–667. H. Arabalibeik, M.H. Khomami, K. Agin and S. Setayeshi, Classification of restrictive and obstructive pulmonary diseases using spirometry data, In Studies in Health Technology and Informatics 142, IOS press, 2009. G.J. McLachlan, Discriminant analysis and statistical pattern recognition, John Wiley & Sons, New York, 1992.


31

Cost-Efficient Suturing Simulation with Pre-Computed Models Venkata Sreekanth ARIKATLAa , Ganesh SANKARANARAYANANa and Suvranu DE a,1 a Rensselaer Polytechnic Institute, Troy, NY

Abstract. Suturing is currently one of the most common procedures in minimally invasive surgery (MIS). We present a suturing simulation paradigm with precomputed finite element models which include detailed needle-tissue and threadtissue interaction. The interaction forces are derived through a reanalysis technique for haptic feedback. Besides providing deformation updates and high fidelity forces, our simulation is computationally less costly. Keywords. Surgery simulation, Suturing, Real-time simulation, Finite elements

Introduction More often than not, surgery simulation involves intricate procedures being performed over complex geometries. The main cost in most physics-based surgery simulation environments is the deformation update. For this reason, pre-computed methodologies [1] are sometimes preferred over iterative or direct-solution procedures. Precomputation based methodologies aid in dramatic cost reduction during run-time. Nevertheless, some limitations include the restriction mostly to linear formulation and no topology changes being allowed. Suturing is now-a-days one of the most common surgical procedures in MIS (Minimally Invasive Surgery). In this paper, we model the suturing procedure using pre-computed methods to simulate the deformation and interaction forces. Unlike in [2], we aim for detailed needle-tissue and thread-tissue interaction. We specifically use the reanalysis technique [3] in conjunction with the superposition principle for linear elasticity to update the deformation and the reaction forces as a result of needle-tissue and thread-tissue interactions. This culminates in high fidelity tissue response while utilizing fewer computational resources.

1. Tools and Methods The suturing procedure in MIS requires bimanual interaction with the needle and the thread in order to suture on the tissue base. The sections below describe the techniques we employed at various stages to achieve this goal. 1

Corresponding Author: Dr. Suvranu De, Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute, Email: [email protected]

32

V.S. Arikatla et al. / Cost-Efficient Suturing Simulation with Pre-Computed Models

1.1. Deformation We adopt a linear elastic material model discretized using tetrahedral finite elements. This is a standard procedure and results in a set of simultaneous equations of the form (1) Where,

is the stiffness matrix,

is the displacement vector and

is the external

force vector. We pre-compute the inverse of as for runtime use. During the user interaction, we use the reanalysis technique as in [3] to compute the deformation field and force. If the degrees of freedom are rearranged based on the interacted node, we can write

(2) The sub-matrices in the above equation are derived according to which node the , . user interacts with. Expanding the above matrix, we obtain Since

is small in dimension, its inverse can be computed on the fly. This technique

can only be used if the interaction is local. In order to handle multiple needle/thread interactions simultaneously, we exploit the property of superposition in our linear formulation.

Figure 1. Various stages during the suturing procedure

1.2. Modeling the Suturing Procedure We model the suturing procedure based on the aforementioned reanalysis technique given that the interaction paradigm in the simulation is point-based. Figure 1 shows the


33

division of the suturing procedure into logical steps based on the needle and thread interaction with tissue. These four stages: (1) The sharp tip of the needle enters the tissue (2) The tip of the needle emerges from the tissue (3) The blunt end goes inside the tissue when the sharp end is pulled by the grasper. The thread also interacts with the tissue (4) the needle is out of the tissue. Only the thread remains inside the tissue. 1.3. Needle-Tissue Interaction When the needle first enters the tissue, the surface triangle which it pierces is determined from the dynamic point algorithm [4] and recorded. The boundary condition from the interaction is applied through displacement conditions on the nearest node of the entry triangle. In essence, the nearest node should follow the point on the needle that is on the tissue surface. Since the needle is curved, we divide it into a set of straight line segments (see Figure 2(a)). At every time step, the nearest point on the needle to the entry/exit is calculated and its displacement is set accordingly. In Stage 2, when the sharp end pierces out of the tissue, the triangle of exit is recorded. Since at this stage two different points on the needle intersect the surface, boundary conditions at both the entry and exit triangles are applied separately and are superimposed to obtain the resulting deformation since the underlying formulation is linear.

Dynamic Point

(a)

(b)

Figure 2. (a) The curved needle is divided into several segments. Each segment has one dynamic point (b) Type 1 and Type 2 interactions of the thread with tissue

1.4. Thread-Tissue Interaction The suture thread is attached to the end of the needle and is meant to hold the tissue in place after the knot is tied. This is modeled with a follow-the-leader (FTL) algorithm [5]. In the FTL algorithm, the nodes away from the one that is being controlled by the user are moved toward their corresponding leader to preserve the length. Although this is a non-physical technique, it is very stable as well as less costly to employ within the requirements of simulating the suture thread. The thread interacts with the base (modeled with FEM) after Stage 2 in the suturing process. Specifically, the thread interacts with the model in two ways. Type 1: The thread is only guided through the point where the needle enters or exits. Type 2: Part of the thread inside the tissue is pulled on either side. Force is imparted to the user.

34


This force is proportional to the frictional resistance that the tissue imparts to the thread while it is being slid through it. These interactions are shown in Figure 2(b). In Type 1 interaction, the nearest node is snapped to the entry point on the tissue after the FTL update. Hence the FTL update is overridden. In case of type 2 interaction, the entry point on the model follows the node i that is associated with the entry point. This associated node may change as the user pulls the suture thread using the grasper. to reset the association. If the number of segments on We use a threshold force the suture thread is high enough, one can feel the stiction force between the thread and the tissue as in reality. After Stage 4 is reached, the suture should be secured by tying a knot. For this purpose, a simple and real-time knot tying algorithm, as proposed in [6], was employed. It is built around the FTL algorithm for simulating the knot. After the user closes the knot onto the tissue, the nodes that form the knot are frozen from any further moment. A snapshot of the suturing simulation is shown in Figure 3.

Algorithm for Type 2 interaction LOOP 1. 2. 3.

Update Thread using FTL1 & FTL2 Find the nearest node to entry/exit IF a.

Set the boundary of the nearest vertex on the entry/exit triangle to node i on suture

b.

Compute

c.

IF i. ii.

d.

Recalculate and set ELSE

i. 4.

Reset boundary condition to follow node (i+1)

Set the already calculated force

ELSE a. Set nearest node’s position to entry/exit triangle’s centroid.

2. Results Our simulation was run on a desktop computer equipped with an Intel core2 quad 2.66 GHz processor, 2.75 GB RAM and NVIDIA® Quadro4 XGL graphics card. Two PHANTOM® OmniTM devices were used to render the reaction forces calculated from the reanalysis technique.


35

The cost in a particular time step was divided among dynamic point update, FTL update and the deformation update through reanalysis. The collision detection and FTL were run in a separate thread and run at a frequency of 295Hz with 25 segments for the suture thread and five segments on the curved needle.

(a)

(b)

Figure 3. Suturing simulator (a) With tool interfaces (b) Snapshot of suturing simulation

3. Conclusion/Discussion We have developed algorithms for detailed needle-tissue and thread-tissue interaction with pre-computed models for laparoscopic suturing procedures. These algorithms can simulate the deformation and forces in real-time with minimal cost. Some of the limitations of the present work include being unable to simulate extended tool-tissue contact and large deformation of tissues, which will constitute our future work.

Acknowledgement This work was supported by grant R01 EB005807 from NIH/NIBIB.

References [1]

[2] [3] [4] [5] [6]

Berkley, J., Turkiyyah, G., Berg, D., Ganter, M., and Weghorst, S. 2004. Real-Time Finite Element Modeling for Surgery Simulation: An Application to Virtual Suturing. IEEE Transactions on Visualization and Computer Graphics 10, 3 (May. 2004) M. Bro-Nielsen, Fast Finite Elements for Surgery Simulation, Studies in Health Technological Information, vol. 39, pp. 395-400, 1997. De, S.; Lim, Y.-J.; Muniyandi, M. & Srinivasan, M. A. Physically Realistic Virtual Surgery Using the Point-Associated Finite Field (PAFF) Approach. Presence, 2006, 15, 294-308. Maciel, A. and De, S. 2008. An efficient dynamic point algorithm for line-based collision detection in real time virtual environments involving haptics. Comput. Animat. Virtual Worlds 19, 2 (May. 2008) Brown, J., Latombe, J.-C., and Montgomery, K. 2004. Real-time knot-tying simulation. The Visual Computer 20, 2-3, 165–179. Sankaranarayanan G, De S. A real-time knot detection algorithm for suturing simulation, Stud Health Technol Inform. 2009; 142: 289-91.

36


Anesthesia Residents’ Preference for Learning Interscalene Brachial Plexus Block (ISBPB): Traditional Winnie’s Technique vs. Ultrasound-Guided Technique Imad T AWADa, Colin SINCLAIRa, Ewen W CHENa, Colin JL MCCARTNEYa, Jeffrey JH CHEUNGa and Adam DUBROWSKIb a Department of Anesthesia, Sunnybrook Health Sciences Centre b Sick Kids Learning Institute, University of Toronto

Abstract. There is a recent shift from traditional nerve stimulation (NS) to ultrasound-guided (UG) techniques in regional anesthesia (RA). This shift prompted educators to readdress the best way to teach these two modalities. Development of a more structured curriculum requires an understanding of student preferences and perceptions. To help in structuring the teaching curriculum of RA, we examined residents’ preferences to the methods of instruction (NS Vs. UG techniques). Novice residents (n=12) were enrolled in this parallel crossover trial. Two groups of 6 residents received a didactic lecture on NS or UG techniques. The groups then crossed over to view the other lecture. After they observed a demo of ISBPB on two patients using NS and US. The residents completed a questionnaire regarding their impression of each technique and the learning experience. UG technique was perceived to be safer and to have more educational value than NS. However, residents felt both techniques should be mandatory in the teaching curriculum. Keywords. Regional anesthesia, teaching curriculum, interscalene block

Introduction The inconsistency of RA teaching in the majority of anesthesia residency programs in North America is due in part to the lack of sufficient clinical exposure [1-3]. As well, the clinical practice of RA in the last six years has undergone transition from traditional NS to UG techniques. This has perhaps diluted the experience residents receive in both traditional landmark and ultrasound imaging techniques. Such transition necessitates a change in our educational models, with an increasing need to develop guidelines and teaching curricula to standardize the practice and teaching of regional anesthesia [4]. In the current study we surveyed novice anesthesiology residents about their preferences of teaching traditional NS and UG methods, as well as their perceptions of the safety and educational value of these two approaches. Understanding the trainees’ needs by assessing their preferences and perceptions is the first necessary step in developing better-structured future educational models.

I.T. Awad et al. / Anesthesia Residents’ Preference for Learning Interscalene Brachial Plexus Block

37

1. Material and Methods With Institutional Ethics Board approval, 12 novice anesthesia residents were recruited in this prospective observational crossover study. The students completed an initial survey to ensure they had no significant experience in either NS or UG regional anesthesia techniques. The residents were then given two 30-minute lectures on interscalene brachial plexus block (ISBPB), one with NS technique and the other with UG technique. Both lectures were standardized in time and content, and were delivered by an expert in the respective technique. Afterward, the residents viewed two real-time demonstration of ISBPB by an expert, one with a NS and the other with US. To avoid an order of training effect (recency effect), the residents were randomized into two groups to counter-balance the viewing order for both lectures and the demonstrations. Residents then completed a questionnaire looking at their understanding of the basic anatomy and ISBPB technique, their preference in technique for future teaching for this block, perceived safety, risk of complications, and educational value of the each technique. The questionnaire was peer-reviewed by a group of five regional anesthetists and a medical psychologist. All reviewers were uninvolved in the development of the original questionnaire. Descriptive statistics were used to summarize the data, with counts and percentages presented for question responses. Analyses were carried out with SAS Version 9.1 (SAS Institute, Cary, North Carolina, USA).

2. Results The initial survey revealed that the residents had minimal experience with regional blocks, in particularly with ISBPB. Residents preferred to have equal emphasis of training in their residency using both traditional NS and UG techniques compared to traditional alone. Residents felt nerve blocks performed under UG would result in fewer complications overall (p10,000 vertices) and large geometric models (>50,000 vertices), the processing times to produce the initial frame can exceed 30 seconds. However subsequent frames are produced at frame rates of 20-30 frames per second.

3. Discussion & Conclusions As illustrated in Figure 1, the SOM adaptation yields a surface that is visually smooth, continuous, and well behaved. Slight errors in registration between the point could and the geometric surface can be accommodated. However, as evident in Figure 2, large registration errors can result in some surface distortion.

Figure 1. SOM adaptation (in light grey) of simple regular surface (in dark grey) to a patch (in white). As seen from the side (on left) and from above (on right)

102

B.M. Cameron et al. / Fast Adaptation of Pre-Operative Patient Specific Models

Figure 2. SOM adaptation showing surface distortion due to large registration errors. Here, the point cloud (white) is not in registration with the original surface (dark grey). The SOM adapted surface (in light grey) shows characteristic distortion. Clockwise from upper left: front view, top view, side view, bottom view.

The transition between those vertices that are displaced by the SOM and those that remain in their original locations is a step function primarily controlled by the initial neighborhood radius. While a small initial radius reduces the time required to compute the adaptation, it also reduces the transition function to a square wave. The transition function can also be affected if there is a large disparity in the sampling frequencies of the initial geometric model and the incoming point cloud. This is illustrated in Figure 3 where a highly detailed pre-operative, patient-specific model is adapted to a sparsely sampled point cloud. The point cloud simulates data segmented from an intra-operative ultrasound image stream. Due to the competitive nature of the SOM, multiple vertices in the geometric model may be mapped to the same point in the point cloud. This phenomenon is most apparent at the transition between the displaced and stationary vertices where the transition takes on a “twisted” appearance. Our initial work shows that by using a pre-processing step to identify those nodes most likely to be displaced during adaptation and to build a map of neighborhood relationships, an SOM can be used to provide fast, near-real time adaptation of preoperative, patient specific models to point clouds which can be derived from intraoperative volumetric data streams. Additional work is needed to reduce the computational time required to perform the pre-processing steps as these steps can result in considerable frame lag when presented with densely sampled data.


103

Figure 3. Patient-specific, pre-operative model of left atrium adapted to simulated intra-operative point cloud data. On left, model (dark grey) and point cloud (white). On right, model after adaptation; altered are shown in light grey.

Acknowledgements The authors would like to thank Mr. Jon Camp, Mr. Liu Jiquan, Dr. Douglas Packer, and the staff of the Biomedical Imaging Resource for their contributions to this work.

References [1]

Lin, D. and F.E. Marchlinski, Advances in ablation therapy for complex arrhythmias: atrial fibrillation and ventricular tachycardia. Curr Cardiol Rep, 2003. 5(5): p. 407-14. [2]. Mansour, M., G. Holmvang, and J. Ruskin, Role of imaging techniques in preparation for catheter ablation of atrial fibrillation. J Cardiovasc Electrophysiol, 2004. 15(9): p. 1107-8. [3] Reddy, V.Y., et al., Integration of cardiac magnetic resonance imaging with three-dimensional electroanatomic mapping to guide left ventricular catheter manipulation: feasibility in a porcine model of healed myocardial infarction. J Am Coll Cardiol, 2004. 44(11): p. 2202-13. [4] Rettmann, M.E., et al., An event-driven distributed processing architecture for image-guided cardiac ablation therapy. Comput Methods Programs Biomed, 2009. 95(2): p. 95-104. [5] Rettmann, M.E., et al., An integrated system for real-time image guided cardiac catheter ablation. Stud Health Technol Inform, 2006. 119: p. 455-60. [6] Rettmann ME, H.I.D., Dalegrave C, Johnson SB, Camp JJ, Cameron BM, Packer DL, Robb RA., Integration of patient-specific left atrial models for guidance in cardiac catheter ablation procedures. Medical Image Computing and Computer-Assisted Interventions (MICCAI) workshop on Image Guidance and Computer Assistance for Soft-Tissue Interventions, 2008 Sep. [7] Rettmann ME, H.I.D., Cameron BM, Robb RA., Interactive Visualization of Cardiac Anatomy in Catheter-Based Ablation Therapy. Medical Image Computing and Computer-Assisted Interventions (MICCAI) workshop on Interaction in Medical Image Analysis and Visualization, 2007 Nov.

104

[8]


Kohonen, T., Self-organized formation of topologically correct feature maps. Biological Cybernetics, 2001. 43: p. 59-69. [9] Kaski, S., Data exploration using self-organizing maps. Acta Polytechnica Scandinavica, Mathematics, Computing and Management in Engineering Series No. 82, 1997: p. 57 pp. [10] Jiquan L., Rettmann M.E., Holmes D. R. III, Huilong D., Robb R. A., A Piecewise Patch-to-Model Matching Method for Image-guided Cardiac Catheter Ablation. Computerized Medical Imaging and Graphics, (submitted).


105

Realistic Visualization of Living Brain Tissue Llyr ap CENYDDa, Annette WALTERb, Nigel W. JOHNa,1, Marina BLOJ b and Nicholas PHILLIPSc a School of Computer Science, Bangor University, UK b Centre for Visual Computing, University of Bradford, UK c Department of Neurosurgery, Leeds General Infirmary, UK Abstract. This paper presents an advanced method of visualizing the surface appearance of living brain tissue. We have been granted access to the operating theatre during neurosurgical procedures to obtain colour data via calibrated photography of exposed brain tissue. The specular reflectivity of the brain’s surface is approximated by analyzing a gelatine layer applied to animal flesh. This provides data for a bidirectional reflectance distribution function (BRDF) that is then used the rendering process. Rendering is achieved in realtime by utilizing the GPU, and includes support for ambient occlusion, advanced texturing, sub surface scattering and specularity. Our goal is to investigate whether realistic visualizations of living anatomy can be produced and so provide added value to anatomy education. Keywords. Rendering, BRDF, Brain anatomy

Introduction Cadaver dissection is widely accepted as being the ‘gold standard’ for anatomy education. It provides the learner with knowledge of the shape and size of the organs; it gives them an appreciation of spatial relationships between organs; and also introduces students to death in a controlled manner. However, financial and ethical pressures have led to a decrease in the availability and usage of this approach [1]. This has led to anatomy today being taught in a variety of different ways, including prosections, problem-based learning scenarios, and computer graphics based systems such as those using data derived from the Visible Human and similar projects. All of these methods offer potential benefits to the learner. However, one weakness with computer generated anatomy models is that they are rendered as grey scale or use pseudo colour that at best is a gross approximation of the true appearance of healthy living organs. Even cadaveric specimens are very different in appearance from when they were still live tissue. Our hypothesis is that more realistic models of internal human anatomy will improve anatomy education, preparing the novice surgeon or nurse to more easily recognize anatomy the first time they are exposed to an operation on a real patient. This paper presents the results of a multi-disciplinary feasibility study that has been carried out to investigate the above hypothesis using the human brain as the target organ. We have been granted access to the operating theatre during neurosurgical procedures to obtain colour data from calibrated photography of exposed brain tissue. The specular reflectivity of the brain’s surface is approximated by analyzing a gelatine 1

Corresponding Author: Nigel W. John, University of Bangor, UK; E-mail: [email protected]

106

L. ap Cenydd et al. / Realistic Visualization of Living Brain Tissue

layer applied to animal flesh. This provides data for a bidirectional reflectance distribution function (BRDF) that is then used to help generate a realistic model of living brain tissue. The human skin has been extensively modelled using BRDF models e.g. [2], [3]. The only previous example that we have discovered for rendering internal anatomy with the use of a BRDF is within a virtual bronchoscopy application [4]. This solution exploits the particular restrictions of image acquisition from a bronchoscope, however, and is not applicable to recreating anatomy models from the wide range of viewing directions and illuminations required for general use.

1. Methods and Materials A BRDF is a function that defines how light is reflected at an opaque surface [5]. The parameter data for the BRDF can be recovered from calibrated photographs and light sources [6]. We use the Lafortune model [7] to render colour accurate computergenerated images: 9:

&(', )) = +- + ∑40123,4 5'2 )2 + '3 )3 6 + 17,4 '7 )7 8

(1)

where ρd, Cxy,i, Cz,i, and Ni are the fitted parameters (sum over i is the sum of each lobe, specular and retro-reflective in this case), (ux, uy, uz) and (vx, vy, vz) are the incident and excitant light vectors respectively.

Figure 1. Photograph of Exposed Brain Surface before and after colour shift calibration (image shown is after a non-calibrated print process and/or screen display).

1.1. Colour Data Colour samples of living brain tissue have been collected from photographs taken with patient consent during neurosurgery procedures at the Leeds General Infirmary. We used a Canon EOS 5D digital camera with a 50mm lens set on manual mode and with auto focus. We selected the ISO 200 setting under a colour temperature of 6500K. A small white reference card was held by the surgeon near to the region of interest (i.e. an area of exposed brain). This enables the images to be later recalibrated to compensate for the colour shift caused by the illumination and so allow the RGB values of brain tissue, blood vessels and cancer tissue to be calculated independently of the used light source. The pictures were acquired in raw format to avoid automatic recalibration by


107

the camera. Figure 1 is an example of the photographs taken, showing a small area of the exposed brain surface of the patient. 1.2. Reflectance During neurosurgery the brain is constantly drenched with sterilized water, which results in a large uniform specular reflection on its surface. There are several parameters that can affect this specular effect e.g. position of light sources and the camera, distance between camera and region of interest, and size of the region of interest. However, the time made available to us in the operating theatre to take photographs and the inherent restrictions of working in this environment meant it was not possible to collect enough data to fully investigate these parameters in vivo. To ensure we remained unobtrusive to the surgeon and posed no additional risk to the patient we were able to collect a maximum of five photographs using a single fixed light source position. An in vitro experiment was also designed using animal flesh (a lamb steak) as a possible brain substitute for generating the reflectance data. A layer of gelatine was applied to the surface of the meat (just adding water to the surface provided insufficient specularity as the water evaporated too quickly). This model does produce a slightly higher reflectance profile than in the in vivo scenario. However, a diffuse light source is required for a BRDF model and this can be used in this experiment. The strong light source used in the operating theatre focuses the radiation and so gives higher reflectance behaviour than is desirable for our purposes.

Figure 2. Measurements from two illumination positions at 75° and 65° from the surface normal. In each case the five camera positions are indicated by the coloured circles. The curves are fitted to the rgb pixel values of the region of interest, which represents the spatial surface reflectance. The distance from the centre shows the relative intensity of the pixel values. An ideal Lambertion reflectance profile would be represented by a radial curve (semi-circle). Location of specular components are indicated by an increase in intensity.

The Lafortune BRDF parameters in the in vitro experiment can be calculated based on images corresponding to two illuminated locations and five camera locations - see Figure 2. Observe that the peak of the specular reflection is shifted by 10 degrees; 15 degrees would be theoretically expected for a plane surface. The curvature of the gelatine layer at the region of interest is not known. The intensity for all RGB components, under both light positions, is similar but not identical after taking into account the shift of the intensity peak. This indicates an insignificant change of colour for the meat surface due to different light positions. To reach our aim of a realistic visualization of living brain tissue this small position error of the specular reflectance peak will not change the visual fidelity of the rendered surface. It can be ignored

108


because a shift of 5 degrees in reflectance is not discernable for a complex structure like the brain. 1.3. Rendering the Brain The brain surface mesh used in this project has been created in a 3D modelling package. A mesh segmented from volume data did not provide enough accuracy for depicting the folds (gyri) and grooves (sulci) that are abundant across the cerebral surface. Our brain rendering algorithm is implemented on the GPU using the GLSL 2.0 shading language, programmed in ATI's RenderMonkey development environment. In the first pass, diffuse lighting is calculated and a procedural detail texture is generated through a process of multi-texturing, using tile-able vein and corresponding bump-map textures. The brain is then rendered into an off-screen texture, where a series of separable blur passes are performed to produce irradiance textures, simulating subsurface scattering. Finally, in the last pass a final diffuse texture is produced by blending all generated irradiance textures, and the Lafortune BRDF model is used to calculate specular and retro reflection of light. The sum of diffuse, ambient occlusion map and specular gives the final pixel colour. Ambient Occlusion. Ambient occlusion adds realism to local reflection models by taking into account attenuation due to occlusion. It allows for better perception of the 3D shape of objects by enhancing lighting contrast. We pre-compute ambient occlusion maps to highlight the deep sulcus (fissures) of the brain's surface. Texturing the Brain Surface. The surface membrane of the human brain contains a network of blood vessels of various sizes and structure. Our algorithm automatically constructs the micro diffuse detail and topology of the brain's surface by sampling a series of high resolution vessel textures at different scales, colours and strengths. Each tileable vessel texture is created by modeling a representative section of the brain's surface from reference photographs, and baking the geometry into a bump map. The vessel textures are layered to construct a complex pseudo-anatomically correct topology. Each instance is parameterized with scale, colour and bump-strength constants. Creating the diffuse texture through parameterized multi-texturing allows for the generation of an extremely detailed surface diffuse texture and topology. Sub Surface Scattering. The human brain is translucent and so some light waves will penetrate the surface, be scattered, partially absorbed and then exit at a different location. The sub-surface scattering algorithm used in our current rendering pipeline is based on a technique introduced in [8], which takes advantage of the fact that the transport of light though skin and similarly transparent materials (like the brain) are local enough in nature that scattering can be approximated through a process of texturespace diffusion [9]. A diffusion profile describes the manner by which light scatters through the surface of a highly scattering translucent material. By calculating a diffusion patch for every region across a polygonal surface, the aggregate sum of overlapping patches gives a realistic translucent appearance. In texture-space diffusion, diffusion profiles are created by summing multiple Gaussian functions together to create a convolution filter describing how light spreads to neighbouring locations. An initial pass in our algorithm calculates the diffuse colour and per-pixel lighting across the brain surface in texture space, based on the generated detail texture's diffuse and normal map components respectively. This yields a diffuse map of the brain's surface, which is rendered to an off-screen texture using the texture coordinates as position - effectively unwrapping the 3D mesh onto a 2D plane. Our current


109

implementation represents diffusion profiles as a sum of up to six Gaussian functions, with each Gaussian weighted separately for red, green and blue wave-lengths. The weighted sum of the three bands for each Gaussian is normalized so that no colour shift occurs during convolution, as the base diffuse colour has already been deduced in the first pass. A multilayer diffusion profile is necessary for approximating scattering though brain tissue. Specularity. The Lafortune BRDF model is used to provide the specularity of the brain’s surface. From our measured data, for each waveband (RGB), and each incident light angle, there are parameters fitted for each component of this model, describing a two lobe representation of surface reflectance: • • •

Lambertian/Diffuse component. Specular-reflective xy and z components, plus exponent. Retro-reflective xy and z components, plus exponent.

The Final Colour. After calculating the diffuse lighting, irradiance textures and specular reflectance, the final pass acts to process and combine the components. The brain mesh is rendered in 3D using a standard vertex shader. The different blurred textures are linearly combined to compute the total diffuse light exiting the surface, with each texture being multiplied by its corresponding Gaussian weight and renormalized to ensure white diffuse light. The contribution of diffuse, specular and ambient occlusion maps are combined to give the final pixel colour.

2. Results Figure 3 demonstrates the effect of specular highlighting rendered with our Lafortune BRDF model. The results indicate that the system can visually reproduce the reflective properties of tissue from real world measurements. The brain is a very complicated surface with many folds and fissures, properties which make it very difficult to generate a map of the entire surface free of excessive distortion or overlap. Due to the prevalence of texture-space computation, we currently concentrate on rendering a representative section of the human brain. One potential solution to this problem is to use cerebellar flat mapping techniques [10], which use a circle packing algorithm to unfold the complicated gyri and sulci into a simpler map of the cortical surface. Implementing a variation of this technique would also facilitate rendering segmented meshes from volume data. The excessive curvature of the brain's topology also leads to distortion, as distances on the diffuse texture do not correspond directly to distances on the mesh itself. We also plan to compute stretch-correction textures to correct this issue.

Figure 3. The brain surface rendered without (left) and with (right) the in vitro experiment BRDF data applied (75 degrees, 5 camera locations; 60 degrees, 5 camera locations)

110


Sub-surface scattering through texture-space diffusion is a powerful technique that can achieve very realistic results. Figure 4 shows a section of the brain mesh rendered using our Lafortune BRDF model, combined with sub-surface scattering calculations from a 3¬Gaussian diffuse profile. While we currently use hand-crafted Gaussian variance and weighting values, we are in the process of fully parameterizing the system to facilitate further experimentation.

Figure 4. Left: Section of brain rendered with 3 Gaussian sub-surface scattering (8 passes). Right: Detail on brain's surface, created with one vessel texture at differing tile scales.

Figure 4 also gives an example of the surface detail generated by our texture generation algorithm, with a highly specular BRDF designed to replicate water on the surface. Generating a realistic diffuse texture is largely dependent on the quality and accuracy of the underlying vessel texture maps, which we continue to improve. Finally Figure 5 provides a pictorial comparison between a photograph of the exposed brain surface of a patient in the operating theatre with synthetically generated results of our approach. In the latter case, the diffuse colour has been obtained from the calibrated operating theatre photographs, and the reflectance parameters from the in vitro meat with gelatine layer BRDF (75 degrees, 5 camera locations; 60 degrees, 5 camera locations). A second BRDF is mixed in (Cxy = -1, Cz = 1; essentially Phong specularity, direct reflection) to simulate the sharp glistening from the water layer.

Figure 5. Photograph from operating theatre (left); Rendered brain surface (right)

3. Conclusions We have presented an advanced method of visualizing the surface appearance of living brain tissue. We have been granted access to the operating theatre during neurosurgical procedures to obtain colour data from calibrated photography of exposed brain tissue. The specular reflectivity of the brain’s surface is approximated by analyzing a gelatine


111

layer applied to animal flesh. This provides data for a bidirectional reflectance distribution function (BRDF) that is then used the rendering process. Rendering is achieved in realtime by utilizing the GPU, and includes support for ambient occlusion, advanced texturing, sub surface scattering and specularity. The constraints of acquiring all of the data required in an operating theatre environment have prevented a full BDRF model being calculated purely from patient data. However, the reflectance data obtained from lamb/gelatine material has proven to be an adequate alternative that provides specular reflectance that is close to that of living brain tissue. Other anatomy can also be depicted using the techniques described above and in many cases it will be easier to acquire the BRDF data – more surface area of organs is exposed in abdominal surgery, for example, and the organs have far smoother surfaces than the brain, and are not covered with a membrane. In the future we expect to create a library of realistic models of virtual organs that look vibrant and alive. We plan further improvements to the rendering pipeline. A future extension will allow the texture generation process to take into account the properties of the brain's topology when calculating the vein size, colour and opacity. One possible solution is to use the pre-computed ambient occlusion texture, which effectively maps the location and depth of fissures, where vessels tend to be more visible and numerous. Our clinical collaborators have commented favourably on the first results that we have obtained and presented in this paper. The final stage of the current project will be to evaluate our models with medical educators and students. We have repeated the experiment at Keele University School of Anatomy to obtain data from cadaveric brain specimens and so generate equivalent cadaveric brain models to act as a comparison. The validation study is currently under way and will be reported in a follow up paper.

References [1] MCLACHLAN J.C., BLIGH J., BRADLEY P., SEARLE J.: Teaching anatomy without cadavers. Medical Education 38, 4 (2004), 418–424 [2] DEBEVEC P., HAWKINS T., TCHOU C., DUIKER H.-P., SAROKIN W., SAGAR M.: Acquiring the reflectance field of a human face. In ACM SIGGRAPH ’00 Conference Proceedings (2000), 145–156 [3] DONNER C., JENSEN H.W.: A Spectral BSSRDF for Shading Human Skin. In Proc. Eurographics Symposium on Rendering, (2006) 409-427 [4] CHUNG A.J., DELIGIANNI F., SHAH P., WELLS A., YANG G-Z: Patient-specific bronchoscopy visualization through BRDF estimation and disocclusion correction. IEEE Trans on Medical Imaging 25, 4 (2006), 503 – 513 [5] NICODEMUS, F.: Directional reflectance and emissivity of an opaque surface. Applied Optics 4, 7 (1965), 767–775. [6] DEBEVEC P., REINHARD E., WARD G., PATTANAIK S.: High dynamic range imaging. In ACM SIGGRAPH 2004 Courses (2004) [7] LAFORTUNE E.P.F., FOO S-C., TORRANCE K.E., GREENBERG D.E.: Non-linear approximation of reflectance functions. In ACM SIGGRAPH 97 Conference Proceedings, (1997) 117-126 [8] BORSHUKOV, G., LEWIS, J. P: Realistic Human Face Rendering for The Matrix Reloaded. In ACM SIGGRAPH 2005 Courses (2005) [9] D’EON E., LUEBKE D.: Chapter 14. Advanced Techniques for Realistic Real-Time Skin Rendering. In GPU Gems 3 (Nguyen H., Editor). Addison-Wesley, 2007. [10] HURDAL M. K., LEE A., RATNANATHER T., NISHINO M., MILLER M., BOTTERTON K.: Investigating the Medial Prefrontal Cortex with Cortical Flat Mappings. NeuroImage, 19, 2 (2003)

112


A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy Sonny CHAN a , Peter LI b , Dong Hoon LEE b , J. Kenneth SALISBURY a and Nikolas H. BLEVINS b,1 a Department of Computer Science, Stanford University b Department of Otolaryngology, Stanford University Abstract. This article presents a virtual surgical environment whose purpose is to assist the surgeon in preparation for individual cases. The system constructs interactive anatomical models from patient-specific, multi-modal preoperative image data, and incorporates new methods for visually and haptically rendering the volumetric data. Evaluation of the system’s ability to replicate temporal bone dissections for tympanomastoidectomy, using intraoperative video of the same patients as guides, showed strong correlations between virtual and intraoperative anatomy. The result is a portable and cost-effective tool that may prove highly beneficial for the purposes of surgical planning and rehearsal. Keywords. Surgical simulation, surgical rehearsal, haptic rendering, volume rendering, patient-specifc models, temporal bone surgery

1. Introduction Given the current availability of high-resolution three-dimensional medical imaging, surgeons commonly have access to multimodal anatomic data prior to undertaking a surgical procedure. Imaging studies such as computed tomography (CT) and magnetic resonance imaging (MRI) can offer accurate information on tissue composition and geometry, and are often used together given their complementary strengths. However, even after structures are identified on imaging, the surgeon must be able synthesize these data into a conceptual model that can predict what will be encountered intraoperatively. To achieve this, the surgeon often needs to take a number of steps in preparing for surgery: 1. Mentally co-register volumetric data from different modalities, so that the studies can be combined effectively to take advantage of the best aspects of each. 2. Formulate an integrated 3-D representation of the patient from the studies, so that anatomic relationships are understood from a variety of potential viewpoints. 3. Create a mental image that predicts how surgical manipulation and removal of tissues will affect subsequent access and exposure. Accomplishing these steps can be a challenge, especially if the studies are examined as sequential two-dimensional slices, as is the current practice. Despite the use of multi1 Corresponding Author: Chief, Division of Otology/Neurotology, Department of Otolaryngology – Head and Neck Surgery, Stanford University, 801 Welch Road, Stanford, CA; E-mail: [email protected].

S. Chan et al. / A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy

113

planar reconstructions, critical spatial relationships need to be inferred rather than seen directly as occurs in actual surgery. We have developed a virtual surgical environment intended to facilitate these steps, and optimize the benefits of available imaging. Our goal has been to create an environment that can relatively quickly incorporate routine clinical studies, enabling real-time interactive preoperative assessment. Our approach thus far has focused on procedures involving the resection of cholesteatomas (skin cysts) from the middle ear and mastoid, collectively known as tympanomastoidectomy. Such procedures involve the removal of portions of the temporal bone to gain access to these cysts, which are commonly associated with chronic ear infections. The ability to experiment with varied approaches may prove beneficial to the outcome of the procedure. Traditional imaging of the temporal bone prior to tympanomastoidectomy involves the use of high-resolution CT. This demonstrates bone and air quite well, and therefore shows other adjacent structures by the absence of either of these. It does not, however, differentiate between various types of soft tissue, such as scar or fluid, from the more surgically relevant cholesteatoma. Recently, diffusion-weighted MRI sequences have been used successfully to aid in the identification and localization of cholesteatoma [5]. We obtained MR imaging on a series of patients with chronic ear infections, and integrated its specific identification of cholesteatoma into the bony framework from the CT images. A number of virtual environments for simulation of temporal bone surgery have been developed [2]. Wiet et al. describe dissection of a virtual temporal bone derived from CT images of a cadaveric specimen in an early work demonstrating a surgery simulator incorporating both visual and haptic feedback [12]. Agus et al. present a similar surgical training system where volumetric object models directly derived from CT and MRI data can be dissected [1]. They simulate a virtual surgical drill using a realistic, physically based model for haptic force feedback and bone tissue removal. Morris et al. present a comprehensive simulation environment for both training and performance evaluation of bone surgery [7]. They discuss techniques for automated evaluation and feedback, allowing the possibility of using the simulator for surgical skills assessment. In a recent study, Tolsdorff and colleagues have proposed the use of individual patient models derived from CT image data for virtual mastoid surgery [11], though the study was conducted using a training-oriented simulator [8], and only allowed for the import of bone tissue. The majority of surgical simulation work to date has focused on surgical education, training, and assessment using standardized libraries of anatomic models. In contrast, the system described here is intended as a step towards surgical rehearsal, with which a surgeon can prepare for an upcoming case by practicing dissections on a virtual representation of the patient’s specific anatomy. To do this, it makes direct use of multi-modal preoperative imaging data with minimal need for preprocessing. It also incorporates new and efficient methods to render multiple volumetric data sets visually and haptically to enable interaction with the virtual anatomy in a manner familiar to surgeons.

2. Mathods & Materials Preoperative image data from a growing library of 8 patients, each of whom was a candidate for tympanomastoidectomy, was collected for evaluation. Imaging for each patient consisted of a clinical CT scan of the temporal bone (Figure 1a) and two MR images: a T2-weighted FIESTA sequence and a diffusion-weighted PROPELLER sequence (Fig-

114


(a)

(b)

(c)

Figure 1. A preoperative axial clinical CT scan of the temporal bone (a). Soft tissue is seen filling the middle ear (arrow). The corresponding slice of the MR PROPELLER sequence shows a hyperintense region indicative of cholesteatoma (b). A close-up of the cholesteatoma registered and superimposed on the CT (c).

ure 1b). Conventional CT and MR imaging cannot easily identify cholesteatoma within the temporal bone, but diffusion-weighted MR imaging shows potential as the modality of choice for this purpose (Figure 1c). These images contain complementary information, and are used collectively to create a virtual model of the patient’s anatomy. 2.1. Data Processing Registration of the temporal bone CT and the MRI sequences was performed using Amira 5.3 (Visage Imaging Inc., San Diego, CA). The DICOM images were imported into Amira and registered in three dimensions using the automated registration tool with a mutual information metric and the conjugate gradient optimization algorithm. Anatomic structures of interest were extracted from the different imaging datasets using Amira’s computer-assisted segmentation tools. The CT images were used to segment the semicircular canals, facial nerve and ossicles. The FIESTA image sequence was used to segment the carotid artery and sigmoid sinus, and the PROPELLER image sequence was used to segment cholesteatoma. Segmentations were exported both as label volumes and as Wavefront object meshes to be used in our virtual surgical environment. Data processing, including segmentation of all vital structures and smoothing of the resulting models, took approximately two hours for each patient. Not all processing steps are necessary in every case though: a clinical CT can be used directly by our virtual surgical environment for visualization and dissection of bone without any preprocessing. 2.2. Volume Rendering Our system uses three-dimensional preoperative image data as the principal representation of the virtual patient. As this model consists primarily of volumetric data, it is natural to adopt a real-time volume rendering approach for data visualization within the simulation. We have developed a multi-modal volume rendering method based on a GPUaccelerated ray casting approach [9] that is capable of simultaneously displaying the different forms of data supplied to the virtual surgical environment. We simultaneously display a clinical CT volume (with both isosurface and direct volume rendering), a segmentation volume, and a dissection mask volume in a single rendering pass. Each data volume is represented as a separate 3D texture on the graphics processor, but is co-located in texture coordinate space with the primary (CT) volume. This allows the rendering algorithm to perform ray casting in lock step through all data


(a)

(b)

115

(c)

Figure 2. A slice of a combined and dilated segmentation volume showing the cholesteatoma, cochlea and semi-circular canals, and carotid artery in different colors over the CT scan (a), and the low-pass filtered union of the structures for volumetric isosurface rendering (b). Note the cholesteatoma in (a) is a result of dilation from an adjacent slice, and is absent in (b). The black area in a close-up of a mask volume shows the smooth edge of a region removed by a spherical burr (c).

volumes simultaneously. A ray is cast for each pixel in the rendered image from the viewpoint in virtual space through the volume data, accumulating color and opacity information that determines its final appearance. A shader program samples all volumes along the ray at a regular spatial interval, taking advantage of the highly parallel computational architecture of modern GPUs to achieve interactive visualization. The preoperative clinical CT serves as the primary volume, and is represented at its native resolution as a 16-bit single-channel texture in video memory. The rendering algorithm traverses the ray until it first encounters a sample value greater than the isosurface value. Several interval bisection steps [3] are then performed to refine the rayisosurface intersection point, and the surface is shaded with configurable material properties. If the surface is partially or fully transparent, the ray is continued, accumulating color and opacity through the interior of the volume in a front-to-back composition using a pre-integrated color transfer function [9], until the exiting isosurface is found. Thus, the appearance of the primary volume is controlled by an isosurface value (in CT Hounsfield Units) with material properties and a user-defined transfer function that maps Hounsfield Units to optical properties, both of which can be specified interactively. In our data, we have found that an isosurface provides a good indication of the tissue-air boundary, and direct volume rendering generates a realistic visualization of the bone tissue (Figure 3c). A segmentation or label field for an anatomical structure consists of a binary volume which indicates inclusion of contained voxels into the structure. We apply a preprocessing step that combines all non-overlapping label volumes into a two-channel, 8-bit luminance-alpha texture to be rendered volumetrically (Figure 2a, 2b). The alpha channel contains a union of the segmentations, and is low-pass filtered to facilitate isosurface rendering without stepping artifacts [4]. The luminance channel contains a unique index for each constituent structure that maps to a set of material properties, and a dilation operation is performed on the final image to ensure that texture sampling retrieves the correct index. The ray-casting algorithm samples this volume in lock step with the primary volume, and if the sample is found to lie within a segmented structure, material properties are retrieved from an auxiliary texture to perform shading (Figure 3b). Finally, the mask volume, which controls visibility of the model, is an 8-bit, singlechannel representation that can represent smooth surfaces to a sub-voxel resolution (Figure 2c). During ray casting, both the primary and segmentation volumes are modulated

116


(a)

(b)

(c)

Figure 3. Temporal bone anatomy, including the sigmoid sinus, semicircular canals, facial nerve, ossicles, carotid artery, and a cholesteatoma lesion rendered as polygonal meshes (a), the larger structures rendered as volumetric isosurfaces with a semi-transparent bone surface in front (b), and direct volume rendering of the bone in a full composite, ready for surgery (c).

by the mask value. Thus, editing of the volume data can be accomplished by attenuation or zeroing of the mask volume. In addition to the volumetric data, our virtual surgical environment uses polygonal models to represent surgical instruments and certain segmented structures. Anatomical structures with details near to or finer than the native voxel resolution of the data may be better represented as polygonal meshes (Figure 3a). Examples of these structures include the facial nerve and the ossicles. However, the rendering of polygonal meshes is not affected by changes in the mask volume described earlier, and thus any structure that permits dissection should be represented as part of the segmentation volume. 2.3. Haptic Rendering The primary representation of the patient’s anatomy in our virtual surgical environment is in volumetric form, and thus a method for haptic interaction with volume data is required. McNeely et al. proposed a popular algorithm for haptic rendering of volumetric geometry [6], and several variants have been described for use in surgical simulators [1,7,8]. However, these methods do not prevent tool-tissue interpenetrations in the simulation environment, and can suffer from the virtual instrument passing completely through a thin layer of tissue (Figure 4a), especially when using commercially available haptic devices with lower force output and limited stiffness capabilities. Proxy-based rendering algorithms constrain the virtual instrument to the surface of the geometry, preventing pop-through problems (Figure 4b). Salisbury & Tarr have described an algorithm for proxy-based haptic rendering of implicit surfaces [10] that can readily be adapted to render isosurface geometry embedded within volumetric data. A volume can be treated as a discrete sampling of a scalar field, and can be sampled at arbitrary positions through interpolation. Rather than evaluating an analytical gradient, the surface normal can be estimated using a central difference. The primary limitation here is that a tool can only be modeled as a single point for interaction with the volume. A virtual surgical drill allows dissection of the anatomy in our virtual surgical environment. We model the spherical burr of the drill by extending the proxy-based, point interaction algorithm to incorporate elements of the method described by [7]. The burr is discretized into individual points in three-dimensional space that occupy the volume of the sphere. During interaction, the volume is sampled at these points, and any point found to lie within the surface exerts a fixed amount of force toward the center of the


117

F F F

(a)

(b)

(c)

Figure 4. With a point sampling algorithm, the contact force can be in the wrong direction when the instrument is pushed into a thin object (a). A proxy-based algorithm can constrain an interaction point to the surface (b). We combine these algorithms to prevent the virtual instrument from popping through thin layers of bone (c).

burr. In addition, we treat the center of the burr as a proxy point, so that if the haptic device moves in a way that this point penetrates into an object, the proxy-based algorithm constrains the center of the virtual instrument to lie on the surface of the object. A virtual spring exerts additional force proportional to the displacement between the device position and the proxy point. The end result is that superficial interaction forces are computed primarily from point sampling, whereas the virtual spring force dominates during haptic exploration involving deep tool-tissue interpenetration (Figure 4c). Tissue resection is modeled by attenuating or removing voxels from the mask volume in a manner similar to that described in [8]. When the virtual drill is on, an antialiased voxelization [4] of the spherical burr is subtracted from the mask volume, preserving a smooth, accurate cut surface (Figure 2c). This technique allows modeling of tissue dissection at a sub-voxel resolution, and prevents the jagged or stepped appearance that normally results from voxel-resolution modification of volume data. 3. Results & Conclusions Images from the use of our virtual surgical environment on data from two selected patients are shown in Figure 5. We have been able to replicate salient anatomic detail in the virtual environment as compared to the video images taken during actual tympanomastoidectomy. The geometry from the CT dataset yields a subjectively accurate representation of the bony contours seen during surgery. Similarly, the cholesteatoma volume derived from PROPELLER MR imaging is accurately placed within the bone, and presents a realistic representation of what the otologic surgeon will encounter in the patient. By rendering the bone transparent, other segmented vital structures can be seen in their familiar relative locations. Our preliminary subjective experience suggests that our virtual surgical environment can offer an accurate and interactive representation of patient-specific anatomy. Our system represents a step towards the use of a virtual environment to prepare for tympanomastoid surgery. It enables the relatively rapid integration of multi-modal imaging datasets, direct volume rendering, and a means of manipulating preoperative clinical data in a surgically relevant manner. We anticipate that the methods described can be generalized to a variety of surgical procedures. Clearly tools such as this require objective validation to ensure that they can benefit a surgeon in preparing for an operative case. We intend to carry out such studies in the future as our system becomes further refined. We are encouraged by the assumption that the more a surgeon is familiar with working in and around specific anatomy, the more he or she is likely to be effective. Offering surgeons such an opportunity holds great potential.

118


(a)

(b)

(c)

(d)

Figure 5. An intraoperative video capture during right tympanomastoidectomy (a) and the corresponding image from a virtual dissection (b) using that patient’s preoperative imaging data. The cholesteatoma (white) was automatically segmented from MR imaging, and has been exposed following removal of overlying bone. Images (c) and (d) demonstrate similar images from a different patient during left tympanomastoidectomy. The size and extent of cholesteatoma in each case was accurately predicted and superimposed onto the bone.

References [1] [2] [3] [4] [5]

[6] [7] [8] [9] [10] [11] [12]

M. Agus, A. Giachetti, E. Gobbetti, G. Zanetti, and A. Zorcolo, Real-time haptic and visual simulation of bone dissection, Presence 12 (2003), 110–122. M.P. Fried, J.I. Uribe, and B. Sadoughi, The role of virtual reality in surgical training in otorhinolaryngology, Current Opinion in Otolaryngology & Head and Neck Surgery 15 (2007), 163–169. M. Hadwiger, C. Sigg, H. Scharsach, K. Buhler, and M. Gross, Real-time ray-casting and advanced shading of discrete isosurfaces, Computer Graphics Forum 24 (2005), 303–312. S. Lakare and A. Kaufmany, Anti-aliased volume extraction, Proceedings of the Symposium on Data Visualization (2003), 113–122. P. Lehmann, G. Saliou, C. Brochart, C. Page, B. Deschepper, J.N. Vallée, and H. Deramond, 3T MR imaging of postoperative recurrent middle ear cholesteatomas, American Journal of Neuroradiology 30 (2009), 423–427. W.A. McNeely, K.D. Puterbaugh, and J.J. Troy, Six degree-of-freedom haptic rendering using voxel sampling, Proceedings of SIGGRAPH (1999), 401–408. D. Morris, C. Sewell, F. Barbagli, K. Salisbury, N.H. Blevins, and S. Girod, Visuohaptic simulation of bone surgery for training and evaluation, IEEE Computer Graphics and Applications 26 (2006), 48–57. B. Pflesser, A. Petersik, U. Tiede, K.H. Höhne, and R. Leuwer, Volume cutting for virtual petrous bone surgery, Computer Aided Surgery 7 (2002), 74–83. S. Roettger, S. Guthe, D. Weiskopf, T. Ertl, and W. Straßer, Smart hardware-accelerated volume rendering, Proceedings of the Symposium on Data Visualization (2003), 231–238. K. Salisbury and C. Tarr, Haptic rendering of surfaces defined by implicit functions, ASME Dynamic Systems and Control Division (1997), 61–67. B. Tolsdorff, A. Petersik, B. Pflesser, a. Pommert, U. Tiede, R. Leuwer, and K.H. Höhne, Individual models for virtual bone drilling in mastoid surgery, Computer Aided Surgery 14 (2009), 21–27. G.J. Wiet, D. Stredney, D. Sessanna, J.A. Bryan, D.B. Welling, and P. Schmalbrock, Virtual temporal bone dissection: An interactive surgical simulator, Otolaryngology–Head & Neck Surgery, 127 (2002), 79–83.


119

Acquisition of Technical Skills in Ultrasound-Guided Regional Anesthesia Using a High-Fidelity Simulator Jeffrey JH CHEUNGa, Ewen W CHENa, Yaseen AL-ALLAQa, Nasim NIKRAVANa, Colin JL MCCARTNEYa, Adam DUBROWSKIb, Imad T AWADa a Department of Anesthesia, Sunnybrook Health Sciences Centre b Sick Kids Learning Institute, University of Toronto

Abstract. Despite the increasing popularity of ultrasound-guided regional anesthesia (UGRA), structured training programs during residency are often lacking. The lack of a regional block area, lack of expertise, and lack of structured training programs have limited hands-on experience in residency programs. However, these constraints may be circumvented through the use of simulation. This observational study looked at the use of a high-fidelity simulator for training novice undergraduate students UGRA techniques. Despite some improvement in the second trial with the simulator, the ability to maintain visualization of their needle (p 0.5, < 1mm), yellow for intermediate agreement ( 2mm), orange for poor agreement (>2, < 3mm) and red for very poor agreement (> 3mm).

322

V. Luboz et al. / Guidewire and Catheter Behavioural Simulation

Table 1. Comparison between the real and virtual instruments based on the distance between corresponding pairs of points. It shows the average distance, the standard deviation and the minimum and maximum distances between the two instruments. All distances are in mm. kb tip

kb body

Average distance)

Standard deviation

Minimum distance

Maximum distance

Terumo angled

0.8

0.8

2.21

1.3

0.43

5.59

Terumo stiff

0.8

1.1

2.63

1.91

0.54

7.45

Amplatz

0.8

1.2

2.43

1.43

0.56

5.48

Bentson

0.7

1

2.88

2.02

0.59

6.68

Cook J

0.9

1

2.05

1.18

0.4

4.93

Cook Straight

0.9

1

2.24

1.48

0.24

5.92

Rosen

1

1.05

2.43

2.37

0.33

7.29

Beacon 5Fr

0.9

1

1.75

1.39

0.46

4.11

Terumo ST 4Fr

0.9

1

2.09

1.12

0.52

4.69

Terumo ST 5Fr

0.9

1

1.94

1.19

0.3

4.93

Instrument name Guidewires

Catheters

Table 2. Distance agreement for each instrument along their length in term of number of particles. The distance range is the same as in Figure 3. The number of particles (Nb. parti.) for each anatomical location is also presented. Nb parti. at iliac artery

Nb parti. at aortic bifurc.

Nb parti. at renal artery

% Very good

% Good

Terumo angled

90

122

173

3

18

35

17

27

Terumo stiff

91

120

170

2

19

34

11

34

Amplatz

83

95

165

1

11

47

16

25

Instrument name

% Interm

% Poor

% Very poor

Guidewires

Bentson

88

119

167

2

16

41

8

34

Cook J

94

127

171

3

14

45

19

20

Cook Straight

89

119

167

6

20

32

15

27

Rosen

93

119

169

2

16

43

13

26

Catheters Beacon 5Fr

67

115

165

5

10

61

15

8

Terumo ST 4Fr

85

119

168

1

12

48

20

18

Terumo ST 5Fr

81

118

168

2

14

59

8

18

Table 2 shows the distance agreement for each instrument along their length and the number of particles at each instrument position. It ranges from very good agreement (< 0.5mm), good agreement (> 0.5mm, < 1mm), intermediate agreement (>1mm < 2mm), poor agreement (>2mm, < 3mm) and very poor agreement (> 3mm). On average, 3% of the particles are in very good agreement, 15% of the particles are in

V. Luboz et al. / Guidewire and Catheter Behavioural Simulation

323

good agreement, 44% of them are in intermediate agreement, 14% are in poor agreement, while 24% are in very poor agreement. 3. Discussion and Conclusion This paper presents the design of realistic IR instruments and their integration within our training environment. Use of realistic physical parameters such as flexural modulus may significantly improve the relevance of the behaviour of virtual IR instruments and, indeed, of the overall training experience, to actual use of real instruments in real patients. Seven guidewires and three catheters were implemented to match the shape and behaviour of their real counterparts. The shape is modelled by the position of the tip particles of each instrument. The behaviour depends essentially on the 2D bending coefficients for the instrument tip and body. The three other coefficients, related to the external force, the spring force and the 3D bending force, are constant for all instruments and therefore have only a minor influence on their flexibility. The instruments’ 2D bending coefficients were estimated through a set of experiments matching performance of these real instruments, in three dimensions within a silicone rubber vascular phantom, with their virtual representations. Results show good correlation with an average distance of 2.27mm between the real and virtual instruments and standard deviation of 1.54mm. These figures highlight the accuracy and realism of our virtual instruments. Some large errors can still be observed, especially at the instrument tip (up to 7.45mm for the Terumo stiff guidewire), but overall there is at least an intermediate agreement for most of the sampled points (on average 62%). The errors may in part relate to minor involuntary rotations that cannot be controlled in the real environment but are ignored or suppressed in the virtual world. Some error will also occur due to frictional forces of the silicon rubber material of the model being higher than those of the intimal lining of real vessels. Finally, the fact that the same bending coefficient is given to all the tip particles or all the body particles may not be realistic enough since the flexibility of the instrument changes along its whole length. Future work should address these sources of error. References [1] [2] [3] [4] [5] [6] [7] [8] [9]

F. Wang, L. Duratti, E. Samur, U. Spaelter and H. Bleuler, A Computer-Based Real-Time Simulation of Interventional Radiology, Eng. in Medicine and Biology Society (2007). C. Duriez, S. Cotin, J. Lenoir, and P. Neumann, New Approcahes to Catheter Navigation for Interventional Radiology Simulation, Comput Aided Surg. (2005), 11(6):300-8. D.A. Gould, J.A. Reekers et al., Simulation Devices in Interventional Radiology: Validation Pending, J. of Vascular and Interventional Radiology (2006), Vol. 17, Iss. 2, 215-216. J. Dankelman, M. Wentink, et al., Does Vitual Reality Training Make Sense in Interventional Radiology? CardioVascular and Interventional Radiology (2004), 27(5), 417-421. T. Alderliesten et al. , Modeling Friction, Intrinsic of Curvature, and Rotation Guide Wires for Simulation of Minimally Invasive Vascular Interventions. IEEE Trans. Biomedical Eng.(2007), 54 (1). V. Luboz, R. Blazewski, D. Gould and F. Bello, Real-time Guidewire Simulation in Complex Vascular Models, The Visual Computer (2009), vol. 25/9, 827-834. V. Luboz, J. Zhai, P. Littler, T. Odetoyinbo, D. Gould, T. How, F. Bello, Endovascular guidewire flexibility simulation. International Symposium on Biomedical Simulation (2010), 171-180. P. Schneider, Endovascular Skills: guidewire and catheter skills for endovascular surgery, 3rd ed, 2008. P.A. Yushkevich, J. Piven, et al., User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability. NeuroImage (2006), 31 (3), 1116-1128.

324


Design and Implementation of a Visual and Haptic Simulator in a Platform for a TEL System in Percutaneuos Orthopedic Surgery a

Vanda LUENGOa , Aurelie LARCHER a and Jérôme TONETTI b Laboratoire Informatique de Grenoble, Université Joseph Fourier, Grenoble, France b CHU de Grenoble, Service d’Orthopédie-Traumatologie, Grenoble, France

Abstract. Within a research project whose aim is to promote the learning of percutaneous operation in orthopedic surgery we design a Technological Enhanced Learning (TEL) system. This project belongs to a multidisciplinary field including computer, orthopedic surgery, medical imaging, didactic and cognitive sciences. The article presents the design principles of TEL with a particular interest in the development of a simulator. This simulator allows a virtual exercise interacting with the learner in visual, temporal and haptic dimension. Keywords. Technology Enhanced Learning, Simulation, orthopedic surgery.

Introduction The TELEOS Project (Technology Enhanced Learning Environment for Orthopaedical Surgery) is principally aiming to promote the learning of percutaneous orthopaedic surgery. There are three types of knowledge that are at stake during this learning activity: declarative, pragmatic (often empirical) and perceptivo-gestural. The objective of the system is to let the learner train himself freely on a percutaneous orthopaedic surgery in order to give him an epistemic feedback according to his actions. The feedback accompanies the subject in the learning process, by provoking reinforcements, destabilisations, hints, scaffolding, etc.

Figure 1. Architecture of the TELEOS Project.

These research works involve the developing of a TEL platform (Figure 1). Trails are produced during the problem solving activity in order to analyse the learner's behaviour.

V. Luengo et al. / Design and Implementation of a Visual and Haptic Simulator

325

The gathered data processing is made by another application entity: the diagnosis agent. This agent consults Bayesian networks in order to make a knowledge diagnosis of the learner's activity. A didactic decision agent is also involved to make an epistemic feedback to the learner in accordance with the diagnosis. Depending on the diagnosis, the feedback can consist in another exercise to do with the simulator, in lessons to consult online or in clinical cases to study. These three knowledge entities were based in a cognitive and didactic analysis of the surgeon activity formalised in one cognitive theoretical model [7]. In this article, we are going to explain how the simulation agent has been developed, this agent being trying to assimilate some characteristics of virtual reality: interaction, immersion, and autonomy.

1. Description of the System In order to encourage the extensibility and the re-usability of the software, we have chosen to take into account two kind of percutaneous operations in the simulator. These operations need a singular gesture and concern a distinct anatomic part: the vertebroplasty and the sacro-iliac screw. The first one is a spinal column operation which consists in injecting cement into a broken or a shrunken vertebra. The second one is a pelvis operation consisting in reducing and fixing the sacrum to the hip-bone. In these two situations, the act can be done if the surgeon manages to locate his tool in the patient's body. To do so, two indicators are available: the radiographies and the pressures felt during the introduction of the tool in the body. These radiographies have needed to be validated after some adjustment on the fluoroscope. Regardless of the simulated operation, the TEL system gives to the learner the opportunity to train himself to practise a surgical operation thanks to several functionalities: Choose the type of patient and the type of operation; Visualize in 3D the tool and the patient's model; Adjust the position and the incidence of the fluoroscopic image intensifier; Draw the cutaneous marking off on the body of the patient's model; Produce and visualize radiographies; Manipulate the surgical tool through a mouse or through haptic interface; Verify the trajectory when it was validated.

Figure 2. Simulator's graphical interface when the user adjusts the fluoroscopic, during the insertion and when the trajectory is validate by the user.

We can see in some of the presented interfaces, two 2D images representing the last two radiographies produced by the user, the 3D model of the patient, and the surgical tool and some graphical interface components such as a button or a cursor, to make some adjustments for the exercise.

326


1.1. The 3D Patient's Model and the 2D Patient's Radiography Our objective, in a learning point of view, is to propose a variety of patients’ models which represented different cases. Therefore, for the patient's model, the data used to create the volume comes from data files of real patients that had a scan before being operated. As a consequence, the simulator's 3D model will have a fractured, compressed, or slipped bone. The snapshots in DICOM format are gathered and used for 3D modelling. For the model to reach the highest quality, the scanner type must be a bodyscan, that is to say that all the body must be showed on the snapshots, and no injection must have been made to the patient because it would alter the tissue's aspect. Using these transverse sections and taking the spaces between intra and inter sections into account, the 3D model can be created. To do so, the C++ VTK library (Visualization ToolKit) [5] offers many classes to help designing 3D models. By giving the same contrast to all snapshots from all models, we make sure that there is just one correspondence between the scalar value of the image's pixel and the density. The use of a ray tracing algorithm [4], [2] coupled with an isosurface algorithm allows us to examine the volume and to assign a graphical property to all the voxels whose scalar value corresponds to the founded values, the other voxels not being displayed. From the same model, we can obtain, in the same way, a volume with the entire cutaneous surface or the entire bony surface visible. In the two situations, we assign a particular coloration according to the scalar value corresponding to the searched density. A radiographic image is a grayscale image with a number of transparency scales according to the tissues' density. Ray tracing algorithm [2] particularly meets the requirements to produce a radiographic image. But, contrary to rendering surface skin or bone, the use of an isosurface algorithm is not suitable. We are going to use a composition algorithm, considering different scalar values in order to assign distinct graphic rendering. The radiographic reproduction obtained is a volume, as the model with the aspect of skin or bone also is. In order to obtain the frontal, inlet, outlet, and profile snapshots, we are simply going to change the camera's position, and the camera's focal position from the 3D scene, being the place where the radiographic volume and the tool are, and then we are going to record the rendering image. It is this image which is going to be displayed in the 2D image of the simulator.

2. Towards an Immersive Interaction In our system, pseudonatural immersion consists in setting the learner standing in front of the graphic and haptic interfaces. With the mouse he can modify directly the graphical interface, for example by activating some buttons, or by shifting cursors. The Qt library is required as its signal and slot system simplifies the factual management. The user can also change the 3D scene using the VTK library to calculate in real time the 3D objects' transformations. The graphical interface modifications can alter the 2D and 3D result, like, for example, when the user makes a 2D radiography. With the haptic interface, the user physically modifies the stylus' position and orientation, as he would do manipulating a surgical instrument during an operation. The rotations and translations undergone by the stylus are directly retransmitted visually in the 3D graphic rendering of the simulator. At the same time, according to the body's density, a force feedback is produced by the device. As a consequence, we differentiate the


327

graphic rendering from the haptic rendering. A timer is in charge of the realization of these two renderings. The visual and tactile result seems fluent for the user and does not imply visible latency. The user's autonomy rests in his capacity to coordinate his perceptions and his actions during the interaction with others entities' process [6]. Our goal being to enable the learner to train himself freely and to allow him to make mistakes, the concept of autonomy is a crucial characteristic of our simulator.

3. Towards an Epistemic Retroaction The original approach of our TEL is based on the interest of producing epistemic retroactions [3]. From knowledge elicitation, experimentations are developed which generates different types of experimental data. This data is processed through Bayesian network. The goal is to inform the Bayesian networks, which is used by the diagnosis agent to set up a report according to the learner's exercise and by the didactic decision agent to evaluate the feedback method and content. From the simulator's conception point of view, it is essential to take the concept of trace into account for the diagnosis to be made. We have seen that the 3D model was made from a real patient's DICOM files. This step belongs to a unique procedure for each model of patient. It also includes segmentation steps used to find traces. The knowledge model underlines the anatomic parts that need to be segmented in subvolume or in interest point in order to record the interest point's position in a text file. The goal is to make some areas identifiable and to take them into account in the kinematics of the patient's gesture. For each new exercise, whether it is Sacro-iliac screw or vertebroplasty, the user selects the 3D model that is going to be used for the exercise. When a user realizes an exercise, a file containing the generated traces is created. The traces include information about the model chosen, and about the user's data, all dynamically generated by some events. Each significant action of the user will generate traces. We consider two kinds of events, the direct events, directly related to the user's action on the graphical interface, and the indirect events, which are related to the model's state. For the direct events, we associate the user's interaction on the graphical interface with an event generating traces. Most of the characteristic actions can be identified as such thanks to a direct action of the learner on the graphical interface. In other situations, the state of the collision between the tool and the model will be studied, particularly the cutaneous and the bony entry point. To locate these two events, we use the voxel's scalar value corresponding to the collision point between the tool and the volume. This scalar value is known all along the exercise. For example, the first time this value exceed zero, the tool get in contact with the body, this corresponds to the cutaneous entry point. In the same way, the first time this value is equal or superior to the value corresponding to a cortical bone's density, this is the bony entry point. When a significant action from the user is detected, data relative to the user is recorded. These are elements that could have been altered by the user. It can be adjustments, force exerted, speed, but also, more generally, chronological order or redundancy of the actions. For example, the learner can adjust the fluoroscope's position and its incidence rays, this action altering directly the radiographies. Thus, for each radiography, the simulator records the position of the camera and the focal. All the information, with the model's data about some interest points could be used by the

328


diagnosis agent to determine the knowledge required during the modifications made by the learner.

4. Evaluation and Discussion The current version of the TEL system includes a finalised version of the simulator and intermediate versions of Bayesian networks, diagnosis and didactic decision agents. Our architecture and methodology allow us a progressive validation. We specify and validate our computer models in interaction with two kinds of experts (medical and didactical).The simulator's functions reliability has been demonstrated by the methods used for its conception: To train oneself using real data, the used data's coherency is guaranteed by the data acquisition process for the construction of 3D models. To evaluate intelligible radiographies, the principle of radiography generation guarantees the homogeneity of the 2D rendering, in spite of the variability of the models. To insure a perceptual coherence, the graphic rendering is in accordance with the haptic rendering. Our first validations, with the medical’ experts, show that, for the mouse mode, the usability and correctness properties seem in accordance with the expected ones, i.e. for learning use. For the haptic mode, these fits test shows some correctness problems due our physical device. Indeed, the needle added is not perfectly aligned with the Ommi’ pen. Consequently there are a gap between the physical model and the 3D model. Otherwise, the preliminary evaluations with the didactical experts show that the completeness factor seem in accordance with the learning scenarios proposed. These validations allow us to install the stable version in the hospital. The system was used in June 2010 be used by ten students and currently we evaluate it in ergonomic and learning points of views. The first results are in concordance with the medical experts validations, i.e. the visual feedback are acceptable but it is necessary to work in an improved version of the hardware device in order to have better results with the added needle.

References [1] Ceaux, E., L. Vadcard, M. Dubois, et V. Luengo. «Designing a learning environment in percutaneous surgery: models of knowledge, gesture and learning situations.» European Association for Research on Learning and Instruction. Amsterdam, 2009. [2] Levoy, M. « Efficient ray tracing of volume data,,.» ACM Transactions on Graphics. Stanford University, 1990. 245-261. [3] Luengo, V. «Take into account knowledge constraints for TEL environments design in medical education.» International Conference on Advanced Learning Technologies. Santander: Springer, 2008. 5 pages. [4] Roth, S.D. « Ray casting for modeling solids.» Computer Graphics and Image Processing, 1982: 109144. [5] Schroeder, W., K. Martin, et B. Lorensen Schroeder. The Visualization Toolkit: An Object Oriented Approach to3D Graphics 3rd Edition. Kitware, Inc. Publisher., 2003. [6] Tisseau, J. Réalité virtuelle : autonomie in virtuo. Rennes, France: Thèse en informatique, Université de Rennes I, 2001. [7] Vadcard, L., et V. Luengo. «Réduire l'écart entre formation théorique et pratique en chirurgie : conception d'un EIAH,.» Environnements informatiques pour l'Apprentissage Humain. Montpellier: INRP, 2005. 129-140.


329

Computational Modeling of Human Head Electromagnetics for Source Localization of Milliscale Brain Dynamics Allen D. MALONYa,1, Adnan SALMAN b, Sergei TUROVETS c, Don TUCKER c, Vasily VOLKOV d, Kai LI c, Jung Eun SONG c, Scott BIERSDORFF b, Colin DAVEY c, Chris HOGE b, and David HAMMOND b a Dept. Computer and Information Science, University of Oregon b Neuroinformatics Center, University of Oregon c Electrical Geodesics, Incorportated d Dept. Mathematics and Mechanics, Belarusian State University

Abstract. Understanding the milliscale (temporal and spatial) dynamics of the human brain activity requires high-resolution modeling of head electromagnetics and source localization of EEG data. We have developed an automated environment to construct individualized computational head models from image segmentation and to estimate conductivity parameters using electrical impedance tomography methods. Algorithms incorporating tissue inhomogeneity and impedance anisotropy in electromagnetics forward simulations have been developed and parallelized. The paper reports on the application of the environment in the processing of realistic head models, including conductivity inverse estimation and lead field generation for use in EEG source analysis. Keywords. Electromagnetics, head modeling, brain dynamics, EEG, localization.

Introduction Advances in human brain science have been closely linked with new developments in neuroimaging technology. Indeed, the integration of psychological behavior with neural evidence in cognitive neuroscience research has led to fundamental insights of how the brain functions and manifests our physical and mental reality. However, in any empirical science, it is the resolution and precision of measurement instruments that inexorably define the leading edge of scientific discovery. Human neuroscience is no exception. Brain activity takes place at millisecond temporal and millimeter spatial scales through the reentrant, bidirectional interactions of functional neural networks distributed throughout the cortex and interconnected by a complex network of white matter fibers. Unfortunately, current non-invasive neuroimaging instruments are unable to observe dynamic brain operation at these milliscales. Electromagnetic measures (electroencephalography (EEG), magnetoencephalography (MEG)) provide high temporal resolution (≤1 msec), but their spatial resolution lacks localization of neural source activity. Hemodynamic measures (functional magnetic resonance 1

Corresponding Author.

330

A.D. Malony et al. / Computational Modeling of Human Head Electromagnetics

imaging (fMRI), positron emission tomography (PET)) have good 3D spatial resolution 1mm3, but poor temporal resolution on the order of seconds. Our research goal for the last six years has been to create an anatomicallyconstrained spatiotemporally-optimized neuroimaging (ACSON) methodology to improve the source localization of dense-array EEG (dEEG). Anatomical constraints include high-resolution three-dimensional segmentation of an individual's head tissues, identification of head tissue conductivities, alignment of source generator dipoles with the individual's cortical surface, and interconnection of cortical regions through the white matter tracts. Using these constraints, the ACSON technology constructs a fullphysics computational model of an individual's head electromagnetics and uses this model to map measured EEG scalp potentials to their cortical sources.

1. Methods Modern dense-array EEG (dEEG) technology, such as the Geodesic Sensor Net [19] from Electrical Geodesics, Inc. (EGI) shown in Figure 1(left), can measure micro-volt potentials on the human scalp at up to 256 sensors every 1 msec or less. EEG signals are the consequence of current dipoles associated with postsynaptic activities of neuronal cells. A single postsynaptic potential produces a current-dipole moment on the order of 20 fAm (femtoampere × meter) [9]. A 10 mm2 patch of the cortex surface contains approximately 100,000 neurons with thousands of synapses per neuron. At least 10 nAm is required to detect extracellular fields, and measurable EEG signals with a good signal-to-noise ratio require tens of millions of simultaneously activated synapses. As seen in Figure 1 (right), cortical neurons are arranged parallel to each other and point perpendicular to the cortical surface. It is this structural arrangement that allows currents from groups of thousands of neurons to accumulate and generate an equivalent current dipole for a cortex surface region. Therefore, scalp potentials measured by dEEG can be modeled by the combined electrical potentials (called lead fields) produced by up to 10,000 or more cortex patches. That is the good news. The bad news is that the scalp potentials are a linear superposition of all the distributed source lead fields and the individual EEG contributors (i.e., the distribute source dipoles) must be disentangled to determine the dynamics of each brain region.

Figure 1. (Left) EGI 256-channel Geodesic Sensor Net for dEEG recording and topographical potential maps showing epileptic spike wave progression between 110-310 msec with 10 msec samples. (Right) Neuronal current flows perpendicular to the cortex and creates dipole fields. Because of cortex folding, these fields can be radial, tangential, and oblique in orientation.

Localization Model. The general distributed source localization problem can be stated as follows: Φ = KS + E, where Φ=[φ1,...,φNt] are Ne measured EEG signals over


331

Nt time (NexNt), K is the lead field matrix (LFM) linking Ns current sources to their electrical potential (NexNs), S=[s1,...,sNt] are the current source values over time (NsxNt), and E is error over time. Since the only variables are the source dipole magnitudes S, their solution is a classic linear inverse problem obtained by inverting Φ. Unfortunately, NsNe, making the problem ill-posed. Methods for solving the underdetermined distributed source inverse problem apply minimum norm estimates and their generalization with various regularization schemes to overcome the ill-posed nature of the problem [8,13,14]. No matter how sophisticated the inverse technique, they all depend on determining the forward projection of current dipoles with unit magnitudes to scalp electrical potentials at known sensor locations (i.e., the lead field matrix K). Building K requires a model of the head electromagnetics. Electromagnetics Model. Given a volume conductor Ω with an arbitrary shape and ΓΩ as its boundary, a current density within the volume induces electric and magnetic fields E and B that can be measured on the conductor surface. If the conductivities σ and the electrical current sources S are known, the electric and magnetic fields inside the volume are fully described by Maxwell’s equations. Thus, the electrical forward problem for the human head can be stated as follows: given the positions and magnitudes of neuronal current sources (modeled as distributed dipoles), as well as geometry and electrical conductivity of the head volume Ω, calculate the distribution of the electrical potential on the surface of the head (scalp) ΓΩ. Mathematically, it means solving the linear Poisson equation: ∇ · σ(x, y, z)∇φ(x, y, z) = S in Ω with no-flux Neumann boundary conditions on the scalp: σ(∇φ) · n = 0. Here n is the normal to ΓΩ, σ = σij (x, y, z) is an inhomogeneous tensor of the head tissues conductivity and S is the source current; if the head tissues are considered to be isotropic, σ is a scalar function of (x, y, z), and — when they are orthotropic, σ is a diagonal tensor with off-diagonal — components σij =0, i≠j. Conductivity Inverse Model. If the head tissue conductivities are not known, it is necessary to solve the conductivity inverse problem by applying a general tomographic structure with a known current source, in this case current injected into the head at the scalp surface (this substitutes for neuronal current sources). From an assumed set of the average head tissue conductivities, σij, and given an injection current configuration, S, it is possible to predict the set of potential measurement values, φp, given a forward model, F, of head electromagnetics as the nonlinear functional by solving the Poisson equation above: φp = F(σij(x,y,z)). Once an appropriate objective function describing the difference between the measured scalp potentials, V, and the predicted potentials (at the sensor locations), φp, is defined (e.g., least square norm), and a search for the global minimum is undertaken using advanced nonlinear optimization algorithms [10,15]. When head tissue conductivities are determined, the forward model can be used to create the lead field matrix K by individually activating each current dipoles with unit magnitude and calculating the scalp electrical potentials at the sensor locations. With the LFM formed, it is then possible to solve for the spatiotemporal source dipole magnitudes S given a dEEG waveform.

2. ACSON Design The most critical component for source localization of dEEG measurements is the computational modeling of the electromagnetics of each subject. To build an

332


electromagnetics head model of the highest quality for an individual requires accurate anatomical constraints and biophysical parameters: High-resolution segmentation of head tissues. Various imaging methods (e.g., magnetic resonance imaging (MRI) and computerized axial tomography (CAT)) can provide volumetric data of the human head. Since the biophysical properties of each tissue are different and we want to employ quantitative (as opposed to qualitative pixel-to-pixel) piece-wise constant tomographic reconstruction, image segmentation is necessary for modeling. The physical geometry of the segmented tissues forms the basis for the 3D computational model. Determination of tissue conductivities. The human head tissues are inhomogeneous (different tissues have different conductivities) and anisotropic (conductivity can change with respect to orientation and other factors). None of the internal head tissues can be measured directly and noninvasively. They must be determined through bounded electrical impedance tomography (bEIT) and inverse modeling [4,15,16,17,20,21,22]. Cortex surface extraction and tesselation. To build a lead field matrix, dipole generators must be place at locations normal to the cortex surface. Cortex tesselation creates regions for dipole placement. Our research has produced methods and technologies to address these requirements. The ACSON environment shown in Figure 2 integrates the tools in a processing workflow that inputs head imagery (MRI, CT), bEIT data, and EEG sensor registration information and generates automatically accurate LFMs for use in source localization

Figure 2. The ACSON framework supports a workflow of MRI/CT image processing and electromagnetics modeling to deliver a lead field matrix for a single individual to use in source localization. The brain images on the right portray scalp EEG source-mapped to cortex locations.

3. Results The ACSON environment implements all the head modeling capabilities necessary for high-resolution source localization, but it has never been used until now to produce a


333

real head model and LFM for an individual that can be applied in source localization. We selected Dr. Colin Holmes (a.k.a. “colin27” in the Montreal Neurological Institute (MNI) BrainWeb database [2]) for this purpose. The MNI wanted to define a brain representative of the standard adult male population. They took 250 normal MRI scans, scaled landmarks to equivalent positions on the Talairach atlas [18], and averaged them with 55 additional registered images to create the “MNI305” dataset. In addition, one of the MNI lab members (Dr. Holmes) was scanned 27 times, and the scans were coregistered and averaged to create a very high detail MRI dataset of one brain. When compared to MNI305, it turned out that Dr. Holmes’ brain was (is) very close to the average head standard! While colin27 provides the necessary MRI data for segmentation, ACSON also requires bEIT scans. Luckily, Dr. Holmes has been a longtime collaborator with our group. Last year, he agreed to have 64 bEIT scans made. 3.1. Head Electromagnetics Forward Solver. The ADI and VAI forward solution methods for electromagnetic should first be validated with respect to a known solution. The source localization field has long used a concentric k-shell sphere model (k=3,4) as a theoretical standard of reference (each shell represents a head tissue), since analytical solutions are known for the isotropic and anisotropic case [3,5]. We created a 4-sphere testcase with 100x100x100 voxels and achieved a near-perfect correspondence between the theoretical isotropic and ADI results for a set of shell conductivities. Analytical solutions for spherical anisotropic models [3] are also available for VAI validation. We achieved very good accuracy with respect to the spherical model in both cases, lending strong confirmation that the algorithm is working properly. Based on these findings, the colin27 MRI dataset was segmented at (2mm)3 and 1mm3 resolutions into five tissue: scalp, skull, CSF, gray matter, and white matter. We built ADI and VAI head models and computed a forward solution for each resolution case for known conductivities and current sources. These models were evaluated relative to each other and then used for conductivity inverse and lead field calculations. 3.2. Conductivity Inverse Solution The ADI and VAI forward solvers for electromagnetic head modeling are the core computational components for the conductivity inverse and lead field matrix calculations. The conductivity inverse problem will need to process the bEIT measurements for up to 64 current injection pairs in the general case. Depending on the number of conductivity unknowns, each conductivity search for a single pair will require many thousands of forward solutions to be generated. Placement of current injection points is important to maximize the bEIT measurement value. Running the full complement of pairs enables the solution distribution to be better characterized. For all of our experiments, we set the number of tissue conductivity parameters to three: scalp, skull, and brain. Using the 1mm3 colin27 head model, a simulated annealing optimization process was applied to search for optimum values for all 64 EIT pairs. Histogram plots of conductivity solutions for all pairs were fitted with a normal distribution to determine mean and standard deviation. While other groups have reported research results for human head modeling and conductivity analysis (see [1,6,11,12]), our results are impressive because they are the first results in the field determined by dense array bEIT scanning, high-resolution subject-specific MRI/CT

334


based FDM of the human head, and simultaneous 3D search in the space of unknown conductivities. The derived brain/skull resistivity ratio is confirmed to be in the 1:20 to 1:30 range reported by other research groups [7,23]. 3.3. Lead Field Matrix Generation Once tissue conductivity estimates are determined, they can be used to calculate the lead field for all current dipoles of interest. Because the ACSON methodology is based in finite difference modeling, it is necessary to represent the dipoles normal to the cortex surface as vector triplets in x, y, z whose weighted combination determines the normal vector. The consequence is that three forward solves must be run, one for each axis orientation, for every dipole in three-space. We created an isotropic LFM and an anisotropic LFM for colin27 based on 4,836 axis dipoles. This required 9,672 forward solutions to be computed (half for ADI, half for VAI) by activating only one dipole and calculating the scalp projection. For each projection, we capture the value for 1,925 potential sensor locations. Thus, each LFM is 4836 x 1925 in size. 3.4. Source Localization Our efforts at building the most accurate electromagnetics head model culminate in the use of the LFM for source localization. We created an anisotropic LFM from a 1mm3 head model for 979 dipoles at 8mm spacing (2937 axis dipoles). For each dipole, we chose the LFM column representing that dipole’s scalp EEG projection at 1925 potential sensors locations and input the values for source localization. Magnitudes for all the dipoles were computed using sLORETA [14] and the one with the maximum intensity was determined and the 3D distance from the “true” dipole measured. Even with a noise level of 10%, the maximum magnitude dipole source localized with a anisotropic LFM is within 6.37mm of a 8mm spaced target dipole. The isotropic LFM is significantly worse. The bottom line is that modeling anisotropy in human head electromagnetics simulation is important for improving the accuracy of linear inverse distributed source solutions.

4. Conclusion We have created the ACSON methodology and environment to address one of the most challenging problems in human neuroimaging today – observing the high-resolution spatiotemporal dynamics of a person’s brain activity noninvasively. If such a capability existed, it would significantly advance neurological research and clinical applications, providing a powerful tool to study neural mechanisms of sensory/motor and cognitive function and plasticity, as well as improving neuromonitoring and neurorehabilitation for epilepsy, stroke, and traumatic brain injury. Our work provides an initial demonstration of the utility of full-physics modeling of human head electromagnetics and accurate head tissue conductivity assessment in improving the accuracy of electrical source localization. The ACSON modeling methods have been validated with analytical solutions and experimental results confirming prior research findings in the field.


335

Acknowledgment. This work was supported by a contract from the Department of Defense, Telemedicine Advanced Technology Research Center (TATRC).

References [1] M.Clerc, G.Adde, J.Kybic, T.Papadopoulo, J.-M.Badier, In vivo conductivity estimation with symmetric boundary elements, International Conference on Bioelectromagnetism, May 2005. [2] C. Cocosco, V. Kollokian, R. Kwan, G. Pike, A. Evans, Brainweb: Online interface to a 3D MRI simulated brain database, NeuroImage, 5:425, 1997. [3] J. de Munck, T. Faes, A. Hermans, R. Heethaar, A parametric method to resolve the ill-posed nature of the EIT reconstruction problem: a simulation study, Annals of the New York Academy of Sciences, 873:440–453, 1999. [4] B. Esler, T. Lyons, S. Turovets, D. Tucker, Instrumentation for low frequency studies of the human head and its validation in phantom experiments, International Conference on Electrical Bioimpedance, April 2010. [5] T. Ferree, J. Eriksen, D. Tucker, Regional head tissue conductivity estimation for improved EEG analysis, IEEE Transactions on Biomedical Engineering, 47(12):1584–1592, 2000. [6] S.Goncalves, et al., The application of electrical impedance tomography to reduce systematic errors in the EEG inverse problem: a simulation study, Physiological Measurement, 21(3):379–393, 2000. [7] S. Goncalves, et al., In vivo measurement of the brain and skull resistivities using an EIT-based method and realistic models for the head, IEEE Transactions on Biomedical Engineering, 50(6):754–767, June 2003. [8] R. Greenblatt, A. Ossadtchi, M. Pieger, Local linear estimators for the bioelectromagnetic inverse problem, IEEE Transactions on Signal Processing, 53(9):3403–3412, Sept. 2005. [9] M. Hamaläinen, J. Sarvas, Realistic conductivity geometry model of the human head for interpretation of neuromagnetic data, IEEE Transactions on Biomedical Engineering, 36:165–171, Feb 1989. [10] S. Kirkpatrick, C. Gelatt, M. Vecchi, Optimization by simulated annealing, Science, 4598:671–680, May 1983. [11] J. Meijs, O. Weier, M. Peters, A. van Oosterom, On the numerical accuracy of the boundary element method, IEEE Transactions on Biomedical Engineering, 36:1038–1049, 1989. [12] T. Oostendorp, J. Delbeke, D. Stegeman, The conductivity of the human skull: results of in vivo and in vitro measurements, IEEE Transactions on Biomedical Engineering, 47(11):1487–1492, 2000. [13] R. Pascual-Marqui, Review of methods for solving the EEG inverse problem, International Journal of Bioelectromagnetism, 1(1):75–86, 1999. [14] R. Pascual-Marqui, Standardized low resolution brain electromagnetic tomography (sloreta): Technical details, Methods and Findings in Experimental and Clinical Pharmacology, 24(5):22612, 2002. [15] A. Salman, A. Malony, S. Turovets, D. Tucker, Use of parallel simulated annealing for computational modeling of human head conductivity, In Y.S. et al., editor, International Conference on Computational Science, LNCS 4487:86–93. Springer-Verlag, 2007. [16] A.Salman, S.Turovets, A.Malony, Computational modeling of human head conductivity, In V. S. et al., editor, International Conference on Computational Science, LNCS 3514:631–638, Springer-Verlag, May 2005. [17] A. Salman, et al., Noninvasive conductivity extraction for high-resolution EEG source localization, Advances in Clinical Neuroscience and Rehabilitation, 6:27–28, 2006. [18] J. Talairach and P. Tournoux., Co-planar stereotaxic atlas of the human brain, Thieme, Stuttgart, 1988. [19] D.Tucker, Spatial sampling of head electrical fields: the geodesic sensor net, Electroencephalography and Clinical Neurophysiology, 87(3):154–163, 1993. [20] S. Turovets, et al., Bounded electrical impedance tomography for non-invasive conductivity estimation of the human head tissues, Electrical Impedance Tomography Conference, June 2009. [21] S. Turovets, et al., Conductivity analysis for high-resolution EEG, International Conference on BioMedical Engineering and Informatics, 2:386–393, 2008. [22] V. Volkov, A. Zherdetsky, S. Turovets, A. Malony, A fast BICG solver for the isotropic poisson equation in the forward EIT problem in cylinder phantoms, International Conference on Electrical Bioimpedance, Gainesville , FL, April 2010. [23] Y. Zhang, W. van Drongelen, B. He, Estimation of in vivo brain-to-skull conductivity ratio in humans, Applied Physics Letters, 89:2239031–3, 2006.

336


Simulation and Modeling of Metamorphopsia with a Deformable Amsler Grid Anabel MARTIN-GONZALEZa,1, Ines Lanzl b, Ramin Khoramnia b and Nassir NAVAB a a Chair of Computer Aided Medical Procedures (CAMP), TUM, Germany b Ophthalmology Department, Klinikum rechts der Isar, Germany

Abstract. A method to simulate and model metamorphopsia by means of a deformable Amsler grid is proposed. The interactively deformable grid is based on cubic B-splines to obtain a locally controlled deformation. By simulating metamorphopsia on normal sight volunteers, acquisition of a correction percentage is possible as a result of analyzing the magnitude of the simulated distortion and the applied correction model. The correction percentage obtained is 75.78% (7.36% standard deviation). This can express the feasible correction rate with the guidance of the patient qualitative feedback. The present work is motivated by the idea of obtaining a correction model of a patient with metamorphopsia and to implement this model into a head-mounted display to compensate the patient’s deformation in the near future. Keywords. Visual impairment, augmented reality, metamorphopsia

Introduction In ophthalmology, augmented reality (AR) is playing an important role as a result of enhancing the view of visually impaired people with different methodologies in order to improve their degenerated vision. The head-mounted display (HMD), a relevant interface in AR, seems to be a suitable medium to assist people with diverse eye diseases [1]. Macular disorders such as age-related macular degeneration (AMD), idiopathic epiretinal membrane (ERM) and macular hole have been found to cause metamorphopsia, a symptom described as the perception of distortion of objects. Patients with metamorphopsia visualize a straight line as an irregular or curved line. It is known that one of the main causes of metamorphopsia in individuals with macular diseases is the displacement of photoreceptors in the sensory retina [2]. Nevertheless, this disorder is not completely well understood. The use of Amsler charts is a common clinical approach for detecting metamorphopsia [3]. An Amsler grid is a printed squared grid (10×10 cm) containing equally spaced parallel horizontal and vertical lines. Variants of the Amsler grid have been elaborated to examine central vision [4], but the original chart seems to perform 1

Corresponding Author: Anabel Martin-Gonzalez, Technische Universität München (TUM), Boltzmannstr. 3, 85748, Garching, Germany; E-mail: [email protected].

A. Martin-Gonzalez et al. / Simulation and Modeling of Metamorphopsia

337

better. Although it is a standard mean for diagnosing and measuring metamorphopsia, an Amsler grid cannot evaluate quantitatively the degree of metamorphopsia. Some methods have been proposed for measuring metamorphopsia. Matsumoto et al., have developed a method for quantifying metamorphopsia in patients with ERM by the use of M-CHARTS [5]. The Preferential Hyperacuity Perimetry [6] (PreView PHP, Carl Zeiss Meditec, Dublin, CA) is suitable for mapping the area in the visual field affected by metamorphopsia and to follow the progression of deterioration. The PHP is based on hyperacuity (Vernier acuity), that is the ability to recognize the relative spatial localization of two or more stimuli [7]. Trevino [8] and Crossland [9] show different methods for macular assessment, but the main problem in these methods is the difficulty to reliably assess due to the lack of knowledge of the ground truth of the patient’s vision, some defects may not be covered by the grid lines. In [10], a dynamic Amsler grid created in computer-graphics was developed to overcome these deficiencies. Nevertheless, these methods do not provide the way patients perceive their distorted world in detailed, this is, their visual deformation model. As a first step on this research, we propose a deformable Amsler grid based system to simulate distorted vision in healthy eyes in order to analyze reliability of the system for obtaining an inverted deformation model, termed correction model of metamorphopsia. By having a deformation model, it could be possible to localize the macular areas causing deformations in optical coherence tomography (OCT) images of the patient retina and analyze them to see if it is viable to find any macular pattern related to the deformation’s shape. A correction model could be applied to images of AR display devices (i.e., head-mounted display) and therefore, a correcting system for patients with distorted vision could be achieved.

1. Methods & Materials The method consists of simulation of distorted vision for the human eye; acquisition of the correction model of the simulated visual distortion; and finally the analysis of the obtained results. The hardware used includes a 17” monitor, an eye tracker device, and a workstation with Intel Core Duo CPU at 2.40 GHz, 2GB of RAM (see Figure 1). 1.1. Deformable Amsler Grid In order to create a deformable Amsler grid, we have chosen a free form deformation (FFD) model [11], based on B-splines, which is a feasible tool for modeling deformable objects. Basically, the idea of FFD’s is to deform an object (i.e., Amsler

Figure 1. Simulation and modeling system (left); volunteer and examiner performing an experiment (right).

338


grid) by manipulating an underlying mesh of control points. The resulting deformation gives us the metamorphopsia correction model of the patient and produces a smooth and C2 continuous transformation. To define a spline-based FFD, we denote the domain of the image as = {(x, y) | 0 x < X, 0 y < Y}. Let denote a nx × ny mesh of control points i,j with uniform spacing . Then, the FFD is defined as the 2D tensor product of the 1D cubic B-spline functions and the displacements of the control points: 3

T ( x, y )

3

¦¦ B

m

(u ) Bn (v)4i m , j n

(1)

m 0n 0

where i ¬x / n x ¼ 1 , j ¬y / n y ¼ 1 , u x / nx ¬x / nx ¼ , v y / n y ¬y / n y ¼ , and where Bm presents the m-th basis function of the B-spline: B0(u) = (1 – u)3/6, B1(u) = (3u3 – 6u2 + 4)/6, B2(u) = (–3u3 + 3u2 + 3u + 1)/6, B3(u) = u3/6. In contrast to thin-plate splines [12], B-splines are locally controlled. The resulting transformation from changing the control point i,j will affect only the local neighborhood of that control point. The degree of non-rigid deformation which can be modeled depends on the resolution of the mesh of control points. A deformable Amsler grid based on cubic B-splines will provide smooth transformations. 1.2. Simulation of Distorted Vision In order to simulate metamorphopsia on a normal sighted person, an image deformation is generated and placed on a specific location of the grid (Figure 2). Considering the grid center as the visual center (visual angle 0º), the eye tracker follows the gaze so that the deformation is displayed all the time in the same selected location of the person’s visual field. As a result, it is possible to recreate the visual imperfection caused by the displacement of photoreceptors in the human eye. In the same manner as with real metamorphopsia patients, to digitally correct distorted vision it is necessary to deform the grid section, located in the affected retinal field, in the opposite direction of the real perceived deformity. Therefore, to correct the simulated distortion an interaction with the mouse on the monitor moves the grid lines in the opposite trend of the perceived distortion until the person sees no deformation in the lines. This procedure will provide the metamorphopsia correction model. 1.3. Experiments In order to evaluate whether the grid resolution can affect the discernment of distortions or not, a squared and a non-squared grid are tested. The dimension for every grid is 46.9×26.2 cm. Eight points on both grids are selected for locating a pre-

Figure 2. Deformation example on the non-squared grid.


339

generated distortion (Figure 3). Every location corresponds to a specific retinal area of the visual field (the grid center corresponds to visual angle 0º). The magnitude of deformation on every point of the grid is randomly defined covering a visual angle range of 0.18º to 1º. A group of volunteers is selected to participate in the experiments. The criteria for inclusion is best corrected visual acuity equal to 20/20. The experiment is as follows (Figure 1): the volunteer is located in front of the external monitor at a distance of 80 cm. One eye is covered and the other one is fixating the center of the displayed grid during the whole experiment. One of the eight predefined distortions is simulated on the screen. The examiner performs a blind correction; this is, without seeing the projected distortion to the volunteer (to recreate the real situation with a patient where the examiner cannot see what the patient perceives). During this simulated environment, the examiner obtains from the volunteer a description of the perceived distortion (location and shape orientation). Once the description is provided, the examiner stops the simulated distortion. The grid will become regular again (i.e., with straight lines) and the examiner will now see the screen to interact with the grid in order to acquire the correction model for that specific distortion; this is, to deform the grid, in the provided location and in the opposite given orientation. After this step, the examiner does not look the screen anymore and turns on the simulated distortion on the grid that includes the examiner’s modification. At this time, the simulated distortion may be reduced (or disappear completely) and the deformed grid line affected by the simulated distortion may seem straighter (or completely straight). The blind correction procedure will be performed until the person cannot perceive any distortion. The procedure will be done with the squared and nonsquared grid (the first one being tested is randomly chosen for each volunteer). Once the experiment is finished, the correction percentage related to the original simulated deformation is analyzed. The correction percentage will be measured by calculating which percentage of the magnitude of the displacement vector of the control point for simulating the distortion was reduced after the correction procedure. If the magnitude of the displacement vector after correction is 0, a 100% of correction will be measured. The eye tracker avoids the natural instinct of trying to fixate not the grid center, but the simulated distortion projected out of the central vision. Therefore, the eye tracker moves the simulated distortion to its corresponding location in the visual field according to the gaze movement, thus the distortion shifts its location and the person cannot focus it with the central vision. The visual angle V corresponding to each one of the eight points in the grid can be obtained by the equation V = 2 arctan (S/2D), where S is the object length in the real world and D is the distance from the eye to the object (i.e., monitor).

Figure 3. Squared grid (left) and non-squared grid (right) with evaluated locations.

340


Table 1. Total correction rate (percent) on squared and non-squared grids. Squared Grid 75.75 10.12

AVG SD

Non-Squared Grid 75.78 7.36

Table 2. Correction rate (percent) and minimum recognizable visual angle (degrees) for each evaluated location. Location

Visual Angle

p0 p1 p2 p3 p4 p5 p6 p7

0.00 2.08 4.15 5.94 6.22 8.29 8.60 11.85

Correction Rate Squared Grid AVG SD 90.63 7.58 81.91 14.26 57.23 31.57 74.07 22.64 70.36 17.44 79.16 12.89 79.71 19.07 72.90 25.15

Non-Squared Grid AVG SD 80.23 12.97 81.13 8.69 72.85 23.47 73.49 23.50 76.05 16.29 81.49 11.31 66.61 26.47 74.38 12.02

Minimum Recognizable Visual Angle Squared Grid Non-Squared Grid AVG SD AVG SD .0364 .0303 .1290 .1147 .0871 .0621 .0986 .0722 .1259 .0804 .1163 .1120 .1518 .1224 .1192 .0993 .1662 .1032 .1328 .0922 .1643 .1170 .0918 .0852 .1204 .1346 .1336 .0860 .1593 .1182 .1705 .0908

2. Results The system provides a model of distorted vision in a range of 32.67º horizontal and 18.60º vertical visual angles. The non-squared grid has a horizontal line spacing of 2.97º and a vertical line spacing of 2.08º in visual angles. In the case of the squared grid, the lines have an approximately equal spacing of 1.98º horizontal and 2.08º vertical. The resolution of the mesh of control points on the grid for modeling transformations is 35×19 points. In total, 17 normal subjects (17 eyes) could reliably fulfill the task. The average (AVG) and standard deviation (SD) of the correction rate obtained are presented in Table 1. The Table 2 shows the visual angle (in degrees) corresponding to each selected location for evaluation on the grid with its correction rate. These results are plotted in a graph for easy visualization and analysis (see Figure 4). According to the results, the correction rate averages do not show any significant difference between using a squared or a non-squared grid. To go deeper in the studies, it is relevant to analyze the minimum recognizable visual angles for every selected location; this is obtained with the final magnitude of the displacement vector of the control point for simulating the distortion after the correction procedure. Table 2 presents the results. In Figure 4, it is shown that with a non-squared grid it is possible to distinguish smaller deformations in the middle

Figure 4. Correction rate (left); minimum recognizable visual angles (right).


341

peripheral vision (p2 to p5); for locations near the visual center (p0 and p1) and far away from it (p6 and p7) a squared grid has slightly better results. As it could be expected, there is an incremental trend relating the minimum recognizable deformation and its location in the visual field, this means that the farther away the deformation is from the visual center the less a subject can recognize it. However, for these locations, a subject can still guide the examiner to perform a reasonable correction as seen in the results of Figure 4. An eye tracker device plays a very important role for simulation of metamorphopsia for a healthy eye, not only because it reproduces the feeling of having a real distortion moving with the gaze, but also for preventing the person from looking directly at the distortion during the experiment to describe it, instead of fixing the center of the grid. Moreover, an eye tracker can provide us with the feedback of whether the real patient is fixating the grid center or not, increasing the accuracy of correction and localization of the affected areas in the macula for further analysis. Therefore, tracking’s accuracy of a single eye is an essential parameter for success in modeling and correcting distorted vision. In addition, the system can provide a measurement of the deformation seen in the grid in order to have an estimation of the size of the affected area in the macula (in mm) according to the patients’ visual perception; which helps to evaluate the progression of the disease. The resolution of the mesh of control points in the grid can be increased to model more difficult metamorphopsia cases. The final correction model acquired can be implemented in a head-mounted display based augmented reality system. Thus, it will be possible to compensate distorted vision in real patients. Furthermore, the deformation model (inverted correction model) can be used as a model to describe metamorphopsia for medical diagnosis by locating the affected areas in macular OCT and fundus images with the help of a system we have already developed (see Figure 5). The OCT imaging technology defines a 3D volume of the retina by means of a set of cross-sectional images orthogonal to the retina’s surface plane, which is visible in a fundus image. An integrated eye tracker guides the OCT scan to the selected location (normally the fovea or vision center), and in relation to that position it acquires the cross-sectional images of the retinal layers including the photoreceptors layer. For registering the OCT images and the distortion information obtained using our Amsler grid experiments, we generate the projected image (in mm) on the retina produced by the grid at the distance of 80 cm. This is based on the fact that the visual plane of the subject is parallel to the grid and fixating its center (as done in the experiments). Thus, it is possible to approximately align the grid on top of the retina (fundus image) and therefore place the OCT images in the position where they were

Figure 5. Alignment of macular OCT image and deformable Amsler grid.

342


scanned, see Figure 5. This alignment of information could help for a better understanding of metamorphopsia and could create a set of valuable multi-modal data for evaluating the possibility of development of novel quantitative and/or qualitative analysis of visual distortion based on OCT images.

3. Conclusions A system based on a deformable Amsler grid for simulating and modeling metamorphopsia was developed. The grid based on cubic B-splines provides smooth transformations suitable to correct visual deformations. The feasible correction rate for distorted vision by using a deformable non-squared grid and the guidance of the patient qualitative feedback is 75.78%. The use of an eye tracker increases the reliability of the results. A deformable Amsler grid based system could provide a simple and useful method for modeling distorted vision in patients with metamorphopsia. By implementing the correction model on an AR display device, it would be possible to improve the vision of patients with metamorphopsia.

Acknowledgment This project is supported by the Bayerische Forschungsstiftung and partly by the Secretaría de Educación Pública de México.

References [1] E. Peli, G. Luo, A. Bowers, and N. Rensing, Development and evaluation of vision multiplexing devices for vision impairments, Int J Artif Intell Tools 18 (2009), 365–378. [2] E. Arimura, C. Matsumoto, S. Okuyama, S. Takada, S. Hashimoto, and Y. Shimomura, Retinal contraction and metamorphopsia scores in eyes with idiopatic epiretinal membranes, Invest Ophthalmol Vis Sci 46 (2005), 2961–2966. [3] M. Amsler, Earliest symptoms of diseases of the macula, Br J of Ophthalmol 37 (1953), 521–537. [4] M.F. Marmor, A brief history of macular grids: from Thomas Reid to Edvard Munch and Marc Amsler, Surv Ophthalmol 44 (2000), 343–353. [5] C. Matsumoto, E. Arimura, S. Okuyama, S. Takada, S. Hashimoto, and Y. Shimomura, Quantification of metamorphopsia in patients with epiretinal membranes, Invest Ophthalmol Vis Sci 44 (2003), 4012– 16. [6] A. Loewenstein, R. Malach, M. Goldstein, I. Leibovitch, A. Barak, E. Baruch, Y. Alster, O. Rafaeli, I. Avni, and Y. Yassur, Replacing the Amsler grid: a new method for monitoring patients with age-related macular degeneration, Ophthalmology 110 (2003), 966–970. [7] M. Goldstein, A. Loewenstein, A. Barak, A. Pollack, A. Bukelman, H. Katz, A. Springer, A.P. Schachat, N.M. Bressler, S.B. Bressler, M.J. Cooney, Y. Alster, O. Rafaeli, and R. Malach, Results of a multicenter clinical trial to evaluate the preferential hyperacuity perimeter for detection of age-related macular degeneration, Retina 25 (2005), 296–303. [8] R. Trevino, Recent progress in macular function self-assessment, Ophthalmic Physiol Opt 28 (2008), 183–92. [9] M. Crossland and G. Rubin, The Amsler chart: absence of evidence is not evidence of absence, Br J of Ophthalmol 91 (2007), 391–393. [10] L. Frisén, The Amsler grid in modern clothes, Br J of Ophthalmol 93 (2009), 714–716. [11] T.W. Sederberg and S.R. Parry, Free-form deformation of solid geometric models, in: Proceedings of SIGGRAPH ’86, Computer Graphics 20 (1986), 151–160. [12] F.L. Bookstein, Principal wraps: Thin-plate splines and the decompositions of deformations, IEEE Trans Pattern Anal Machine Intell 11 (1989), 567–585.


343

Development of a Customizable Software Application for Medical Imaging Analysis and Visualization Marisol MARTINEZ-ESCOBAR, Catherine PELOQUIN, Bethany JUHNKE, Joanna PEDDICORD, Sonia JOSE, Christian NOON, Jung Leng FOO 1, and Eliot WINER Virtual Reality Applications Center, Iowa State University, Ames, Iowa, USA

Abstract. Graphics technology has extended medical imaging tools to the hands of surgeons and doctors, beyond the radiology suite. However, a common issue in most medical imaging software is the added complexity for non-radiologists. This paper presents the development of a unique software toolset that is highly customizable and targeted at the general physicians as well as the medical specialists. The core functionality includes features such as viewing medical images in two- and three-dimensional representations, clipping, tissue windowing, and coloring. Additional features can be loaded in the form of ‘plug-ins’ such as tumor segmentation, tissue deformation, and surgical planning. This allows the software to be lightweight and easy to use while still giving the user the flexibility of adding the necessary features, thus catering to a wide range of user population. Keywords. Software Framework, User Interface Design, Visualization

1. Introduction Visualizing patient data has become an integral part of the diagnosis and clinical evaluation stages. As medical imaging and graphics technologies improve, visualization software applications are being expanded to include more advanced tools. Surgeons and doctors now have the ability to perform data visualization, analysis, and diagnosis outside of the traditional radiology setting. However, these advanced tools can sometimes be a hindrance when they are not developed with Human Factors considerations and the software tends to be unusable or require a steep learning curve. To address this problem, the Isis software framework was designed (Figure 1) by understanding the needs of the user and recognizing the physical and mental capabilities of the targeted user group. The framework was developed first by identifying several critical and common tasks performed by doctors and surgeons. The tools required to complete these tasks were then organized in an effective manner that would allow the user to understand and learn the software easily. Currently, the Isis software framework consists of three primary components: 1) Examiner, 2) Surgical planning, and 3) Data Analysis.

1 Corresponding Author: Jung Leng Foo, Virtual Reality Applications Center, 1620 Howe Hall, Iowa State University, Ames, Iowa, 50011, USA. E-mail: [email protected]

344

M. Martinez-Escobar et al. / Development of a Customizable Software Application

Figure 1. Breakdown of the various features and components in the Isis software framework. Isis Core has the features of a basic Examiner, while the other categories can be implemented as Isis Plug-ins.

Earlier work on this project was the development of Isis VR [1], an immersive environment to interact and manipulate three-dimensional representations of patient medical image data in real time. In an effort to foster collaboration between surgeons, work was also done to create a collaborative multi-modal environment for surgical planning [2]. A basic desktop viewer was developed to sync over the network with the virtual environment, where any user interactions on one would be updated on the other. The desktop viewer has since been renamed Isis, the desktop counterpart to Isis VR. The work presented in this paper discusses the redevelopment of Isis. A highly customizable software framework with a user interface tuned to the needs of primary care physicians and specialists (e.g., oncologists and surgeons). Isis provides a simple and effective environment for interaction and visualization of patient image data.

2. Methods 2.1. Display and Interaction Widgets The Isis interface can be divided into four separate sections: 1) The main display window, 2) Examiner features panel, 3) Mode switcher, and 4) Mode specific features panel. These features make up a simple, straightforward, and basic application to visualize and interact with digital medical image data. By organizing and grouping the examiner features panel into the right side, the user can access common features in all the modes. This arrangement allows the user to create effective mental mappings between the controls and the functions in the interface. The main display window can display two- and three-dimensional representations with which the user can interact (rotation, translation, and zoom). Additional controls are available in the examiner features panel, where the user can change the views, colors, and tissue types displayed. A screenshot showing a sample configuration of the main display with the features panel is shown in Figure 2.


345

Figure 2. The Isis software in ‘Core’ mode displayed in a tiled viewed.

The ‘View’ option in the figure allows a user to switch amongst predefined 2D and 3D views. A user can select all available views (as shown in the figure) or any combination. The ‘Color’ menu gives users quick access to predefined color schemes to highlight specific anatomical features or structures. ‘Tissue Type’ contains several predefined windowing settings as well as the ability to define custom values to isolate specific densities such as bone, muscle, cartilage, etc. 2.2. Software Libraries Isis was developed using several open source software libraries: • Qt (http://qt.nokia.com) • DICOM Toolkit (DCMTK) (http://dicom.offis.de/dcmtk.php.en) • Visualization Toolkit (VTK) (http://www.vtk.org). The user interface (UI) was designed using Qt, an open source cross platform UI framework. This facilitates a consistent look of the user interface across various operating systems for a consistent user experience. Currently, Isis is compatible with Windows 7, Windows XP, and Mac OSX. The software architecture of the Qt framework also decouples the UI and the backend code, allowing the UI to be designed independently of the source code development. Intermediate functions are used to connect the inputs from the UI to actual function calls in the source code. DCMTK was used to parse the input files and it has a large collection of tools to access the information stored in a DICOM/PACS file as well as perform the necessary functions to parse the image data. The main advantage of DCMTK is its compatibility with files created from the various scanner manufacturers.

346


VTK is the primary rendering engine for Isis. VTK contains extensive threedimensional rendering routines such as real time volume rendering, surface generation, and multi-planar reconstruction (MPR) based on the OpenGL standard. This allows Isis to take advantage of 3D graphics hardware acceleration as well as rendering on the computer processor. The rendering routines in VTK also includes features such as coloring and transparency of the 2D and 3D representations, as well as clipping planes to interactively ‘slice’ through a 3D volume. 2.3. Isis Core and Isis Plug-Ins During the software design phase, a decision was made to develop Isis as separate components: Isis Core and Isis Plug-ins. Isis Core contains the basic features such as two- and three-dimensional viewing, clipping, tissue windowing, and coloring; and additional features can be implemented into Isis Core as plug-ins. Plug-ins that are currently available are: 1. Tumor segmentation 2. Surgical planning The tumor segmentation plug-in will be based off segmentation algorithms developed by the authors. These include a fuzzy segmentation algorithm [3] and a probabilistic segmentation algorithm [4], and a colorization method to colorize grayscale images based on user inputs on properties of the tumor and region of interest. For diagnosis and treatment planning purposes, the segmented tumor can also be overlaid on the original patient data, providing the doctor with context information such as the shape and size of the approximated tumor relative to critical anatomical structures when combined with the surgical planning plug-in (Figure 3). This is a feature from Isis VR [5] and was redesigned for the desktop environment.

Additional features will include measurement tools (Figure 4) and landmark placements to provide additional information such as the size of the tumor, as well as the distance of critical structures, i.e. arteries, from the tumor. These tools will assist in


347

trocar placements for laparoscopic surgeries or incision locations in open surgeries. The advantage of having these advance tools as plug-ins is that each user can customize Isis based on their own needs and preferences. For example, a medical student in training might only require Isis Core, while a practicing surgeon will want Isis Core with the tumor segmentation and surgical planning plug-ins. As the development of Isis progresses, additional plug-ins can be implemented to cater to a variety of target user groups while still ensuring that the basic software application remains lightweight and robust.

3. Discussion and Conclusion As the research into medical imaging continues to produce more advance analysis and visualization tools, medical imaging software will continue to grow in complexity. However, the average user might not require or even understand half of the tools and features included in these software packages. The Isis software framework proposed in this paper creates a highly customizable and expandable platform. From the first year medical student to a practicing physician, the Isis Plug-ins can cater to a wide user group. Additional development is currently being planned, as well as a formal usability study for the user interface and the various features built in to Isis. The evaluation methods will measure the performance and the user experience as it relates to performance, i.e. time to localize anatomical features. User experience will be measured by conducting usability studies that will identify issues with the interface that may interfere with the performance. In addition, cognitive load testing will be assessed for students who want to learn anatomy using 3D visualization methods. Cognitive load can help to identify if the information on the interface overloads the processing capabilities of the user, preventing the student from making new associations in longterm memory and hindering learning [6].

References [1]

[2]

[3]

[4]

[5]

[6]

Foo JL, Lobe T, and Winer E. A Virtual Reality Environment for Patient Data Visualization and Endoscopic Surgical Planning, Journal of Laparoendoscopic & Advanced Surgical Techniques, 18 (5) 697-706, 2008. Foo JL, Martinez M, Peloquin C, Lobe T, and Winer E. A Collaborative Interaction and Visualization Multi-Modal Environment for Surgical Planning, Proceedings of 17th Medicine Meets Virtual Reality (MMVR) Conference, 142 (2009), 97-102. Published by IOS Press. Foo JL, Miyano G, Lobe T, and Winer E. Three-Dimensional Segmentation of Tumors from CT Image Data using an Adaptive Fuzzy System, Journal of Computers in Biology and Medicine, 62 (2009) 869878. Foo JL, Lobe T, and Winer E. Automated Probabilistic Segmentation of Tumors from CT Data using Spatial and Intensity Properties, Proceedings of SPIE Medical Imaging, Lake Buena Vista, FL, February 8-10, 2009. Foo JL, Miyano G, Lobe T, and Winer E. A Framework for Interactive Examination of Automatic Segmented Tumors in a Virtual Environment, Proceedings of 16th Medicine Meets Virtual Reality (MMVR) Conference, 132 (2008), 120-122. Published by IOS Press. Wickens, C, Lee J, Liu Y, and Gordon S. An Introduction to Human Factors Engineering, 2004. Published by Prentice Hall.

348


Pneumoperitoneum Technique Simulation in Laparoscopic Surgery on Lamb Liver Samples and 3D Reconstruction MARTÍNEZ-MARTÍNEZ F.a,1, RUPÉREZ M. J. a, LAGO M.A. a, LÓPEZ-MIR F. a, MONSERRAT C. a, and ALCAÑÍZ M.a a Instituto Interuniversitario de Investigación en Bioingeniería y Tecnología Orientada al Ser Humano, Universidad Politécnica de Valencia. Camino de Vera s/n 46022, Valencia, Spain. Phone/Phax: +34 96 387 75 18 (Ext. 67020) / +34 96 387 95 10

Abstract. In this paper, a procedure to experimentally simulate the behavior of the liver when the pneumoperitoneum technique is applied in laparoscopic surgery is presented, as well as methodology to make the comparison of each sample before and after insufflating the gas. This comparison is carried out using the 3D reconstruction of the volume from the CT images when either pneumoperitoneum is applied and when it is not. This methodology has showed that there are perceptible changes of volume when the pneumoperitoneum is applied. Keywords. Laparoscopy, reconstruction.

pneumoperitoneum,

liver,

simulation

and

3D

Introduction Laparoscopy is a minimally invasive technique based on the execution of little incisions on the abdomen aimed at introducing the instrumental needed to perform the intervention (Figure 1). At the beginning of this, CO2 is insufflated in the abdomen through a trocar in order to create enough space to make possible the intervention. This technique is referred to as Pneumoperitoneum [1]. In this paper, a procedure to experimentally simulate the behavior of the liver when the pneumoperitoneum technique is applied is presented. This procedure will show that there are perceptible changes of volume when the pneumoperitoneum is applied.

1

Corresponding Author. Email address: [email protected] (Martínez-Martínez F.)

F. Martínez-Martínez et al. / Pneumoperitoneum Technique Simulation in Laparoscopic Surgery

349

Figure 1. Laparoscopic surgery

1. Materials and Methods Two circular samples were obtained from four lamb livers, one of 100 mm of diameter and another smaller of 80mm of diameter. The samples were introduced in a glass receptacle which was hermetically sealed but connected to a tube through which the CO2 was insufflated (Figure 2, left). The device used to insufflate the CO2 was Wolf IP20 (Figure 2, middle). The pressure was kept constant during all the procedure. Two different ranges of pressure values were tested: 10-14 mmHg (the most common used in abdominal interventions) was applied to 4 samples and 16-19 mmHg was applied to the other 4 samples. The receptacle was introduced in the multi-detector spiral CT GE LightSpeed VCT–5124069 (Figure 2, right) and CT images were acquired from each sample before applying CO2 and when the CO2 was applied. The axial slices interval was of 0.625 mm. The experiments were carried out in Hospital Clínica Benidorm (HCB). The CT images were processed using the software ScanIP v4.0 from Simpleware in order to obtain the volume of each sample. The steps were: Segmentation and filtering, 3D reconstruction and volume calculation (Figure 3).

Figure 2. Material used for the experiment: Liver sample inside the hermetical glass receptacle (left), insufflating device Wolf IP20 (middle) and Multi-detector spiral CT GE LightSpeed VCT-5124069

2. Results The results are shown in Table 1. Table 1 shows the volume in cm3 of every sample when the CO2 is applied and when it is not applied for the two ranges of pressure. It also shows the volume difference between the same sample when CO2 is applied and when it is not. As volume of one voxel was 1.49x10-3 cm3, the precision was 1x10-3 cm3.

350

F. Martínez-Martínez et al. / Pneumoperitoneum Technique Simulation in Laparoscopic Surgery

Figure 3. Steps for the volume calculation

Table 1. Results Liver Samples Volume (cm3)

No CO2 With CO2 Δ Volume

LS1 178.552 176.923 1.639

10-14 mmHg LS2 LS3 211.785 71.639 211.313 71.256 0.472 0.383

LS4 203.438 203.355 0.083

LS5 94.796 94.290 0.506

16-19 mmHg LS6 LS7 200.268 83.776 200.247 83.660 0.021 0.116

LS8 195.307 194.841 0.466

3. Conclusions As a result of this work, it can be concluded that there is perceptible changes in the volume of the samples when CO2 is applied. This means that the liver is lightly compressed under these pressures when a laparoscopy surgery is carried out. This confirms that the Poisson’s coefficient largely responsible of the compression is high, which agrees with the value that can be found in the literature (0.45 according to [2]).

4. Discussion and Future Works The results of this paper are very useful for planning liver laparoscopic surgeries since it allows to simulate the compression of this organ when the pneumoperitoneum technique is applied. In future works, the methodology presented in this paper will be extended to the rest of abdominal organs present in an abdominal laparoscopic surgery.

Acknowledgements This project has been partially funded by MITYC (reference TSI-020100-2009-189).

References [1] H.J. Bonjer, Open versus closed establishment of pneumoperitoneum in laparoscopic surgery, Brithis Journal of Surgery 84 (1997), 599–602. [2] H. Shi, Validation of Finite Element Models of Liver Tissue Using Micro-CT, IEEE transactions on biomedical engineering, 55, 3 (2008), 978-984


351

Technology Transfer at the University of Nebraska Medical Center Kulia MATSUO, MD, MBAa,1 , Henry J. RUNGE, JDb, David J. MILLER, PhDc, Mary A. BARAK-BERNHAGEN, BSc, and Ben H. BOEDEKER, MD, PhDc a University of Nebraska Medical Center, Omaha NE b UNeMed Corporation, Omaha, NE c Dept. of Anesthesiology, University of Nebraska Medical Center, Omaha, NE Abstract: The course of developing a new product from an idea is a complicated process. This paper will discuss that process, in detail, from conception to product. We approach this by first discussing what the inventor must do begin the process of developing his or her idea, and then two pathways that occur simultaneously: the Technology Transfer process of patenting, marketing, and licensing the invention; and the engineering process of developing, modifying, and manufacturing the invention. Although the process is lengthy and most ideas never become a marketed product, there are those few ideas that do become realized into marketed products. Keywords. Technology transfer office, patent, prototype

Background The development process of an invention from idea to market is lengthy and complicated. There are multiple stages that occur when bringing a new idea to market and an idea can fail at each stage. Later stages of the technology transfer process are routinely referred to as “The Valley of Death.” The purpose of this work is to describe the general development of an invention from idea to product.

Methods The development of an invention begins with the conception of the idea arising from a perceived need. The inventor then submits a New Idea Notification (NIN) Application to the Technology Transfer Office (TTO). From there, the invention is evaluated and, based on an initial patentability and marketability analyses, the TTO may or may not decide to proceed to file a patent application and begin marketing the invention. If the TTO decides to proceed with the invention, a patent application, either provisional or non-provisional, is filed with the United States Patent and Trademark Office (USPTO). After processing and prosecuting the patent, the USPTO may decide to grant or not grant a patent. Throughout this process, a licensing associate at the TTO is assigned to

1 Kulia Matsuo, MD, MBA, 1633 North Capital Ave, MT, Suite 640, Indianapolis, IN 46202; E-mail: [email protected]

352

K. Matsuo et al. / Technology Transfer at the University of Nebraska Medical Center

manage the invention through its evaluation and marketing. The licensing associate develops non-confidential marketing materials about the invention, and contacts potential licensees. Typically at this point, the patent has not been granted yet, so the interested licensees must sign a Confidential Disclosure Agreement (CDA). If the licensee decides to pursue the invention, a license agreement with terms and a contract is developed regarding the manufacturing and selling of the product. While the TTO handles the protection and the marketing of the invention, continued engineering and development of the invention also occurs. Although the invention is beyond proof of concept, the invention needs additional testing to be marketable or a complete product. The inventor typically meets with an engineer to develop 3-D CAD models of the invention. Rapid prototypes are made to evaluate the geometry and shape of the invention. The inventor and engineer test and refine the prototype generating the next generation of prototypes. This process continues until the inventor and engineer feel the prototype is of high quality. The prototypes must undergo biocompatibility testing, and be FDA approved. During this time, a research protocol is generated and submitted to the university’s Institutional Review Board (IRB). After revisions to the prototype, a manufacturer is identified through the TTO or by the inventor. The manufacturing company makes working prototypes and sends it to the inventor and engineer for final approval. Once approved, the first production run occurs. The manufacturer may or may not be the same company that will market the device. Funding for the invention may come from different variables, such as established industrial partners or governmental grants.

Results Figure 1 shows a case study describing the development cycle of a video laryngoscope suction blade at the University of Nebraska. Other institutions’ procedures may vary. Idea: When intubating a trauma patient, the inventor noticed the airway was often obstructed with blood and secretions, making intubation difficult. Also when using a videolaryngoscope (VL) blade, fog created from the patient’s airway would obscure the camera’s view. The inventor thought that incorporation of suction into a VL blade would clear secretions. In addition, the suction could be dually used for either the insufflation of oxygen or drug delivery. A quick survey found no similar devices readily available on the market. Feasibility Study: Under FDA Exemption Sect. 21 CFR807.65D, the inventor tested the concept by taping a suction tube to the VL blade to clear secretions during intubation. This rough prototype was feasible and successful. Patenting & Marketing: New Invention Notification (NIN). Inventor initially disclosed the idea to his Technology Transfer Office (TTO) and suggested approaches to integrate both suction and insufflation capabilities into a variety of medical and industrial instruments. An initial search of the prior art uncovered a limited number of patents and patent publications in the same field as the inventor’s invention. Several of the approaches that the inventor took were substantially different than the other patents/patent applications. Protection & the USPTO. The inventor’s TTO filed a provisional patent application that included the inventor’s invention disclosure, data from the feasibility study and the specifications of prototypes that his laboratory had reduced to practice. With a filing date secure, the inventor was free to discuss the idea with commercial partners without creating any public disclosures. A year from the filing date of the provisional patent application, the inventor’s TTO would need to convert the provisional patent application to a utility application. During this year, market interest was assessed, and the TTO further solidified the patent position while also drafting the utility application. Marketing & Market Interest. After discussing the project with several companies, the inventor identified multiple partners that he could move forward in developing the invention. One partner, a mid-sized medical device firm, expressed interest in licensing the patent application filed by the inventor’s TTO. License Agreement & Commercialization. The mid-sized medical device firm negotiated a license agreement with the inventor’s TTO and became a licensee. The license contained standard terms for an academic institution. With the protection of the patent, the licensee could launch the

K. Matsuo et al. / Technology Transfer at the University of Nebraska Medical Center

353

product into a protected market. The inventor’s innovation would not only be sold as one of the licensee’s products, but it would also have the protection of the exclusive rights for over the life of the patent. Given the standard terms of the license, and as required under Bayh Dole, the University of Nebraska, the Department of Veterans Affairs, and the internal intellectual property policies of these institutions, the inventor’s lab and the inventor himself would all share in the upside of the invention. Engineering & Development: Using information generated from the rough prototype (which consisted of a disposable suction tube being taped to a video laryngoscope blade), a 3D model of the suction blade and disposable suction catheter was generated by computer-assisted design (CAD). Protocol. A protocol was written to develop and test the suction blade. Reduction to Practice. The inventor confirmed multiple embodiments of the device were effective in order to reduce the invention to practice and demonstrate that the concept was viable. Based on the 3D model and the inventor’s rough prototype, the suction blade was modeled by taking a stainless steel laryngoscope blade and cutting a channel from stainless steel tubing for containment of the suction catheter. The channel was glued onto the blade (an example of low cost prototype development). Visual Inspections. Based on visual inspection of these rapid prototypes, it was determined that this channel would be too sharp to be used on a patient; therefore, the next generation prototype was developed with the edge blunted. The cycle of visual inspections of the prototype and changes to the blade and catheter resulted in nearly a dozen generations of prototypes, after which a fully functional prototype blade was made by a large international endoscopy company. These prototypes were strong enough to be tried on mannequins to test the function of the suction blade and catheter during intubation. To accomplish further testing, a simulator was designed to adequately reproduce a severely traumatized upper airway. After the IRB accepted the protocol and signed non-disclosure agreements, anesthesia providers tested the prototyped blade on mannequins to test the effectiveness and user-friendliness of the device. Based on the tests, further alterations were made to the suction blade. Companies Produce Prototype. The inventor had previous relationships with a major endoscopy company that produces video laryngoscope systems, as well as a variety of local engineering and research firms. Utilizing these collaborators, the inventor developed a more refined prototype of the blade and a more refined rapid prototype of the disposable suction tube. After each company received drawings and the rough prototypes of the suction blade, they made rapid prototypes which were then sent to the inventor and engineer. In addition, the licensee manufactured the integrated catheter, which was developed in parallel to the laryngoscope blade, so the prototypes and drawings were also sent to this company. Final Prototype. The inventor tested the prototypes as they became available, made changes, and passed the information back to each respective company for further revisions. These revisions were used to develop the next generation prototype. This cycle continued until a final prototype of both the blade and the catheter was agreed upon. Biocompatibility Testing. The plastic used in the suction catheter was sent for biocompatibility testing by a third-party testing company once sufficient design iterations had been done such that the developers were relatively certain they had a working design. In this case, the VL blade was exempt because it was made from stainless steel, and the biocompatibility was already known and it was a modification to an existing, validated device.FDA Approval. The licensee compiled the design history file, including documents from the inventor and his collaborators, to submit to the FDA for review. Each alteration to the blade and catheter must be noted; the biocompatibility and bioburden testing of the licensee’s facilities must also be submitted. Based on the licensee’s analysis, the catheter was exempt from 510(k) premarket approval because it was a Class I device. The VL blade was Class II. Since none were Class III, the premarket approval application did not need to be filled out. The Medical Device Listing, Medical Device Labeling, and Good Manufacturing Practices applications also needed to be filled out by the manufacturers. Manufacturing. The international endoscopy company is currently preparing to manufacture the blade. The licensee is preparing tooling for manufacturing the suction catheter. Sales. Currently, the blade and the catheter will be sold separately. A potential partnership between the international endoscopy company and the licensee could allow the blade and the catheter together. Alternatively, the two companies may decide to develop symbiotic products that are not mutually exclusive, meaning the blade will work without the catheter and the catheter will work without the blade. However, the suction blade will be marketed in catalogues in the near future. Figure 1. Case Study: development cycle of a novel video suction blade at the University of Nebraska

Conclusions Many inventions begin as great ideas, but throughout the rigorous development process, very few inventions are actually commercialized. Although the process is long and complicated, all medical innovations must occur in a process similar to the one described. The “Valley of Death” is a good depiction of this developmental process.

354


CvhSlicer: An Interactive Cross-Sectional Anatomy Navigation System Based on High-Resolution Chinese Visible Human Data Q. MENG a , Y.P. CHUI a , J. QIN a,c , W.H. KWOK b , M. KARMAKAR b , P.A. HENG a,c a Dept. of Computer Science & Engineering, The Chinese University of Hong Kong b Dept. of Anaesthesia & Intensive Care, The Chinese University of Hong Kong c Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences Abstract. We introduce the design and implementation of an interactive system for the navigation of cross-sectional anatomy based on Chinese Visible Human (CVH) data, named CvhSlicer. This system is featured in real-time computation and rendering of high-resolution anatomical images on standard personal computers (PCs) equipped with commodity Graphics Processing Units (GPUs). In order to load the whole-body dataset into the memory of a common PC, several processing steps are first applied to compress the huge CVH data. Thereafter, an adaptive CPU-GPU balancing scheme is performed to dynamically distribute rendering tasks among CPU and GPU based on parameters of computing resources. Experimental results demonstrate that our system can achieve real-time performance and has great potential to be used in anatomy education. Keywords. CvhSlicer, Anatomy navigation system, CPU-GPU balancing scheme

1. Introduction Compared with radiological images (e.g. CT/MR and ultrasound images), Visible Human (VH) data [1,2] is of unique value in research and education of human anatomy because of the fine anatomical details presented in the high-resolution cross-sectional photos. Developing an interactive navigation system to visualize the cross-sectional anatomy in VH data, however, is a challenging task, especially on a standard personal computer (PC). The system, which should allow real-time navigation of the cross-sectional anatomy, is challenging due to the huge size of VH data and limited computation resources. Several navigation systems have been developed based on the data achieved from the Visible Human Project, such as the Real-time Slice Navigator [3] and the virtusMed [4]. The main limitation of these two systems is that the resolution of rendered sections is relatively low, which is not able to present small but important anatomical structures.In this paper, we aim to develop an interactive navigation system for high resolution Chinese Visible Human (CVH) data, which runs on common PCs equipped with commodity

Q. Meng et al. / CvhSlicer: An Interactive Cross-Sectional Anatomy Navigation System

Figure 1. An overview of the CvhSlicer system.

355

Figure 2. VH data compression. (a) Compression steps. (b) Cut Tree for bounding box.

Graphics Processing Units (GPUs). In order to load the whole-body data to a PC memory, compression techniques are applied on the huge high-resolution CVH data, without compromising anatomical details. Given time-critical interaction requirement and large size of the CVH data, an adaptive load-balancing scheme between GPU and CPU is implemented to maximize the performance of the system. Experiments demonstrate that users can interactively observe the cross-sectional anatomy from any given position and direction in a real-time manner.

2. System Overview The proposed system is designed with two main modules, namely VH data compression module and slice pixels computation module (see Figure 1). As frequent data exchange between the main memory and hard disk is too time-consuming for the system to achieve real-time performance, an alternative approach is to load all data to the main memory. However, this is not possible when the data size is very large. Generally, the size of VH data can be up to dozens, even hundreds, of GB. Thus, data compression is indispensable. In the proposed system, an almost lossless image compression procedure is first adopted to compress the VH data, and then the compressed VH data is fully loaded to the main memory for subsequent slice pixels computation. In some cases, even after compression, the data for rendering some large images is still too large to be loaded into the memory of a commodity GPU. To the end, we design a computation balancing method to make use of the computational power of both GPU and CPU for resultant image rendering.

3. Visible Human Data Compression Visible human images, which are photos of transverse section of a human cadaver, are captured with ultra-fine resolution (up to 100 microns). In order to retain most of the informative anatomical detail, it is necessary to compress the data and load it to main memory of a PC. In our framework, the compression is performed in three steps: (i) Simple scaling, (ii) Cut Tree with bounding boxes and (iii) DXT algorithm (as shown in Figure 2(a)). Visible human images are scaled firstly. Then, as shown in Figure 2, there are large areas of background in a typical VH image. Bounding boxes are used to detect useful blocks (i.e., foreground regions) and only the pixels in these blocks will be loaded to the main memory. In order to obtain bounding boxes that do not overlap with each other, we

356


defined two cutting operations: Column Cut and Row Cut, which are applied on a VH image alternately and recursively to form a Cut Tree. If neither Column Cut nor Row Cut can cut (i.e., separate) an image block, the cutting operations are terminated. Finally, the leaves of the Cut Tree are the bounding boxes we need. Figure 2(b) is a demonstration of the Cut Tree. Finally, the DXT algorithm [5] is used to perform a further compression. The DXT algorithm is advantageous in its low quality loss and the capability of preserving image details. More importantly, decompression of DXT is very fast and has little effect on cutting slice pixels calculation. After the three steps compression, VH data is compressed to about 1GB, which can be easily loaded into a PC main memory.

4. Slice Pixels Calculation Based on the compressed VH data being loaded into the main memory, the color of each pixel on the slice can be calculated. For a single pixel, it is easy to find out the corresponding “voxel” located in the “VH volume”. In order to get smooth images, trilinear interpolation is used for interpolating the 6 voxels around the target pixel. Since each pixel on the slice can be calculated independently, thus the whole slice can be calculated in parallel. In order to obtain a real-time performance, the computation task is distributed to both CPU and GPU based on parameters of computing resources such as the GPU’s memory size and the number of CPU cores. 4.1. CPU-GPU Cooperative Computation Framework In GPU-based computation, only memory on the display card can be accessed. The VH data being loaded to the main memory has to be copied to the display card memory for computation. However, the memory on the display card is usually not big enough to store the whole compressed VH data. In other words, for some large slices, GPU can not compute all the pixels at once. Since the CPU and GPU computation can be performed in parallel, the computation task can be accomplished cooperatively by CPU and GPU. Initially, the compressed VH images are loaded to a PC main memory from the hard disk. Afterwards, the VH images around the initial slice are loaded to the display card memory from PC main memory. The loaded data size depends on the size of the display card memory. Once the initial loading finishes, users can move and rotate the slice freely by using a Virtual Reality (VR) Input Device. The CPU computation starts as soon as the computation balancing finishes. In GPU, unused VH images are first deleted from the display card memory. In order to enable the GPU to compute as many pixels as possible, an adaptively determined number of VH images are then copied to display card memory from PC main memory. Results from both CPU and GPU are rendered to the screen directly. 4.2. CPU-GPU Computation Balancing Method In CPU, pixels are calculated on multiple cores in parallel. According to Amdahl’s Law nC tC , where [6], ideally, the time for computing all the pixels in parallel is: TC (nC ) = P C TC is the computational time for one image on CPU, nC is the number of pixels calcu-


357

lated on CPU, PC is the number of CPU cores, and tC is the computation time for one pixel. However, in practice, as parallel overhead [6] has to be considered, it requires additional time to make the sequential program run in parallel. Hence, a more accurate representation of the total CPU computation time should be: 1 TC (nC ) = OC · PC + nC tC (1) PC The parallel overhead is denoted by OC · PC where OC is assumed to be a constant. In practice, OC and tC can be calculated offline. We run the program in serial and an average computing time for one pixel can be obtained, and it is tC . On the other hand, the parallel overhead constant OC can be obtained by running the program in parallel. To obtain the GPU calculation time, we also need to consider the time of deleting unused VH images and the time of copying useful images into the GPU memory, which can be calculated as: q s 1 nG tG S(i) + B S(i) + OG · PG + (2) TG (nG ) = A PG i=p i=r where S(i) is the size of the i-th VH image’s size. p, q, r, s are images indices, indicating images within [p, q] will be deleted from the display card memory, and images within [r, s] will be copied from the PC main memory to the display card memory. A and B are coefficients. In other words, the first term in Eq.(2) corresponds to the time of “deleting unused data from the GPU memory”; the second term corresponds to the time of “copying data to the GPU memory”; and the third term of Eq.(2) corresponds to the time of “GPU computation”. In Eq.(2), the values of tG , OG , A and B can be determined offline. In our GPU computation model, each pixel is computed on a single thread and the average thread running time can be obtained as tG . OG can be calculated in the same way as OC . We pre-compute the deleting and copying time for various numbers of slices in order to estimate A and B offline. p, q, r, s are calculated in runtime, i.e., during the calculation balancing stage. The objective of the computation balancing is to minimize: max {TC (nC ) , TG (nG )}

(3)

Since most parameters in Eq.(1) and Eq.(2) can be calculated offline, the computation balancing aims at optimizing p, q, r, s in order to minimize the total computational time. A binary search algorithm [7] is developed to search the optimal values of them.

5. System Implementation and Experimental Result The CvhSlicer System was developed according to the architecture illustrated in Figure 1. As shown in Figure 3, there are two screens in the system – one for displaying the 3D CVH model, and the other for displaying the cross-sectional images. Users can trans-

358


200

Computation Time (ms)

150

CPU−Serial CPU−Parallel CPU−GPU−Balance

100

50

0 100x100

Figure 3. A photo of the CvhSlicer system.

200x200

400x400 Slice Size

800x600

1200x800

Figure 4. Performance Comparison.

late, rotate and scale the 3D CVH model by mouse control. A SensAble Phantom Omni is used to manipulate a multi-planar reconstruction (i.e., translate or rotate the cutting plane) on CVH model. Tactile output can also be provided in form of a feedback force rendered on the Omni in order to enhance realism of our navigation system. We have tested CvhSlicer with the Chinese Visible Human Female (CVHF) dataset, which was obtained at 0.5mm intervals. There were 3640 slices in the CVHF dataset, each with 3872 × 2048 pixels, and the total size is about 71.6GB. We first rescaled the data to 1/3 of the original size, which was further compressed to 1.13GB. Experimental results demonstrated that, for different sizes of slice, the system can still give real-time rendering. Figure 4 shows a comparison of the performance among CPU-Serial, CPU-Parallel, and the proposed CPU-GPU-Balance computation modes. All the experiments were performed on a PC with a Intel(R) Core(TM) 2 CPU, 2.66GHz, and a GPU of Nvidia Geforce 8800 GTX. From this comparison, it is observed that the CPU-GPU-Balance model achieved the best performance and the fastest rendering time for different sizes of slice. Acknowledgements The work described in this paper was fully supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region (Project No.CUHK412510). References [1] M. J. Ackerman, “The visible human project: a resource for education”, J. Acad. Med. Jun. 1999, vol. 74, pp. 667-670 [2] S. X. Zhang, P. A. Heng, “The Chinese Visible Human (CVH) datasets incorporate technical and imaging advances on earlier digital humans”, J. Anat, March 2004, vol. 204, pp. 165-173 [3] R.D. Hersch, B. Gennart, O. Figueiredo, M. Mazzariol, J. Tarraga, S. Vetsch, V. Messerli, R. Welz, L. Bidaut, “The Visible Human Slice Web Server: A first Assessment”, Proceedings IS&T/SPIE Conference on Internet Imaging, San Jose, Ca, Jan. 2000, SPIE vol. 3964, pp. 253-258 [4] virtusMED http://www.umi.cs.tu-bs.de/virtusmed [5] L. Renambot, B. Jeong, J. Leigh, “Real-Time Compression For High-Resolution Content”, Proceedings of the Access Grid Retreat 2007, Chicago, IL, May, 2007 [6] C. Barbara, Using OpenMP : Portable shared memory parallel programming, The MIT Press, 2008, pp. 33-34 [7] S. G. Akl, H. Meijer, “Parallel Binary Search”, IEEE Transactions on Parallel and Distributed Systems, April 1990, vol. 1, no. 2, pp. 247-250 [8] NVIDIA Corporation, NVIDIA CUDA Programming Guide, 3rd ed. Santa Clara, CA. 2009


359

Generation of Connectivity-Preserving Surface Models of Multiple Sclerosis Lesions Oscar Meruvia-Pastora, Mei Xiaoa,1, Jung Soha, Christoph W. Sensena a SunCenter of Excellence for Visualgenomics Department of Biochemistry and Molecular Biology Faculty of Medicine, University of Calgary 3330 Hospital Drive NW Calgary, Alberta, T2N 4N1, Canada

Abstract. Progression of multiple sclerosis (MS) results in brain lesions caused by white matter inflammation. MS lesions have various shapes, sizes and locations, affecting cognitive abilities of patients to different extents. To facilitate the visualization of the brain lesion distribution, we have developed a software tool to build 3D surface models of MS lesions. This tool allows users to create 3D models of lesions quickly and to visualize the lesions and brain tissues using various visual attributes and configurations. The software package is based on breadth-first search based 3D connected component analysis and a 3D flood-fill based region growing algorithm to generate 3D models from binary or non-binary segmented medical image stacks. Keywords. Multiple Sclerosis, Lesions, Connected Components, Segmentation

1.

Introduction

Medical image slices are frequently reconstructed into a 3D surface model to facilitate realistic visualization. Such reconstruction usually results in a single 3D model built from all regions of interest in a stack. When the target object is an anatomically welldefined structure, the reconstructed 3D model can be explored as is. However, when the reconstructed model represents a randomly distributed collection of objects with similar characteristics, such as brain lesions formed by the progression of multiple sclerosis (MS) ([1, 2]), it is useful to select certain clinically meaningful portions of the model to investigate in more detail. Topological information contained in 3D image stacks is very useful. For example, instead of visualizing all the lesions together, users might want to distinguish each individually connected lesion from the whole model. It will be useful for the users to visualize the selected brain lesions and other brain tissues such as the cortex and subcortex with different visual attributes such as color and transparency. Connectivity can play a key role in these cases and we need well-connected models in order to

1

Corresponding Author: [email protected]

360

O. Meruvia-Pastor et al. / Generation of Connectivity-Preserving Surface Models of MS Lesions

separate them easily and quickly. Therefore, a well formatted surface mesh structure is needed. The goal of our tool is to quickly and accurately create 3D models from a segmented medical image stack, such that connected 3D regions are easily separable from the whole model. In this case, the users can easily select the lesions and related brain tissues and view them in any visual configurations they want. There are two main algorithms that can be used to create 3D models of MS lesions from MRI stacks: the marching cubes algorithm and its various extensions ([3-9]), and region-growing based surface creation algorithms ([10-13]). Most marching cubes algorithms work on binarylabeled medical image stacks. However, to study the dynamics of MS lesion changes, it is important to use non-binary segmented image stacks that can store both lesion and brain tissue information. Region growing algorithms can work on non-binary segmented stacks. We have developed a method for quickly generating surface models from a medical image stack, which can preserve connectivity of voxles such that connected components of a model can be easily selected and visualized. Our method uses breadthfirst search based 3D connected component analysis and 3D flood-fill based region growing.

2.

Materials and Methods

MS patients were recruited for MRI scanning in Halifax, Nova Scotia, Canada. Each patient has been scanned six times at an interval of once a month. T1, T2, and T2 FLAIR MRI images were obtained for use in our model creation tool. A sequence of image processing steps was performed on the head MRI data generated from each scan session, to convert them to input image data for our software. First, the brain area was separated from each scan by using the FMRIB (Functional MRI of the Brain) Software Library (FSL, http://www.fmrib.ox.ac.uk/fsl). This step was performed on the T1 image set. Second, after brain extraction from the images, for each patient, all the scans were registered together. This step was required in order to make meaningful comparisons when visualizing changes in MS lesions of a patient from one scan session to the next. Third, we performed segmentation of the MRI scans to retrieve these types of tissues: cerebrospinal fluid (CSF), cortex, and sub-cortex. Finally, MS lesions were manually segmented from the FLAIR images based on white matter hyperintense areas. The T2 images were consulted frequently, since lesions that were not as clearly bright on FLAIR could be quite apparent on T2. T2 images were also used for confirming lesions identified on FLAIR. The pipeline of the model creation program consists of two major processing steps (see Figure 1): •

A stack of 2D images that contains the MS lesions goes through 3D connected component analysis. The breadth-first search based 3D connected component analysis will separate the MS lesions into several connected components and a seed for each component is saved into a text file.

•

The stack of 2D images in the previous step and the text file that contains the seed information are used in the flood-fill based region growing algorithm. A 3D model that contains all the MS lesions is created and saved into an OBJ file.


361

Figure 1. Procedure for building the lesion models.

The flood-fill based region growing algorithm starts by finding the boundary of the region beginning from the seed. Voxels of intensity within a certain range from the seed are searched for until the boundaries are met. For each boundary voxel with a coordinate (x, y, z), eight surrounding points (x ± 0.5, y ± 0.5, z ± 0.5) are used to created faces. Those points are used as the vertices for the mesh. Each surrounding point is uniquely indexed according to the voxel coordinate (x, y, z). In order to avoid using duplicated vertices, the index and positions of those surrounding points are saved in a hash table. For each boundary voxel, two triangles are created by traversing the surrounding points clockwise in order to guarantee a closed and consistent mesh. The final mesh vertices are saved in an indexed triangle array. At the end we have a mesh writer to save the contents of the indexed triangle array into an OBJ file, which can be display by most 3D mesh visualization and processing tools. The resulting surface might have obvious staircase effects, where a smoothing filter can be applied to each voxel to lessen the effects. The number of smoothing steps can be specified by users.

3.

Results

We have created a Java3D™-based 3D medical image processing and visualization software package for facilitating brain lesion studies such as those required in MS. Although there are many automatic image segmentation programs developed for handling medical images, MS lesions usually have fuzzy boundaries (see Figure 2), and hence they still need to be manually segmented by neuroscience experts. The segmented images usually contain information on multiple brain tissue types. Each tissue type may include pixels in certain intensity value ranges, resulting in non-binary image stacks. Our program can be used to detect the 3D connected components in those

362


non-binary segmented images. Once the users set up the intensity range for the tissue type they are interested in detecting, a breadth-first search algorithm is applied to the image stack, recursively searching the 26 surrounding pixels in an image stack. After the search, the total number of connected components found and the starting pixel of each component will be saved in a text file. If the users are interested, it is also possible to record the volume of each component and even the positions of pixels comprising each component.

Figure 2. A slice from the MRI scan of an MS patient. Highlighted by the circles are inflammation areas.

The locations and volumes of MS lesions are of great significance for clinical studies to understand the damage to the brain tissues and cognitive abilities of the MS patients. By flood-fill based region growing from a pixel in an image stack, the boundaries of a separated MS lesion can be detected and triangular faces can be used to approximate the shape of that specific lesion. The mesh models that are created (see Figure 3) can be loaded into a 3D visualization program. By displaying and manipulating the visual attributes of 3D models for different types of brain tissues and lesions, MS lesion development patterns can be investigated in the context of surrounding anatomical structures. Polygonal mesh models also have an advantage for lesion pattern studies. Unlike volume rendered models, the inner structures of polygonal models allow us to start from one vertex and traverse all the connected vertices to extract the 3D connected component and visually distinguish it from the rest of the model, in real time. For example, in our software package, we have a simple interface that allows the user to double click on a polygonal model, such that the single lesion that is connected to the clicked point can be extracted from the whole lesion model and displayed in another visualization window (see Figure 4). The connectivity calculation required for this component separation feature is performed by using the vtkPolydataConnectivityFilter class from VTK ([14, 15]).


363

Figure 3. Rendering of a cortex model generated by our surface model building method (left) and rendering of cortex and MS lesion models generated by our surface model building method (right).

Figure 4. A connected lesion can be selected and highlighted by double clicking on it.

4.

Discussion

3D models reconstructed from medical image stacks such as MRI and CT have been widely used in diagnosis and medical research. The marching cubes algorithm can be used to create high-resolution 3D mesh models. However, the marching cubes algorithm creates at least one triangle per voxel while it passes through the surfaces, which creates a huge amount of triangles ([13]). Depending on the implementation of the marching cubes algorithm, the mesh might not be consistent with the actual layout of the voxels. For example, Figure 5 shows two screen shots after applying vtkPolydataConnectivityFilter to two different mesh models. They both were generated from the same image stack (one MRI stack of MS lesion images). The left model was rendered using our approach. The right model was rendered using the vtkMarchingContourFilter. We can see that on the left image, once we double click on a lesion, the whole connected lesion is selected and its visual attributes can be changed as one single model. However, on the right image, once we double click on a lesion,

364


only part of the lesion is selected as shown by the changed color, and the rest of the same lesion has not been selected as shown by the same color as the rest of the whole model. If the connectivity of mesh models cannot be fully detected, the developmental patterns of the MS lesions can not be accurately expressed.

Figure 5. Comparison of connectivity in surface models. The model on the left is created using our surface model building software. The selection can clearly separate a connected component from the whole model. On the right, the model is created using the VTK marching cubes algorithm. The selection retrieves only part of a connected component from the whole model.

Marching cubes algorithms are usually applied to a whole image stack and every voxel will be checked. Therefore, for MRI and CT image stacks, it usually takes a long time to create a high-resolution mesh. Some versions of the marching cube applications also try to find the connected components, but processing a high-resolution mesh to find the connected components is very time-consuming and needs a large amount of computing resources. We tested a marching cube based program Afront ([16]) on the same MS lesion image stack. It took longer time and did not create a similar mesh as shown in Figure 5. Lewiner et al (2003) ([5]) developed a software package to create topologically correct manifold mesh. We tested the C++ implementation of the program. For any implicit functions applied to a grid, the resulting surface is consistent and the quality of the mesh seems to be very good. However, they did not provide any mesh creation method from medical image stacks. Therefore, the rendering time for our MS lesion image stacks in their program is not known. Our software provides a valuable tool to study disease development by creating 3D models of the observed patterns in medical images. By building consistent mesh models quickly and efficiently directly from a medical image stack, researchers can create a large number of models that represent different individuals and different developmental stages of a disease. The ability to retrieve connected components from such a model is a key feature of our software which enables the user to effortlessly focus on a clinically important part from the whole image stack. In addition to double clicking a certain potion of the model to highlight the region of interest, other cutting-based model creation tools will add more flexibility to our method. For example, Xiao et al (2010) ([17]) provided a generic model building


365

algorithm. By using a virtual dissection method, various models can be built quickly and efficiently. Together with our current model building tool, a highly flexible virtual dissection and selection based biological scene creating tool can be developed for studies and discoveries of disease patterns.

Acknowledgement This work has been supported by Genome Canada through Genome Alberta; Alberta Science and Research Authority; Western Economic Diversification; the Governments of Canada and of Alberta through the Western Economic Partnership Agreement; the iCORE/Sun Microsystems Industrial Research Chair program; the Alberta Network for Proteomics Innovation; and the Canada Foundation for Innovation. We thank Heather Angka, Carl Helmick, Jordan Fisk and John Fisk for MRI data acquisition and processing. We also thank Megan Smith for comments on the manuscript.

References [1] M. Inglese, R. I. Grossman, M. Filippi, Magnetic resonance imaging monitoring of multiple sclerosis lesion evolution, Journal of Neuroimaging 15(4 Suppl) (2005), 22S-29S. [2] F. Zipp, A new window in multiple sclerosis pathology: non-conventional quantitative magnetic resonance imaging outcomes, Journal of the Neurological Sciences 287(1 Suppl) (2009), S24-S29. [3] H. C. Hege, M. Seebass, D. Stalling, M. Zöckler, A generalized marching cubes algorithm based on non-binary classifications, Konrad-Zuse-Zentrum für Informationstechnik Berlin Technical Report SC97-05 (1997). [4] W. E. Lorensen, H. E. Cline, Marching cubes: a high resolution 3D surface construction algorithm, ACM SIGGRAPH Computer Graphics 21(4) (1987), 163-169. [5] T. Lweiner, H. Lopes, A. W. Vieira, G. Tavares, Efficient implementation of Marching Cubes’ cases with topological guarantees, Journal of Graphics Tools 8(2) (2003), 1-15. [6] G. M. Nielson, Dual marching cubes, In Proc. IEEE Conf. Visualization (2004), 489-496. [7] G. M. Nielson, On marching cubes. IEEE Transactions on Visualization and Computer Graphics 9(3) (2003), 283-297. [8] S. Raman, R. Wenger, Quality isosurface mesh generation using an extended marching cubes lookup table, Computer Graphics Forum 27(3) (2008), 791-798. [9] S. Schaefer, J. Warren, Dual marching cubes: primal contouring of dual grids, In Proc. 12th Pacific Conf. Computer Graphics and Applications (2004), 70-76. [10] M. del Fresno, M. Venere, A. Clausse, A combined region growing and deformable model method for extraction of closed surfaces in 3D CT and MRI scans. Computerized Medical Imaging and Graphics 33(5) (2009), 369-376. [11] I. Cohen, D. Gordon, VS: a surface-based system for topological analysis, quantization and visualization of voxel data, Medical Image Analysis 13 (2009), 245-256. [12] B. Reitinger, A. Bornik, R. Beichel, Consistent mesh generation for non-binary medical datasets, In: Bildverarbeitung für die Medizin (2005), 183–187. [13] Y. Xi, Y. Duan, A novel region-growing based iso-surface extraction algorithm, Computers & Graphics 32(6) (2008), 647-654. [14] S. Pieper, B. Lorensen, W. Schroeder, R. Kikinis, The NA-MIC Kit: ITK, VTK, Pipelines, Grids and 3D Slicer as an open platform for the medical image computing community, In: Proc. 3rd IEEE Int. Symp. on Biomedical Imaging (2006), 698-701. [15] W. Schroeder, K. Martin, B. Lorensen, The Visualization Toolkit, Prentice-Hall, 2006. [16] J. Schreiner, C. E. Scheidegger, C. T. Silva, High-quality extraction of isosurfaces from regular and irregular grid, IEEE Transactions on Visualization and Computer Graphics 12(5) (2006), 1205-1212. [17] M. Xiao, J. Soh, O. Meruvia-Pastor, E. J. Schmidt, B. Hallgrimsson, C. W. Sensen, Building generic anatomical models using virtual model cutting and iterative registration, BMC Medical Imaging 10(5) (2010).

366


A Comparison of Videolaryngoscopic Technologies David J. MILLER, PhDa,b, Nikola MILJKOVICc,d, Chad CHIESAc,d, Nathan SCHULTEc,d, John B. CALLAHAN, Jr. BSd, and Ben H. BOEDEKER, MD, PhDa,b,1 a Department of Anesthesiology, University of Nebraska Medical Center, Omaha, NE b Research Service, Omaha VA Medical Center, Omaha, NE c College of Information Science &Technology, University of Nebraska, Omaha, NE d The Peter Kiewit Institute and Complex, Omaha, NE

Abstract. Difficulty in managing the airway is a major contributor to the morbidity and mortality of the trauma patient. The video laryngoscope, with its camera at the distal tip of the intubation blade, allows the practitioner an improved view of the glottic opening during intubation. The image from this viewer is transmitted to a monitor, allowing the intubating practitioner to “see around the corner” of a patient’s airway. The purpose of the present study was to assess and compare the video quality of commercially available video laryngoscopy systems. It was found that between the STORZ C-MACTM and the Verathon GlideScope®, there was little difference between the video quality; the difference came down to user preference. Keywords. Video laryngoscope, intubation

Background Inadequate airway management is a major contributor to patient injury, morbidity and mortality [1-2]. Indirect laryngoscopy provides a method to improve the view of the glottic opening during intubation. The video laryngoscope has a camera or lens at the distal tip of the intubating blade. The image from this viewer at the distal tip is transmitted to a monitor. This permits the intubating practitioner to “see around the corner” during intubation [3]. The purpose of the study was to analyze commercially available laryngoscope products to determine functionality, ease of use and feature sets.

Methods & Materials A team of four investigators at the University of Nebraska (Omaha) and the Peter Kiewit Institute (Omaha, NE) performed simulated intubations using a number of video laryngoscopy systems. The analyzed systems included the GlideScope Portable GUL (Verathon Inc., Bothell, WA) (Figure 1), a prototype system developed by STORZ as a predecessor to their C-MAC™ (a standard STORZ Macintosh blade with USB

1

Corresponding Author: Ben H. Boedeker, MD, PhD, Professor, Department of Anesthesiology, Director, Center for Advanced Technology & Telemedicine, University of Nebraska Medical Center, 984455 Nebraska Medical Center, Omaha, NE 68198-4455, USA; E-mail: [email protected]

D.J. Miller et al. / A Comparison of Videolaryngoscopic Technologies

367

connectivity to an ultra mobile PC; “UMPC”) (Figure 2), and the STORZ C-MAC™ (KARL STORZ Endoscopy, Culver City, CA) (Figure 3).

Figure 1. The Verathon GlideScope®

Figure 2. Storz prototype to the CMAC™

.

Figure 3. The Storz CMAC™ (photo courtesy of KARL STORZ Endoscopy-America, El Segundo, CA).

Testing was performed with a Laerdal Difficult Airway Trainer™ (Laerdal Medical Corporation, Wappingers Falls, NY) in a setting that simulated difficult airways, adverse lighting conditions and various system configurations (e.g. maximizing screen contrast, minimizing screen brightness, maximizing screen color hue, etc.). The equipment was assessed based on the investigator’s perceptions of color, clarity and brightness of the onscreen image for each of the systems. Perceptions were

368

D.J. Miller et al. / A Comparison of Videolaryngoscopic Technologies

given one of three possible ratings: 1=High, 2=Moderate or 3=Low. The statistics were performed using a two-tailed Wilcoxon Rank Sum test for independent samples.

Results A summary of the test results are shown in Table 1. Statistical analysis showed that there was no statistical differences between image, clarity, color, brightness or overall score of any of the systems tested (=0.05).

Table 1. Summarized Results of Tested Video Laryngoscopes System

CMAC

GlideScope

UMPC

# of scenerios Clarity Color Brightness Ave of Total Score

8 2.13 1.75 2.50 6.38

8 2.38 1.38 2.38 6.13

8 1.88 1.75 1.88 5.50

SD

2.50

1.96

2.20

Conclusions Results showed that there were no significant differences in video quality between the three systems; thus, the choice of systems falls to user preference (which can vary from person to person) and qualitative analysis of features that are outside the scope of this study. Future investigations are planned to evaluate additional videolaryngoscopy solutions in an effort to create a platform-agnostic videolaryngoscopy suite.

References [1] [2] [3]

L. Hussain, A. Redmond. Are pre hospital deaths from accidental injury preventable? Br Med J 308 (1994), 1077-1080. C.G. Miller. Management of the difficult intubation in closed malpractice claims. ASA Newsletter 64 (2000), 13-16 & 19. B.H. Boedeker, S. Hoffman, W.B. Murray. Endotracheal intubation using virtual images: learning with the mobile telementoring intubating video laryngoscope. Stud Health Technol Inform 125 (2007): 4954. Published by IOS Press.


369

Telemedicine Using Free Voice over Internet Protocol (VoIP) Technology David J. MILLER, PhDa,b, Nikola MILJKOVICc,d, Chad CHIESAc,d, John B. CALLAHAN, Jr., BSd, Brad WEBB, MPASa,b, and Ben H. BOEDEKER, MD, PhDa,b,1 a Department of Anesthesiology, University of Nebraska Medical Center, Omaha, NE b Research Service, Omaha VA Medical Center, Omaha, NE c College of Information Science & Technology, University of Nebraska, Omaha, NE d The Peter Kiewit Institute and Complex, Omaha, NE

Abstract. Though dedicated videoteleconference (VTC) systems deliver high quality, low-latency audio and video for telemedical applications, they require expensive hardware and extensive infrastructure. The purpose of this study was to investigate free commercially available Voice over Internet Protocol (VoIP) software as a low cost alternative for telemedicine. Keywords. Telemedicine, Voice over laryngoscope, VTC, C-MAC, Skype™

Internet

Protocol

(VoIP),

video

Background There has been a recent increase in telemedicine projects, ranging from instruction and distance mentoring to performance of medical procedures [1-3]. These require a large amount of Internet bandwidth and expensive hardware to perform. Voice over Internet Protocol (VoIP) uses existing Internet infrastructure to transmit voice/video images over a distance. Though the technology has been used previously to provide medical care to patients far separated from definitive care centers, the systems developed were designed for the specific user requirements and took time to develop and perfect. The purpose of this study was to analyze the feasibility of using a commercially available VoIP technology to provide telemedical care.

Material & Methods Researchers at the University of Nebraska Medical Center (UNMC) and the Peter Kiewit Institute (PKI) downloaded the free version of SkypeTM, a VoIP software package that allows full duplex video and audio communication between users (www.skype.com). PKI used a typical PC on a wired LAN with 2GB of RAM, a 2.66 GHz Intel® Core™ 2 Duo processor running Windows XP Professional and a

1 Corresponding Author: Ben H. Boedeker, MD, PhD, Professor, Department of Anesthesiology, Director, Center for Advanced Technology & Telemedicine, University of Nebraska Medical Center, 984455 Nebraska Medical Center, Omaha, NE 68198-4455, USA; E-mail: [email protected]

370

D.J. Miller et al. / Telemedicine Using Free Voice over Internet Protocol (VoIP) Technology

microphone with external speakers, while the UNMC researcher (located in Long Beach, CA) used a notebook with an Intel® CoreTM 2 Duo CPU (2.20GHz) with 1.99 GB of RAM, integrated stereo speakers, an integrated 1.3 mega pixel web camera with digital microphone, and an AT&TTM 3G cellular broadband access card, connected at 3.6 MBps with a signal strength of -92dBm. Instead of a web camera, PKI used a CMAC™ video laryngoscope blade (KARL STORZ, Tuttlingen, Germany) with an NTSC/USB conversion module. The second setup was between two offices at UNMC, both on the wired LAN. The first computer was the notebook listed above (UNMC 1), while the other was a notebook with an Intel® CoreTM 2 Duo CPU (2.40GHz) 4.00 GB of RAM, and integrated stereo speakers (UNMC 2). Each user was using a MicrosoftTM LifeCam VX-5000 1.3 megapixel web camera with integrated digital microphone. The web camera was pointed at the C-MAC screen and video was transmitted without direct connection to the source computer. In both trials, one participant (PKI and UNMC 1, respectively) was the “student” being taught intubation, while the other participant (UNMC 1 and UNMC 2, respectively) was the “instructor” who was directing the student’s actions via VoIP to properly intubate a Laerdal Difficult Airway Manikin (Laerdal Medical Corporation, Wappingers Falls, NY).

Results It is possible to instruct the basics of airway management, both by video demonstration and audio instruction, using a 3G cellular or local area network and using a nonintegrated webcam as an “analog” transmission method. The various setups and the resultant video quality are illustrated by Figures 1-3.

Figure 1. SkypeTM interface showing view from USB-enabled C-MAC blade over 3G cellular broadband network.

Figure 2. (a) View of glottic opening visible on C-MAC screen during instruction; (b) View of glottic opening as seen during SkypeTM intubation session.

D.J. Miller et al. / Telemedicine Using Free Voice over Internet Protocol (VoIP) Technology

371

Figure 3. “Non-connected” SkypeTM setup including: (1) C-MAC monitor, (2) webcam pointed at C-MAC screen, (3) notebook PC and (4) intubation mannequin.

Image quality and video lag times were evaluated by an anesthesiologist from UNMC and were deemed acceptable for use in training and could be used for clinical practice, given the appropriate clearances by the FDA and local IRB.

Conclusions Providing instruction in airway management does not require high-dollar video teleconferencing equipment. Although image quality is less than that of expensive VTC equipment, it is more cost effective and can be used over long distances and using commercially available technology to provide video and audio, adequate to teach and perform intubations.


B.H. Boedeker, S Hoffman, W.B. Murray. Endotracheal intubation training using virtual images: Learning with the mobile telementoring intubating video laryngoscope. Stud Health Technol Inform 125 (2007), 49-54. Published by IOS Press. R.E. Link, P.G. Schulam, L.R. Kavoussi. Telesurgery remote monitoring and assistance during laparoscopy. The Urologic Clinics of North America 28 (2001), 177-188. R.H. Taylor, J. Funda, B. Eldridge, S. Gomory, K. Gruben, D. LaRose, et al. A telerobotic assistant for laparoscopic surgery. Engineering in Medicine and Biology Magazine, IEEE 14 (1995), 279-288.

372


iMedic: A Two-Handed Immersive Medical Environment for Distributed Interactive Consultation Paul MLYNIECa,1, Jason JERALDa, Arun YOGANANDANa, F. Jacob SEAGULLb, Fernando TOLEDOc, Udo SCHULTHEISc a Digital ArtForms b University of Maryland c Wichita State University

Abstract. We describe a two-handed immersive and distributed 3D medical system that enables intuitive interaction with multimedia objects and space. The system is applicable to a number of virtual reality and tele-consulting scenarios. Various features were implemented, including measurement tools, interactive segmentation, non-orthogonal planar views, and 3D markup. User studies demonstrated the system’s effectiveness in fundamental 3D tasks, showing that iMedic’s two-handed interface enables placement and construction of 3D objects 4.5-4.7 times as fast as a mouse interface and 1.3-1.7 times as fast as a one-handed wand interface. In addition, avatar-to-avatar collaboration (two iMedic users in a shared space—one subject and one mentor) was shown to be more effective than face-to-face collaboration (one iMedic user/subject and one live mentor) for three tasks. Keywords. Two-handed interface, 3D interaction, 3D visualization, volume rendering, avatar collaboration, user studies

Introduction Under Congressional funding administered by the U.S. Army Telemedicine and Advanced Technology Research Center (TATRC), Digital ArtForms has developed an immersive 3D tele-consultation system called iMedic (Immersive Medical Environment for Distributed Interactive Consultation). iMedic enables remote experts to share 3D DICOM imagery and other 2D/3D assets via the Internet, immersing themselves in common data sets for purposes of diagnosis, planning, and education. To make it appropriate for use in medicine, the project required the development of certain common features such as volumetric rendering, clipping, and measurement capabilities. It also required the development of less common features such as 3D markup and intuitive two-handed DICOM data examination. The goal was to build a broadly capable system, applicable to a wide range of subspecialties such as radiology and surgical planning. Three user studies investigated the effectiveness of the system.

1

[email protected].

P. Mlyniec et al. / iMedic: A Two-Handed Immersive Medical Environment

373

1. The iMedic System An iMedic system consists of a PC, a set of Spacegrips™ (two 6-DOF handheld tracked controllers with 4 buttons each, see lower left corner of Figure 4), a visual display (optionally stereo), and the iMedic software. iMedic is built upon Digital ArtForms’ Two-Handed Interface (THI) software libraries that had previously enabled immersive CAD, command & control, and other applications. THI was believed to be a good fit for the spatially intensive visualization and interaction required for 3D medical problems. THI enables intuitive manipulation of objects and space through intuitive hand gestures, similar to that described by Mapes and Moshell [4]. Users intuitively and directly manipulate objects by simply reaching out and grabbing the objects. For navigation, Figure 1 shows a schematic for manipulating the viewpoint. Users translate the viewpoint by grabbing space with one or two hands. Users rotate the viewpoint by grabbing space with both hands and rotating about the midpoint between the hands. Likewise, users scale the world by grabbing space with both hands and moving the hands apart or bringing them closer together. Translation, rotation, and scale can all be performed simultaneously in a single gesture, enabling users to quickly set their viewpoint to a position, orientation, and scale of interest.

Figure 1. Translation, scale, and rotation of the scene.

The iMedic software utilizes a well-established set of libraries. In addition to object/viewpoint manipulation, the pre-existing libraries support the following features:

A 2D control panel (Figure 5) and widgets that float over the non-dominant hand for manipulation with the dominant hand

Tele-collaboration, including collaboration of objects and avatars, enabling two or more physically-separated users to interact in the same virtual scene (Figure 2) with independent viewpoints

A callback architecture that separates interaction, rendering, and application

Advanced real-time rendering through the use of OpenSceneGraph [5]

Various shaders

3D selection capability through the use of the RAPID collision library [3]

Support for various display types (CAVE™s, HMDs, Walls), with an option for stereo

Software/hardware options, scene/object definitions, and component customizations, all easily configured via XML configuration files

374


Figure 2. Avatar collaboration. The user follows another user into musculature.

Figure 3. The Volumetric isosurface and maximum intensity rendering mixed with polygon rendering.

Some of the features implemented over the period of the project are listed below. 1.1. Volume Rendering We built upon basic volume rendering capability provided by OpenSceneGraph (direct volume, maximum intensity, and iso-surface renderings) to provide a powerful volume visualization tool, adding functionality that enables natural interaction with volumes. In most volume rendering applications [2], viewpoint manipulation (a 6-degree of freedom operation), happens via a 2D mouse and/or keyboard. Interactive techniques are limited with a mouse because there are no surfaces to project the mouse cursor onto. These limitations often result in unintuitive and awkward forms of interaction, resulting in loss of control for the user. iMedic solves this problem by enabling the user to move and rotate about arbitrary points in space and to walk through the volume and explore it in an intuitive, semi-physical manner. 1.2. Merging Volumetric Rendering with Polygonal Rendering Integrating polygonal and volumetric data requires a way of merging two independent rendering methods together into a coherent visualization. In particular, occlusion and transparency cues are important for providing a sense of depth when the hands (represented by 3D cursors) are placed in the volume. Our blending technique also


375

enables drawing of transparent volumes and/or maximum intensity projections over iso-surfaces (Figure 3). 1.3. The Slicebox The Slicebox provides a means of rapidly exploring 3D DICOM datasets by passing a hand-held plane through the volume data and displaying the corresponding oblique cross-sectional imagery on that plane (Figure 4). In this way, the Slicebox is a quick and easy means of discovering the shape of anatomic structures that are not aligned to a principle axis of the data. 1.4. Segmentation and Reconstruction The Slicebox is the front-end for seed planting in our segmentation / reconstruction tool. Users drop seeds inside the Slicebox on the cross section and then build a 3D surface construction from the volumetric dataset (Figure 4). THI enables the user to quickly input seeds (inside and outside the structure of interest) and to control the seed size. As the user drags the seed point across the cross-sectional plane, its statistical reliability is reflected by its brightness—the brighter the seed the higher its reliability. 1.5. The Spindle We found that the center of viewpoint rotation/scale, located between the hands, caused confusion for novice users. We added a visual representation that we call the “Spindle”. The spindle consists of geometry connecting the two hands, with a visual indication of the center point—the center of rotation/scale. In addition to visualizing the point between the two hands, the connecting geometry provides depth-occlusion cues that are directly mapped to the hands. These occlusion depth cues are especially important for monoscopic displays where stereo depth cues are absent. 1.6. Measurement and Marking Tools Interactive analysis in virtual environments often requires quantification of dataset features. For example, a user may want to count the number of vessels exiting a mass (e.g., a tumor), or to indicate areas of interest for further investigation. The system provides an array of tools to interactively mark/count (via dropped fiducial count markers) and measure linear segments, surface area, and angles in the data. 1.7. The Viewbox The Viewbox, similar to a world-in-miniature [9], is a way to “capture” a portion of the 3D world and view it from a second viewpoint. The user surrounds an area of interest with a stretchable box and replicates that space via the control panel. The Viewbox is manipulated just like any other object, affording the user an independent and simultaneous view of the scene. The user can also reach into the Viewbox and manipulate the space contained within, in the same way that he can with the standard view. The Viewbox enables users to maintain a micro and macro view of the scene simultaneously. Since the Viewbox space is a reference to the real scene, anything that

376


happens in the world is reflected in the Viewbox and vice versa. The Viewbox can be attached to the panel so that it is carried with the user, or it can be attached to the viewpoint, resulting in a heads-up display of a secondary view of the world. 1.8. 3D Whiteboarding In typical mouse-based 3D painting, the projection of the cursor's 2D position onto 3D surface geometry determines the location to be painted. Such a solution can result in frustration for various reasons—for example, a paint stroke can result in a longer stroke than intended when the surface is tangential to the view direction. iMedic 3D Whiteboarding enables users to naturally mark up, or paint on, surface geometry (the canvas). Users can grab the canvas in one hand and simultaneously paint using the other hand, an action very similar to painting a solid object in real life. In addition to using the hand’s 3D cursor as a paintbrush, users can select any polygonal object in the environment for use as a stamp. This provides infinite options for brush styles since users can create new objects and then use them as custom-created brushes. The user can modify the size of the brush relative to the canvas by scaling the canvas and/or the object used as the brush. 1.9. 2D Multimedia (Images/Video)

Figure 4. The slicebox and vascular segmentation of a brain.

Figure 5. The user inside a colon, the control panel, and a 2D video.

Still images and video (Figure 5) can be manipulated just as any other object. In addition to displaying the image/video on an object in the world, the video is displayed on the video control panel, enabling the user to preview the video even when the video object is not visible. The user controls movie playback via a control panel that includes standard functionality such as play, reverse play, pause, scroll, and speed change. 1.10. Clipping Clipping is an important aspect of 3D medical visualization packages—it helps users survey a particular portion of a dataset in a selective manner. In traditional interfaces, users either analytically define the clipping object's position and orientation or use a mouse interface, often resulting in user frustration. In contrast, iMedic lets the user grab the clipping plane with either or both hands and move it within the volume to control its position and orientation in a manner similar to holding a plate. In this way, users can focus on the information in the dataset, rather than on the adjustment of


377

control parameters. The system supports both surface geometry and volumetric data. In addition to clip planes, the system supports clip spheres and clip cylinders.

2. User Studies Three user studies were conducted to evaluate the system. We chose to focus the studies on the most fundamental aspects of interaction—object and viewpoint manipulation. Based on anecdotal evidence, we believed that THI would significantly outperform both a mouse interface and a one-handed wand interface, but no hard numbers existed. Studies 1-2 were designed to compare the effectiveness of various interface devices and paradigms for visualizing and interacting with 3D datasets. Study 3 was designed to evaluate avatar collaboration. 2.1. Study 1 (Novice Users) Study 1 [7], a preliminary study which used medical imaging of human anatomy from computer tomography (CT) scans, compared two interfaces: THI and Amira (a mature mouse interface commonly used in 3D medical imaging [1]). Novice users with medical background used each system to carry out navigation, visual search, and measurement tasks with abstract synthetic and anatomical objects. Results indicate that for novice users (n=25) with 15 minutes of training and practice, there was no clear advantage to either interface, due to large variability, in subjective or objective measures. However, a case study of an experienced user showed clear advantages in all tasks using THI. This implied that THI users require longer training and practice time to take advantage of the interface. 2.2. Study 2 (Trained Users for Docking and Construction Tasks)

Figure 6. Study 2 docking task (left) and construction task (right).

Study 2 [6] consisted of a docking (Figure 6, left) and a construction task (Figure 6, right) in which users placed geometric objects. Properly trained THI users (n=20, average training and practice time of 48 minutes) completed the tasks with THI, a mouse-driven interface, and a wand-based interface. For the docking task, THI was 4.7 times as fast as the mouse interface (p < 0.001) and 1.8 times as fast as the wand interface (p < 0.001). For the construction task, THI was 4.5 times as fast as the mouse interface (p < 0.001) and 1.3 times as fast as the one-handed wand interface (p < 0.005). We believe the advantage of THI over the wand was greater with the docking task than

378


for the construction task because the docking task required more viewpoint navigation, deriving the full benefit of two-handed interaction with THI. For simple one-handed object manipulation, wand interfaces have functionality similar to THI. 2.3. Study 3 (Collaboration) Study 3 [8] compared performance of three collaborative tasks using iMedic under two collaboration methods: physical face-to-face and virtual avatar-to-avatar collaboration. In face-to-face collaboration, a subject used iMedic while guided by a mentor who was physically co-located with the subject. The mentor could interact with the subject using speech, gesture, facial expressions, and other means without restriction. In the avatar-to-avatar collaboration, the mentor was physically separated from the subject, and they could only interact via iMedic avatars within the same virtual space. Three mentored tasks were investigated: maze navigation, viewpoint replication, and stent placement. Preliminary results indicate that virtual collaboration led to significantly faster completion times for each of the three tasks. Furthermore, preliminary results suggest a higher success rate in virtual collaboration for the stent placement task, as compared to face-to-face collaboration. Full results will be reported elsewhere.

3. Conclusions We built a two-handed 3D medical collaborative multimedia system. This system can be used for surgical planning, radiological review, and tele-consultation. For fundamental 3D tasks, our interface is 4.5-4.7 times as fast as a mouse interface and 1.3-1.8 times as fast as a wand interface. Additionally, preliminary evidence suggests that iMedic collaboration is more effective than face-to-face collaboration for maze navigation, viewpoint replication, and stent placement.

References [1] Amira, 2010, Visual Imaging Software, http://www.amira.com, accessed July 2010. [2] K. Engel, M. Hadwiger, J. Kniss, C. Rezk-Salama, D. Weiskopf. Real-Time Volume Graphics, A K Peters, Ltd, 2006. [3] S. Gottschalk, M. Lin, D. Manocha. OBB-Tree: A Hierarchical Structure for Rapid Interface Detection, Proceedings of SIGGRAPH ‘96 (1996), 171-180. [4] Mapes, D., Moshell, J.M., 1995, A Two Handed Interface for Object Manipulation in Virtual Environments, Presence: Teleoperators and Virtual Environments 4:4 (1995), 403-416. [5] P. Martz, OpenSceneGraph Quick Start Guide: A Quick Introduction to the Cross-Platform open Source Scene Graph API, Skew Matrix Software, 2010. [6] Schultheis, U., Toledo, F., Mlyniec P., Jerald, J., Yoganandan, A., 2011. Comparison of a Two-Handed Interface to a Wand Interface and a Mouse Interface for Fundamental 3D Tasks, Submitted to ACM SIGCHI Conference on Human Factors in Computing Systems (2011). [7] F.J. Seagull, P. Miller, I. George, P. Mlyniec, A. Park, Interacting in 3D Space: Comparison of a 3D Two-handed Interface to a Keyboard-and-mouse Interface for Medical 3D Image Manipulation, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 53:27 (2009), 2004-2008. [8] F.J. Seagull , Mlyniec, P., Jerald, J., Yoganandan, A., 2010. .“Comparison of virtual and face-to-face collaboration using the iMedic system.” Technical report #2010-02-TATRC. Los Gatos, CA: Digital Artforms. [9] R. Stoakley, M. Conway, R. Pausch, Virtual Reality on a WIM: Interactive Worlds in Miniature, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (1995), 265272.


379

Patient Specific Surgical Simulator for the Evaluation of the Movability of Bimanual Robotic Arms Andrea MOGLIA a,1, Giuseppe TURINI a, Vincenzo FERRARI a, Mauro FERRARI a, Franco MOSCA a a EndoCAS, Center for Computer Assisted Surgery, via Paradisa 2, 56124 Pisa (Italy)

Abstract. This work presents a simulator based on patient specific data for bimanual surgical robots. Given a bimanual robot with a particular geometry and kinematics, and a patient specific virtual anatomy, the aim of this simulator was to evaluate if a dexterous movability was obtainable to avoid collisions with the surrounding virtual anatomy in order to prevent potential damages to the tissues during the real surgical procedure. In addition, it could help surgeons to find the optimal positioning of the robot before entering the operative room. This application was tested using a haptic device to reproduce the interactions of the robot with deformable organs. The results showed good performances in terms of frame rate for the graphic, haptic, and dynamic processes. Keywords. Robotic surgical simulation; patient specific surgical simulation; surgical robots setup.

Introduction Pioneered by Computer Motion (Sunnyvale, California, Unites States) with the ZEUS system and become increasingly popular among surgeons over the past few years thanks to the da Vinci system by Intuitive Surgical (Sunnyvale, California, United States), surgical robotics represents a viable solution to complex minimally invasive surgery (MIS) [1]. This novel approach has several advantages over laparoscopy, including: improved manoeuvrability of instruments by allowing wristed and finger movements (da Vinci EndoWrist®), removal of trocar fulcrum effect (inversion of movements), tremor minimization, motion scaling, 3D visualization, and a more comfortable ergonomic position of the surgeon [2]. From the clinical viepoint the benefits of surgical robotics translate into safe and fine scale operations, reduced trauma, shortened recovery time, low level of fatigue for surgeons even after using the robot for prolonged time [3], allowing also to perform new kinds of MIS interventions. As in laparoscopy, in the setup stage before the operation a proper placement of trocars is helpful to prevent possible collisions between the robot ams, as documented in previous works concerning ZEUS and da Vinci robots [3] [4] [5]. The trend of traditional MIS, followed by robotic surgery, is to reduce the number of access ports as pursued by a novel approach called single incision laparoscopic 1

Corresponding Author: Andrea Moglia, EndoCAS, Center for Computer Assisted Surgery, via Paradisa 2, 56124 Pisa, Italy; E-mail: [email protected].

380

A. Moglia et al. / Patient Specific Surgical Simulator

surgery (SILS) [6]. The correct positioning of the robot is of paramount importance for SILS procedures due to the limited workspace of the robot arms, as exemplified by the bimanual robotic arm unveiled by Intuitive Surgical at International Conference on Robotics and Automation 2010 (Anchorage, Alaska, United States) and by the ongoing research activities within ARAKNES (Array of Robots Augmenting the KiNematics of Endoluminal Surgery) Project. In this paper we discuss the development of a simulator for a single port robot with bimanual abilities to validate its potential application, in particular to bariatric surgery and cholecystectomy. In particular, given a bimanual robot with its own geometry and kinematics, and a patient specific anatomy, the proposed simulator allows to evaluate in a virtual environment if a dexterous movability of the robot is achievable, avoiding collisions with the anatomy to prevent potential damages in the real surgical procedure. In addition it can help surgeons before entering the operative room to choose the optimal positioning of the robot and the access port in the abdominal wall. This simulator can be customized for any present or future bimanual surgical robots. The first prototype includes the following features: robot motion via inverse kinematics, robot motion tracking, patient specific virtual anatomy, deformable organs, haptic feedback, and customizable robot configuration. This work is carrying on within the aforementioned ARAKNES initiative and aims at realizing a realistic simulator of the final surgical robot for training and planning, based on patient specific biomechanical modeling.

Methods & Materials Modeling of the Virtual Scene A 55-year old man underwent a total body computed tomography (CT) with contrast agent (stomach insufflated with carbon dioxide (CO2)) at the Radiology Department of Cisanello Hospital in Pisa (Italy). The medical dataset was processed using our segmentation pipeline, developed customizing ITK-SNAP, a software for the generation of 3D virtual models (e.g. in STL format) from the stack of CT images (in DICOM format) [7]. Being the obtained 3D models quite raw, an optimization stage occurred before processing them by the algorithms generating the dynamic models. This task was performed through MeshLab (Visual Computing Lab, ISTI-CNR, Pisa, Italy) and Autodesk® Maya (Toronto, Ontario, Canada), and consists in mesh simplification, artefacts removal, and holes filling [8]. During this stage the complexity of the mesh was heavily reduced, enhancing the simulation performance without losing a good visual appearance. The virtual organs, with the same shapes and dimensions as those of a real patient, were placed in a virtual mannequin, generated after segmenting a CT acquisition of a commercial Phantom OGI by Coburger Lehrmittelanstalt (Coburg, Germany). The simulator enables the loading and rehearsal of different configurations of single port bimanual robots, inserted through the laparoscopic port by a rigid over-tube. Both robotic arms were modelled as a sequence of joints and links and terminate with an end effector. This was designed to host different surgical instruments for a wide range of specialized tasks, including gripper, forceps, and scalpels. A screenshot of the complete virtual scene is illustrated in Figure 1.


381

Figure 1. Screenshot of the simulator.

The configuration properties of the robot arms are described in a script file (RBT, a customized format), loaded at simulation launching or whenever the user selects a different robot component, choosing the links and joints characteristics. Overall, the virtual scene is represented by a script file (PRJ, a customized format) describing the position, orientation and properties of each virtual organs, surgical robot configuration, laparoscopic camera preferences, and application settings. Algorithms and Data Structures For the purposes of this work we chose CHAI 3D, an open-source framework of C++ libraries for computer haptics, visualization and interactive real-time simulation.

Figure 2. Dynamic skeleton of the gallbladder, pointing out fixed spheres in red and those free to move in green (left). Spheres tree for collision detection between robot arms and gallbladder (right).

The dynamic model of the deformable objects is composed of a volumetric model of the object consisting of a skeleton of spheres (nodes) and links and generated automatically, using the filling sphere approach [9]. In our case, the volumetric model approximates the volume of the organ and its nodes and links are connected through

382


elastic links to the vertices of the surface model, generated from the optimized organ surface with point masses on the mesh vertices and damped springs on the edges. This method decouples local deformations, modelled by the surface model and affecting a small surface portion, and global ones, modelled by the volumetric model and influencing a large portion of the surface [9]. In our simulation a dynamic skeleton was associated to stomach, liver, and gallbladder. In Figure 2 (left) the skeleton of gallbladder is represented, with red and green spheres indicating respectively fixed and movable nodes. For the purposes of our work, collision detection concerns only those organs within the workspace of the robot arms, in particular the deformable ones. The remaining organs are static and they do not participate in the collision detection. In particular collisions with the deformable objects are simplified considering only interactions with volume spheres of the volumetric model, as depicted in Figure 2 (right) [9]. In addition not all spheres of each organ participate in the collision detection phase, but only those which can be reached by the robot arms. In this way the number of frame rate, and consequently the performance, can be increased without losing realism during the interaction. The data structures for the collision detection of the robot arms were designed to model different levels of detail for each part of the arm, represented by a joint and the following link in the kinematic chain. This choice is due to the different probability of interaction of each part of the robot arm with both the deformable organs and the other arm. For example, by considering the interaction with the deformable organs, the tip has the highest probability of interaction while the over-tube the lowest. The data structure consists in a spheres tree with an appropriate level of detail, as depicted in Figure 2 (right). Additionally, self-collisions of an arm and itself and between the two robot arms were considered. The former were avoided implicitly by the robot kinematics, while the latter handled as standard collisions thanks to the independent collision detection data structure of each robot arm. Since the data structures for collision detection of both deformable organs and robot arms are based on spheres, the single tests are spheres vs. spheres intersections, resulting in the fastest computation. Each of these tests generates two force components, equal and opposite, resulting in the force feedback and the dynamic simulation of the deformable organs.

Simulation The presented simulation is composed of a graphic, a dynamic, and a haptic processes (Figure 3). The first deals with the user interface and the visualization of the virtual scene, using the scene graph provided by CHAI 3D, based on OpenGL. The second concerns collision detection, dynamics of deformable objects, inverse kinematics of robot arms, and force feedback. The haptic process ensures that the force feedback is updated with a proper frequency for a realistic interaction.


383

Figure 3. Flowchart of the simulation processes.

The set up of the virtual environment includes an abdominal access for the single port bimanual robot with liver, stomach, and gallbladder as targets of the surgical procedures (bariatric surgery and cholecystectomy). In this regard, these organs were dynamically modelled to simulate the deformations caused by the interaction with the robot arms. All the physical models were generated offline and loaded during the simulation startup. The virtual scene comprises also the other organs of the upper abdomen, the backbone, the surgical robot, and a mannequin. The surgical robot used in this simulation has two arms, both with 6 d.o.f (degrees of freedom) and a generic gripper as end effector. The joints have the following configuration: shoulder (one roll and one pitch), elbow (one pitch), and wrist (one roll, one pitch, and one roll). The motion of the robot arms links is computed by inverse kinematics when controlling the end effector with the haptic interface.

Results The current targets of our surgical simulator within ARAKNES project are bariatric surgery and cholecystectomy. The former is a procedure performed in patients suffering from obesity. It can be divided into restrictive (gastric banding, vertical gastroplasty, sleeve gastrectomy), malabsorptive (biliopancreatic diversion), or a combination of both (gastric bypass) [10]. Bariatric surgery was selected since surgical robotics offers improved ergonomics over laparoscopy against large thick abdominal walls [2]. On the other hand, since robotic cholecystectomy has a proven safety and is a sort of benchmark for surgical devices, it was selected as the second target procedure [11]. The virtual scene includes patient specific organs. Since stomach, liver, and gallbladder are the target organs of the selected procedures, they are simulated as deformable objects in order to provide realistic interaction when the robot arms collide with them. The other organs are the pancreas and the kidneys. Our simulation was tested on a workstation running Microsoft Windows 7 (Intel Core i7 – 3 GHz, 12 GByte RAM, 2 GPU nVidia GTX 285) in a virtual scene composed of 52k vertices and 104k triangles. The dynamic skeleton of the deformable organs (stomach, liver, and gallbladder) was made up of 257 nodes and 1.217 links.

384


Figure 4. Real-time visualization of the tracking of the robot arm motion (left). Workstation running the simulation and haptic device (right).

The graphic process provided a frame rate ranging from 30 to 50 fps, the dynamic process from 1.0 to 1.6 kHz, and the haptic process exceeded 1.0 kHz. The required memory to run in real-time the simulation was 94 MByte. During the simulation the user can activate the robot arms motion tracking. These data can be visualized in real-time (Figure 4 on the left) or saved in a log file for a postprocessing analysis. The arms of the bimanual robot are guided by a Falcon, a haptic interface by Novint (Albuquerque, New Mexico, United States), as shown in Figure 4 (right). The user has the possibility to select which arm to control via keyboard input.

Conclusions In this paper we described a simulation for the evaluation of the movability of bimanual surgical robots in a virtual anatomical district of a real patient and reconstructed after segmentation. The motion of the robot is computed by inverse kinematics and can be preoperatively simulated and rehearsed. Thanks to the integration with one haptic interface, surgeons can drag the robot arms in a consistent way with the kinematics of the real bimanual robot. The present version of our simulation enables surgeons to drag the arms of a bimanual robot in order to evaluate its movability to avoid collisions with the surrounding virtual anatomy, which might turn into damages during the real surgical procedure. This simulator provides also force feedback to users when they touch patient specific organs using the haptic device. Moreover, it can help surgeons to plan the optimal positioning of the robot support. Besides experiencing a real-time interaction of the robot arms with the anatomy with visual and haptic feedback, it is possible to perform a detailed analysis on the end effector trajectory and on the distance of each link from the surrounding tissues thanks to the data stored in the log file. The discussed work is in progress. Currently we are extending the dynamic model provided by CHAI 3D for the management of dynamic objects in order to represent deformations in a more realistic way. In particular we are optimizing the skeleton of deformable meshes, tweaking nodes and links properties, and the structure of the


385

skeleton (number of nodes and links and their arrangement). On the other hand we are working on the tuning of the dynamic properties of the virtual organs, based on patient specific data, to reproduce the pathological states of the target surgical procedures of our simulator. Furthermore, since the modeling technique we are using does not lend itself to the implementation of complex tasks like mesh cutting, we are working on alternative approaches to pursue this objective. We are also integrating into the simulation other haptic devices available at our center, as the Freedom 6S by MPB Technologies (Montreal, Quebec, Canada).

Acknowledgment The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement num. 224565 (ARAKNES Project). The authors would like to express a sincere thank to Dr. Lorenzo Faggioni for acquiring the CT dataset of DICOM images, and Ms. Marina Carbone for performing the segmentation of this dataset.

References [1]

C. Nguan, A. Girvan, P.P. Luke, Robotic surgery versus laparoscopy; a comparison between two robotic systems and laparoscopy, Journal of robotic surgery 1 (2008), 263-268. [2] E.B. Wilson, The evolution of robotic general surgery, Scand J Surg 98 (2009), 125-129. [3] M. Hayashibe, N. Suzuki, M. Hashizume, K. Konishi, A. Hattori, Robotic surgery setup simulation with the integration of inverse-kinematics computation and medical imaging, Comput Methods Programs Biomed 83 (2006), 63-72. [4] M. Hayashibe, N. Suzuki, M. Hashizume, Y. Kakeji, K. Konishi, S. Suzuki, A. Hattori, Preoperative planning system for surgical robotics setup with kinematics and haptics, Int J Med Robot 1 (2005), 7685. [5] A. Pietrabissa, L. Morelli, M. Ferrari, A. Peri, V. Ferrari, A. Moglia, L. Pugliese, F. Guarracino, F. Mosca, Mixed reality for robotic treatment of a splenic artery aneurysm, Surg Endosc 24 (2010), 1204. [6] M.B. Ostrowitz, D. Eschete, H. Zemon, G. DeNoto, Robotic-assisted single-incision right colectomy: early experience, Int J Med Robot 5 (2009), 465-470. [7] G. Megali, V. Ferrari, C. Freschi, B. Morabito, F. Cavallo, G. Turini, E. Troia, C. Cappelli, A. Pietrabissa, O. Tonet, A. Cuschieri, P. Dario, F. Mosca, EndoCAS navigator platform: a common platform for computer and robotic assistance in minimally invasive surgery, Int J Med Robot 4 (2008), 242-251. [8] P. Cignoni, M. Callieri, M. Corsini, MeshLab: an Open-Source Mesh Processing Tool, Sixth Eurographics Italian Chapter Conference (2008), 129-136. [9] F. Conti, O. Khatib, C. Baur, Interactive Rendering Of Deformable Objects Based On A Filling Sphere Modeling Approach, Proceedings of IEEE International Conference on Robotics and Automation (2003), 3716-3721. [10] M. Ibrahim, D. Blero, J. Deviere J, Endoscopic options for the treatment of obesity, Gastroenterology 138 (2010), 2228-2232. [11] S. Breitenstein, A. Nocito, M. Puhan, U. Held, M. Weber, P.A. Clavien. Robotic-assisted versus laparoscopic cholecystectomy: outcome and cost analyses of a case-matched control study, Ann Surg 247 (2008), 987-993.

386


CyberMedVPS: Visual Programming for Development of Simulators Aline M. MORAISa,1 and Liliane S. MACHADO b a, b Federal University of Paraíba, UFPB - Brazil

Abstract. Computer applications based on Virtual Reality (VR) has been outstanding in training and teaching in the medical filed due to their ability to simulate realistic in which users can practice skills and decision making in different situations. But was realized in these frameworks a hard interaction of non-programmers users. Based on this problematic will be shown the CyberMedVPS, a graphical module which implement Visual Programming concepts to solve an interaction trouble. Frameworks to develop such simulators are available but their use demands knowledge of programming. Based on this problematic will be shown the CyberMedVPS, a graphical module for the CyberMed framework, which implements Visual Programming concepts to allow the development of simulators by non-programmers professionals of the medical field. Keywords. Virtual Reality, Visual Programming, CyberMed.

Introduction A framework is defined as an abstract implementation for application development on a particular problem domain, with the advantage of reuse of components [6]. An important area of its application is the health sciences, particularly to the development of training simulators or tools to assist specific procedures. Thus, characteristics related to Virtual Reality (VR) as immersion, touch sense, 3D viewing, among others, are increasingly supported by these frameworks in order to assist training [1] [2], planning [3] [4] and assistance [5] tasks. It was observed that frameworks specifically conceived to development of medical simulators based on VR offer few or none resources of usability for non-programmer users. VP is divided in four components: Visual Expressions (VE), Visual Programming Language (VPL), Visual Programming Environment (VPE) and Visual Programming System (VPS). VE are sets of graphical elements called symbols; VPL are languages which have visual expressions; VPE are environments composed by visual expressions aggregated to textual language inputs; and a VPS is defined as a composition of VE in an absolutely graphical programming environment. The definition of which VP components will be adopted is essential for the development process of a VP module integrated to medical framework. A related works revision was made in scientific journals and papers between the years 2005 and 2010. The study was based in parameters like as presence and type of VP, VR features and functionalities of each one. This research found several frameworks utilized in health area which had a few VP technique or some type of 1

Corresponding Author: Aline M. Morais, UFPB/CCEN/LabTEVE, Cidade Universitária s/n, João Pessoa/PB – Brazil, 58051-900. Email: [email protected]

387

A.M. Morais and L.S. Machado / Visual Programming for Development of Simulators

visual resource. A comparison of the characteristics of these frameworks allowed identifying if they had approaches for medical profile and which VP technique was used, for example. Among the medical frameworks with visual approaches for programming found in scientific literature, the MeVisLab [3], SCIRun [4] and NeatVision [5] are characterized as frameworks with VPE due to the presence of visual expressions and code line input. Besides, the ViMeTWizard [2] is other medical framework which applies visual resources, but without a VP approach. Then, none of the tools analyzed are VPS and demands from users some knowledge of programming techniques for the utilization of their full capabilities. In parallel, interviews were conducted in order to identify needs related to tools for these professionals. The interviews showed that 58,06% of health professionals could use a tool with VR resources since this tool is easy to use. The bibliographic review and the interviews result allowed observing a lack of tools for health professionals that allow this public to develop by themselves applications based on VR for education and training. 1. The CyberMedVPS CyberMedVPS was designed to be used by health professionals to allow them to develop their own VR applications. This VPS was projected to be composed by the CyberMed, a framework to the development of VR systems based on PCs whose goal is supporting the creation of medical training applications, particularly those related to the simulation of medical procedures in immersive virtual environments [1]. CyberMed was chosen due to its support to stereoscopic visualization, haptics interactions, interactive collision detection and deformation, tracking devices, collaboration and, specially, user performance assessment. Additionally, this framework is stable, free and has been continuously developed and expanded. The conception of CyberMedVPS as a VPS extinguishes any necessity of textual code input to generate results and interaction happens only in graphical mode based on flowcharts. Thus, textual programming is not necessary for generation or execution of VR applications. Through several graphical components, users can connect boxes to make a flowchart representing steps of instructions required to execute a VR application. A text window (Fig. 1X-D) provides feedback with messages related to the flowchart execution: if there is some kind of error in serialization of instructions, the VPS will point it out in the text window. CyberMedVPS design also include the offer of an option to export the flowchart to textual code for programmer users who wish to modify the VR applications using the native language of CyberMed (C++). The validation and correction of the flowchart (Fig. 1Y) is done automatically when required by user. X

Y

Figure 1. Interface for CyberMedVPS (X) and example of boxes in a flowchart in CyberMedVPS (Y)

388

A.M. Morais and L.S. Machado / Visual Programming for Development of Simulators

The CyberMed components are offered to user by CyberMedVPS. These CyberMed components are able to generate the main configuration file, called main.cpp, and are based in design pattern MVC (Model, View and Core). Together these elements are useful to provide several VR functionalities to user. A changing of parameters is facilitated because the details of subcomponents are graphically elucidated with the usage of VP. 1.1. CyberMedVPS Architecture To the creation of CyberMedVPS was adopted a step sequence following: 1) Applying of a interview with health professionals in order to understand the real needs in RV tools; 2) Test with prototypes created from interview results; 3) CyberMedVPS implementation and 4) CyberMedVPS final test with the target public. As shown in Figure 2 the CyberMedVPS is composed by two layers: the graphical interface layer has all visual elements that can be manipulated by users to produce the medical applications and the integration of this layer to CyberMed depends on the communication layer that will relate the graphical elements of the flowchart to the framework commands. Thus, this communication layer allows bidirectional flow between the graphical interface and the framework.

Figure 2. CyberMedVPS functional architecture

2. Conclusions This project is under development and included in the National Institutes of Science and Technologies - MACC supported by the Brazilian National Council of Scientific and Technological Development. The results of the implementation should be evaluated by professionals to validate the project goals. This project was approved by the Ethics Committee of Lauro Wanderlei Universitary Hospital of Federal University of Paraíba, Brazil. References [1] D. F. L. Souza, I. L. L. Cunha, L. C. Souza, R. M. Moraes and L. S. Machado, A Framework for Development of Virtual Reality-Based Training Simulators, MMVR 17, pp. 174-176, 2009. [2] A. C. M. T. G. Oliveira and F. L.S. Nunes, Building a Open Source Framework for Virtual Medical Training, Journal of Digital Imaging, DOI = 10.1007/s10278-009-9243-3, 2009. [3] J. Rexilius, W. Spindler, J. Jomier, M. Konig, H. K. Hahn, F. Link and H. Peitgen, A Framework for Algorithm Evaluation and Clinical Application Prototyping using ITK, MICCAI Workshop, 2005. [4] C.R. Jonhnson and S.G. Parker, Applications In Computational Medicine Using Scirun: A Computational Steering Programming Environment, Proc. H.W. Meuer, editor, Supercomputer ‘95, p. 2–19. Saur-Verlag, 1995. [5] P. F. Whelan, R. J. T. Sadleir and O. Ghita, NeatVision: Visual Programming for Computer-aided Diagnostic Applications, Informatics in Radiology (infoRAD), Published online 10.1148/rg.246045021, 2004. [6] M. Mattsson, J. Bosch, Framework Composition: Problems, Causes and Solutions, Course Documentation. Technology of Object-Oriented Languages and Systems, 1997.


389

A Bloodstream Simulation Based on Particle Method Masashi NAKAGAWA†, Nobuhiko MUKAI†, Kiyomi NIKI† and Shuichiro TAKANASHI‡ †Graduate School of Engineering, Tokyo City University, Japan ‡Department of Cardiovascular Surgery, Sakakibara Heart Institute, Japan E-mail: [email protected]

Abstract. Many surgical simulators use mesh method to deform CG models such as organs and blood vessels because the method can easily calculate the deformation of models; however, it has to split and reconstruct the mesh of the models when the model is broken such as bleeding. On the other hand, particle methods consider a continuous body such as solid and liquid as a set of particles and do not have to construct the mesh. Therefore, in this paper, we describe how to simulate bloodstream by using MPS (Moving Particle Semi-implicit) method that is one of particle ones. In the simulation, we use the aorta model as the blood vessel model, and the model is constructed with particles. As the result of the simulation, it took 20ms to deform the blood vessel and to simulate bleeding with the model that is constructed with 15,880 particles for the blood vessel and 6,688 particles for the blood. Keywords. computer graphics, virtual reality, surgical simulators, particle method

Introduction In these days, surgical simulators have been more useful for surgical training and preoperative planning because medical treatment is developed with engineering technology and surgeries are more complicated. Surgical simulators use CG (Computer Graphics) and VR (Virtual Reality) technologies, and give us chances for training or preoperative planning with patient’s data [1,2]. Most of the organ models used by these simulators are consisted of finite element model or mass spring model and we have performed faster blood vessel deformation by using mass spring model [3]; however, this model did not consider blood flow inside of the vessel and the inside of the vessel was constructed with an elastic body. Then, this model could not be used for more precise simulation with bleeding and blood flow. In some blood flow simulations, FEM (Finite Element Method) or LBM (Lattice Boltzmann Method) is used [4,5]; however, FEM requires to reconstruct the model when the topology changes and it takes much time especially for bleeding that combines some particles and/or breaks them into small ones. In addition, LBM takes too much time to calculate displacement of particles since the space needs to be divided into some voxies, which number depends on the size and the shape of the boundary. On the other hand, particle methods construct the models as a set of particles so that it does not have to reconstruct the mesh every time the topology changes. Particle methods are very useful to represent complex phenomenon such as bleeding. The only

390

M. Nakagawa et al. / A Bloodstream Simulation Based on Particle Method

thing we have to consider is that the calculation time by particle method depends on the number of particles. There are two major particle methods : SPH (Smoothed Particle Hydrodynamics) and MPS method, which has been developed to deal with incompressible fluid and also used to express breaking wave [6] and analyze red blood cells [7]. Therefore in this paper, we adapt MPS method as the particle method and perform the simulation of bloodstream and blood vessel deformation by using fluid dynamics and elastic dynamics theories.

1. Methods 1.1. Calculation Model The discretization using MPS method needs particle interaction calculation for which a weight function is defined with Eq.(1) under the condition that re is the effective radius within which particles interact each other, rij is the distance between particle i and particle j. Here, the gradient, divergence and Laplacian in the space region of a particle are defined [8].

w(rij ) =

re − 1 (rij ≤ re ), w(rij ) = 0 (rij > re ) rij

(1)

1.2. Blood Model ρ

1 ⎛ ∂vα ∂v β ⎞ Dv α ∂σ αβ α S αβ = ⎜ β + α ⎟ ...Eq.(1 − 3) = + ρ − K ... Eq .(1 1) 2 ⎝ ∂x ∂x ⎠ ∂x β Dt σ αβ = − pδ αβ + λ S γγ δ αβ + 2 μ S αβ ...Eq.(1 − 2)

ρ

Dv α ∂p ∂ 2v γ ∂ 2vα = − α + ( λ + μ ) α γ + μ β β + ρ K α ...Eq.(1 − 4) Dt ∂x ∂x ∂x ∂x ∂x ∂v γ Dρ + ρ γ = 0...Eq.(1 − 5) ∂x Dt Dv α ∂p ∂ 2v α ρ = − α + μ β β + ρ K α ...Eq.(1 − 6) v*, r * Dt ∂x ∂x ∂x k +1 * ⎧ ⎪ vi = vi + v′ ρ n − n0 ...Eq.(1 − 8) 2 ⎨ k +1 * ...Eq.(1 − 7) ∇ p=− 2 ⎪⎩ri = ri + Δtv ′ n0 Δt v′ Figure 1. Relationship among fluid model equations

Blood is generally non-Newtonian fluid; however, when the speed is fast, it can be approximated as Newtonian fluid so that in this paper, blood is treated as incompressible Newtonian fluid because the target vessel is the aorta, within which blood flows very fast. In addition, continuous equation and Cauchy’s equation of motion are used for governing equations of fluid. Cauchy’s equation is given with Eq.(1-1) in Figure 1. The superscripts (α, β and γ) are used for the index notation of generic terms. One of the superscripts is one component of x, y and z in Cartesian

391


coordinate system, where v is velocity, t is time, ρ is density, σ is stress tensor, K is force in unit mass, and D/Dt is Lagrangian difference. In Newtonian fluid, stress tensor is expressed with Eq.(1-2) in Figure 1, where p is pressure, δ is Kronecker delta, μ is viscosity, and S is strain rate tensor shown as Eq.(13) in Figure 1. By substituting the Eq.(1-2) into the Eq.(1-1), we obtain Eq.(1-4) in Figure 1. In addition, Eq.(1-5) in Figure 1 is satisfied by continuous equation under the condition of incompressibility. As a result, Eq.(1-4) becomes Eq.(1-6) in Figure 1. By discretizing Eq.(1-6) in Figure 1, we obtain Eq.(2).

ρi

Dv iα d = 0 Dt n

(

l = ∑ rij j ≠i

2

∑

p j − pi rij

j ≠i

)

2

rij w ( rij ) + μ

2d ln 0

∑ (vα − vα ) w ( r ) + ρ K α j

i

j ≠i

ij

i

(2)

w ( rij ) / ∑ w ( rij ) j ≠i

where d is the number of dimension, n0 is the sum of weights under incompressible state, and rij is the distance between particle i and particle j. Here, particle movement is pre-calculated without the pressure term and then modified after the pressure is calculated with the pre-calculated particle movement. v* and r * in Figure 1 are a temporary velocity and a temporary position which are pre-calculated without pressure component in Eq.(1-6) in Figure 1. By considering the pressure component which we excluded, the temporary velocity v* and the temporary position r * of the particle are modified. Since the density of fluid is proportional to the particle number density, we calculate Poisson’s equation of pressure with Eq.(1-7) and the modified velocity with Eq.(3).

2d ln0

∑( p j ≠i

j

( )

− pi ) w rij = −

Dv′ d ρ i =− 0 Dt n

∑ j ≠i

p j − pi rij

2

ρi n − n 0 Δt 2

n0 (3)

( )

rij w rij

The velocity v and the position r of particle i at time step k + 1 are expressed with Eq.(1-8) in Figure 1. With the above process, the governing equation of fluid can be solved. 1.3. Blood Vessel Model

ρ

Dv α ∂σ αβ = + ρ K α ...Eq.(2 − 1) Dt ∂x β

1 ⎛ ∂uα ∂u β ⎞ + ⎟ ...Eq.(2 − 3) 2 ⎝ ∂x β ∂xα ⎠ σ αβ = λε γγ δ αβ + 2με αβ ...Eq.(2 − 2) α Dv ∂ 2u γ ∂ 2u α ρ = ( λ + μ ) α γ + μ β β + ρ K α ...Eq.(2 − 4) Dt ∂x ∂x ∂x ∂x

ε αβ = ⎜

Figure 2. Relationship among elastic equations

392


In this paper, we assume that blood vessel is an isotopic elastic body. Eq.(2-2) in Figure 2 are used as the governing equations of elastic body under the condition that ε is strain tensor (Eq.(2-3)), u is displacement, and λ and μ are lame constants. We obtain Eq.(4) by substituting Eq.(2-2) into Eq.(2-1) in Figure 2 and discritizing Eq.(2-4) in Figure 2.

ni − n 0 n0 Dv iα d p j − pi 2d = 0∑ rij w ( rij ) + μ 0 ρi Dt n j ≠ i rij 2 ln pi = λ

∑ ( u αj − u iα ) w ( rij ) + ρ i K α

(4)

j ≠i

1.4. Interaction between Blood and Blood Vessel We can simulate the repulsion from blood vessel particle to blood particle by solving Poisson’s equation of pressure without distinguishing blood particles and blood vessel particles. In addition, by calculating particle number density without distinguishing blood vessel particles and blood particles, we can calculate volume strain which includes pressure by collision. 1.5. Simulation Models We have generated some models for the simulation. Surgical tools, blood vessel and blood models consist of rigid, elastic and fluid particles, respectively. The model space is divided into some cells which length is the same as the particle radius, and the surgical tool is built by placing one particle at the center of the cell which center is contained in the polygon composing of the surgical tool. By converting CT image data into polygons with Marching Cubes [9], the polygons of a blood vessel are extracted, and then the above method is applied to build the blood vessel model. By using this method, the model can be built with arbitrary particle size, and the blood particles are also inserted into the blood vessel.

2. Results and Conclusions We have used the aorta model as the blood vessel in the simulation. The blood vessel is deformed by pouring blood particles into the blood vessel model and making contact with stick surgical tools. The initial state is shown in Figure 3(a). Two types of deformation with bloodstream and without bloodstream were simulated and compared. In case without bloodstream (Figure 3(b)), the blood vessel was severely crushed compared to the deformation with bloodstream (Figure 3(c)). Bleeding simulation has also been performed by cutting a part of the blood vessel, and blood particles were breaking out of the blood vessel (Figure 3(d)). It took 20ms to deform the blood vessel and to simulate bleeding with the model that is constructed with 15,880 particles for the blood vessel and 6,688 particles for the blood on a PC consisting of 2.8GHz Intel Core2 Quad CPU and NVIDIA GeForce9800GTX+ Graphics Card.


393

Figure 3. Blood vessel deformation

A bloodstream simulation has been performed with MPS method. Surgical tools, blood vessel and blood models are composed of rigid, elastic and fluid particles, respectively. Blood vessel model was generated as the aorta from CT image data. By comparing two types of simulation, we have confirmed that bloodstream is important for the deformation of blood vessels. We have also performed bleeding simulation, which is difficult with mesh method, by using a particle method. In this simulation, however, the bloodstream in the blood vessel was treated as steady flow of Newtonian fluid. In the further research, we plan to treat blood as pulsatile and non-Newtonian fluid.

Acknowledgements This research was supported by Japan Society for the Promotion of Science (Research No.21500125).

References [1] [2] [3] [4] [5] [6] [7] [8] [9]

J. Mosegaard, P. Herborg and T. Sorensen, “A GPU Accelerated Spring Mass System for Surgical Simulation”, Studies in health technology and informatics, Vol.111, pp.342-348, 2005 M. Nakagawa, N. Mukai and M. Kosugi, “A Fast Blood Vessel Deformation Method Considering Inside Pressure”, ITE report, Vol.63, No.3, pp.371-375, 2009 N. Mukai, M. Nakagawa and M. Kosugi, “Real-time Blood Vessel Deformation with Bleeding Based on Particle Method”, MMVR16, pp.313-315, 2008 T. Kawamura, C. Xian, T. Hisada, K. Tsukahara and K. Tsuchimoto, “Investigations of Mechanical Characteristics of Pulsatile Artificial Heart by Fluid-Structure Finite Element Interaction Analysis”, The Japan Society of Mechanical Engineers, Vol.14, pp.273-274, 2002 W. Li, Z. Fan, X. Wei and A. Kaufman, “GPU Gem2”, pp.687-702, 2005 Q. Wang, Y. Zheng, C. Chun, T. Fujimoto and N. Chiba, “Efficient rendering of breaking waves using mps method”, Journal of Zhejjang University SCIENCE A, Vol.7, No.6, pp.1018-1025, 2006 K. Tsubota, S. Wada, H. Kamada, Y. Kitagawa, R. Lima and T. Yamaguchi, “A Particle Method for Blood Flow Simulation, -Application to Flowing Red Blood Cells and Platelets-“, Journal of the Earth Simulator, Vol.5, 2006 S. Koshizuka, “Particle Method”, Maruzen, 2005 W. E. Lorensen and H. E. Cline, “Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, ACM SIGGRAPH Computer Graphics, Vol.21, No.4, 1987

394


Laser Induced Shockwaves on Flexible Polymers for Treatment of Bacterial Biofilms a

Artemio NAVARRO, bZachary D. TAYLOR, cDavid BEENHOUWER, c David A. HAAKE, aVijay GUPTA, bWarren S. GRUNDFEST a UCLA Department of Mchanical Engineering b UCLA Department of Bioengineering c Los Angeles Veterans Administration

Abstract. Bacterial biofilm-related infections are a burden on the healthcare industry. The effect of laser generated shockwaves through polycarbonate, a flexible polymer, is explored for its ability to generate high peak stresses, and also for its ability to conform to complex wound surfaces. Shockwave pulses in Al coated polycarbonate substrates and a resulting peak stress of greater than 60 MPa was measured which should provide sufficient pressure to kill bacteria. Keywords. Shockwaves, biofilms, wound, laser

1. Introduction Wound infections and infected traumatic wounds impose a major burden on the healthcare system. Treatment of infected wounds can prolong hospitalization and dramatically increase the cost of patient care. Recent studies estimate that 5% of all surgical wounds become infected, and 5%-7% of all traumatic wounds require open therapy for management of infections. Bacterial persistence in wounds is facilitated by the production of biofilms. Biofilms are bacterial communities containing a thick matrix of mucopolysaccharides produced by most bacterial species [1-2]. Current procedures to treat biofilm infected wounds are relatively ineffective and invasive. In this work we explore flexible polymers that can tolerate high peak stresses and with applications to biofilm disruption.

2. Tools and Methods Laser-generated pulses impinging upon a thin metallic surface generate stress waves within the material. The laser energy ablates the thin metallic film, thereby causing a rapid thermal expansion of the film resulting in a compressive wave propagating through the substrate. The laser fluence, pulse width, and the substrate material properties contribute to the temporal characteristics of the stress wave. In this work polycarbonate was used due to its high failure strengths, high stiffness coefficients, and its ability to be manufactured into thin, flexible sheets, thus allowing it to conform to

A. Navarro et al. / Laser Induced Shockwaves on Flexible Polymers

395

the curved surface of the body. A 1,064 nm Nd:YAG laser with a pulse duration of 3~6 ns is focused onto 380µm-thick polycarbonate substrate that is coated with a 0.5 µm aluminum thin-film, and a 50 to 100 μm thick layer of Na2SiO3 (waterglass) as shown in Figure 1. The waterglass is transparent to the 1,064 nm laser and acts as a confining layer, thereby lowering the temporal width of the stress profile.

Figure 1. Displacement interferometer setup.

A novel technique was developed to measure the 100-150 ns duration shockwaves through the use of a Michelson displacement interferometer[3-4]. The system incorporates a 1,064 nm Nd:YAG to generate the stress wave and a frequency stabilized 632.8 nm, 1mW HeNe laser used to measure the free surface velocity as the compressive wave travels and reflects as a tensile wave. The sample is coated with aluminum on both sides; one side is ablated with an ND:YAG laser and the other is used to reflect the HeNe laser. The 632.8 nm HeNe laser is directed through a 50/50 beam splitter to a reference mirror and the free surface of the coated substrate sample. The beams recombine and are focused onto a high speed photodetector coupled to a 5 GS/s digitizer as shown in Figure 1. As the wave propagates within the substrate, the free surface will move, causing the recombined signal HeNe signal to modulate between constructive and destructive interference and produces a down-chirped-like waveform. This can be used to reconstruct the input stress provided by the 1064 nm laser. Figure 2(a) depicts the waveform and a fit which can be used to determine the free surface velocity and ultimately the stress profile and Figure 2(b) displays the reconstructed pulse from the sampled data.

396

A. Navarro et al. / Laser Induced Shockwaves on Flexible Polymers

(a)

(b)

Figure 2. (a) Scan obtained from the Michelson displacement interferometer and the Al coated polycarbonate sample with fit. (b) Reconstructed shockwave profile.

3. Conclusion/Discussion A system to disrupt bacteria using laser generated shockwaves has been developed and characterized. The system produces high instantaneous peak stresses by ablating thin films of aluminum on polycarbonate. A peak stress of >60 MPa was measured using a displacement interferometer. Future experiments will test the efficacy of laser generated pulses on bacterial biofilms by coupling these pulsed shockwaves to biofilms grown on agar plates. Observation of cell death as a result of this technique will be assessed to determine the effectiveness of the technique and eventually optimize the shockwave parameters in order to maximize the bacterial disruption.

4. Acknowledgments The authors would like to thank Ms. Tiffany Chen, Ms. Neha Bajwa, Mr. Anthony Matolek, and Mr. Miguel Rocha for their vast knowledge of the cultivating and handling bacteria. The authors would also like to thank Dr. E. Carmack Holmes and Mrs. Cheryl Hein at CASIT for their support of this project.

References [1] [2] [3] [4]

Leid JG, Shirtliff ME, Costerton JW, Stoodley AP. Human leukocytes adhere to, penetrate, and respond to Staphylococcus aureus biofilms. Infect Immun 2002;70:6339–6345. Post JC, Hiller NL, Nistico L, et al. The role of biofilms in otolaryngologic infections: update 2007. Curr Opin Otolaryngol Head Neck Surg 2007;15:347–351. V. Gupta, A.S. Argon, J.A. Cornie and D.M. Parks, “Measurement of interface strength by laser pulse produced spallation,” Materials Science and Engineering, A126 (1990) 105-117. V. Gupta, A. S. Argon, D.M. Parks, and J.A. Cornie, “Measurement of interface strength by laser spallation experiment,” Journal of the Mechanics and Physics of Solids, 40, 1 (1992) 141-180.


397

Virtual Reality Haptic Human Dissection Caroline NEEDHAM, Caroline WILKINSON and Roger SOAMES Centre for Anatomy and Human Identification, University of Dundee, Scotland.

Abstract. This project aims to create a three-dimensional digital model of the human hand and wrist which can be virtually ‘dissected’ through a haptic interface. Tissue properties will be added to the various anatomical structures to replicate a realistic look and feel. The project will explore the role of the medical artist and investigate the cross-discipline collaborations required in the field of virtual anatomy. The software will be used to train anatomy students in dissection skills before experience on a real cadaver. The effectiveness of the software will be evaluated and assessed both quantitatively as well as qualitatively. Keywords. Haptic, dissection, virtual reality, anatomy, teaching, cadaver, hand and wrist, medical artist

Introduction Three dimensional anatomical models are increasingly used in the teaching of anatomy and in some instances replacing cadaveric dissection all together [1]. This is an area of some debate with arguments both for and against the retention of cadaveric teaching. Benefits of cadaveric dissection are said to include; understanding of structures in three dimensions, witnessing anatomical variability, the feel of different tissues, learning practical dissection skills, team-work, and exposure to death. There are additional benefits to cadaveric dissection which cannot be replicated by using computer models. However, research has shown that digital models can play an important role in anatomy education when used alongside traditional teaching methods [2]. The use of haptic devices to re-introduce the sense of touch to computer based anatomical models could further enhance their effectiveness by making the experience more comparable to the real thing. While haptic technologies are increasingly being used in several areas of medical science (for surgical and clinical skills training, clinical practice including diagnostics and surgery planning and simulation [3]) there has been little research in the area of virtual reality haptic dissection for the teaching of human gross anatomy. This research is being undertaken as a part time PhD at the University of Dundee with an expected submission date of June 2014. The first author currently works as a lecturer in medical art at the University of Dundee.

1. Methods and Materials This research is a work in progress. As such, the achievements to date are discussed followed by future plans.

398

C. Needham et al. / Virtual Reality Haptic Human Dissection

1.1. Achievements • • • • • • •

Dissection of the hand and wrist of two cadavers, one formalin fixed and one Thiel embalmed [4]. Notes and photographs for reference in replicating appearance, tactile experience and process have been taken. Data collected to model the bones. Sourced from CT scan of sixteen year-old hand from the Scheuer Collection of Juvenile bones at the University of Dundee. Data collected to model the muscles and skin. Sourced from The Visible Human Project (VHP), National Library of Medicine (NLM). Cryosection images taken at one-third of-a-millimetre intervals were used. Both data sets were segmented using Amira 5.2.2 to produce 3D reconstructions of each anatomical structure as separate 3D image file (fig 1). The individual structures have been compiled and refined in FreeForm Modeling (version 10, 64 bit) to create the hand and wrist model of the musculoskeletal system (fig 2). Vessels and nerves are being created in FreeForm Modeling and added to the model (fig 3) A crude, prototype dissection of the model is already possible through the FreeForm Modeling interface (fig 4).

Figure 1 (left). Segmentation in Amira. Figure 2 (centre). Musculoskeletal system modeled in FreeForm Modeling. Figure 3 (right). Musculoskeletal system with the addition of vessels, nerves and skin.

1.2. Future Developments The current model first requires completion in FreeForm Modeling. Tissue properties will then be programmed for each anatomical structure. Currently [5] this is achievable through a range of programming options. However, if it were possible to integrate this into existing software (such as FreeForm Modeling) as a menu option/tool it would open up this function to a wider range of individuals from different disciplines. Although it is currently possible to carve into the model, this could be improved to allow more realistic cutting, possibly utilizing a dual haptic interface. Finally a piece of bespoke software will be created for use by and testing of a relevant student population.

C. Needham et al. / Virtual Reality Haptic Human Dissection

399

2. Discussion Although good progress has been made so far, the most complex aspects of the project lie ahead. Collaboration with computer scientists is essential to complete the project to its highest potential. However, the prototype dissection which is already possible (fig 4), demonstrates that even without collaboration it will be feasible to create a pilot VR dissector within the FreeForm interface (minimizing all menus not required by the student to avoid distraction). 2.2. The Role of the Medical Artist The research will also explore the role of the medical artist and investigate the crossdiscipline collaborations required in the field of virtual anatomy. Developments in technology have affected artists since the first pigments were mixed to make paint. Artists have, for the most part, been both familiar with operating technology as well as using it to convey their message. However, as technology continues to advance, this is not always the case. The programming of VR worlds and objects is one such area where a multi-discipline collaboration is often required. The use of technology by the artist highlights an important distinction which must be made between the tool maker and the tool user. In this instance, the tool maker would be the computer programmer and the tool user would be the artist who uses these virtual tools (i.e. software programs) to create their art form. It is postulated that as well as being two different roles they may also usually require two different mindsets and therefore, frequently, two or more individuals.

Figure 4. Prototype dissection of the model is already possible within the FreeForm interface

References [1] [2] [3] [4]

[5]

MCLACHLAN, J., BLIGH, J., BRADLEY, P. & SEARLE, J. (2004) Teaching anatomy without cadavers. Medical Education, 38, 418-424. BIASUTTO, S. N., IGNACIO CAUSSA, L. & ESTEBAN CRIADO DEL RÍO, L. (2006) Teaching anatomy: Cadavers vs. computers? Annals of Anatomy - Anatomischer Anzeiger, 188, 187-190. FAGER, P. J. & WOWERN, P. V. (2005) The use of haptics in medical applications. The International Journal of Medical Robotics and Computer Assisted Surgery, 1, 36 - 42. GROSCURTH, P., EGGLI, P., KAPFHAMMER, J., RAGER, G., HORNUNG, J. P. & FASEL, J. D. H. (2001) Gross anatomy in the surgical curriculum in Switzerland: Improved cadaver preservation, anatomical models, and course development. Anatomical Record, 265, 254-256. MEIER, U., LOPEZ, O., MONSERRAT, C., JUAN, M. C. & ALCANIZ, M. (2005) Real-time deformable models for surgery simulation: a survey. Computer Methods and Programs in Biomedicine, 77, 183-197.

400


The Tool Positioning Tutor: A Target-Pose Tracking and Display System for Learning Correct Placement of a Medical Device Douglas A. NELSONa,c,1 and Joseph T. SAMOSKYa,b,c Department of Bioengineering, University of Pittsburgh b Department of Anesthesiology, University of Pittsburgh c Simulation and Medical Technology R&D Center, University of Pittsburgh a

Abstract. Safe and successful performance of medical procedures often requires the correct manual positioning of a tool. For example, during endotracheal intubation a laryngoscope is used to open a passage in the airway through which a breathing tube is inserted. During training it can be challenging for an experienced practitioner to effectively communicate to a novice the correct placement and orientation of a tool. We have implemented a real-time tracking and position display system to enhance learning correct laryngoscope placement. The system displays a 3D model of the laryngoscope. A clinical teacher can correctly position the laryngoscope to open the airway of a full-body simulator, then set this tool pose as the target position. The system displays to the learner the fixed, target pose and a real-time display of the current, “live” laryngoscope position. Positional error metrics are displayed as color-coded visual cues to guide the user toward successful targeting of the reference position. This technique provides quantitative assessment of the degree to which a learner has matched a specified “expert” position with a tool, and is potentially applicable to a wide variety of tools and procedures. Keywords. Human computer interaction, 3D visual guidance, real-time spatial tracking, simulation-based training, endotracheal intubation, laryngoscopy

Introduction In both clinical and simulation-based learning environments, communication of psychomotor skills from expert to learner for procedural training is typically done by example and by verbal guidance, but this may leave the learner lost when trying to duplicate the manipulations of the expert, and verbal guidance can be a less than optimal medium to specify a three-dimensional psychomotor outcome or how to achieve it efficiently. We developed the Tool Position Tutor to enhance the communication of optimal device positioning. The system provides 3D tracking of the pose of a laryngoscope during manipulation by expert and learner. A display of a virtual laryngoscope shows the desired position, and visual cues guide the learner to match the target pose. The novel foci of this work compared to previous research in tracking of laryngoscope pose 1 Corresponding author: Douglas A. Nelson, Simulation and Medical Technology R&D Center, University of Pittsburgh, 230 McKee Place, Suite 401, Pittsburgh, PA 15213; E-mail: [email protected].

D.A. Nelson and J.T. Samosky / The Tool Positioning Tutor

401

[1] are the real-time 3D visual models, associated target-pose interaction strategies, and superimposed visual cues designed to guide the learner toward expert performance.

Methods and Materials We employed a 3D electromagnetic tracking system (Aurora, Northern Digital Inc.) equipped with a 6 DOF sensor (Traxtal Dynamic Reference Body) to measure the position and orientation of a standard laryngoscope. A custom adapter, fabricated via stereolithography, enabled a press-fit attachment of the sensor to the end of the laryngoscope handle. The Aurora field generator was positioned under the head and neck of a full-body patient simulator commonly used for airway management training (SimMan®, Laerdal Medical AS). A 3D CAD model of the laryngoscope was created in SolidWorks. Software was developed that displayed the virtual laryngoscope tracking the pose of the real laryngoscope. “Pose Set” and “Pose Clear” functions can be specified via footswitches. The instructor positions the laryngoscope in the simulator’s airway, sets the pose, and the system captures the reference position, displayed as a static gray target. A second, dynamically tracked “learner” laryngoscope model is then displayed, which the learner endeavors to match to the target. The distance between the target and learner scope is measured at three locations (blade tip, hinge and handle top) and color-coded error bars corresponding to each distance error metric are displayed. The root-square of the three distance error measures is used as a criterion of closeness to target pose. The learner scope is dynamically color-coded to indicate closeness to the target. In addition, dynamic vectors (termed “bungee error cords”) that connect the three corresponding points on the learner and target scopes are displayed to help guide the learner toward matching the target.

Results We verified tracking within a workspace extending +/- 20 cm to the left and right of the mouth of the simulator, +/- 20 cm inferior and superior, and +20 cm anterior: the position of the laryngoscope during repeated intubation of the simulator’s airway was well within these limits. Figure 1a illustrates the set up and display interface of the system. The three screen shots of Figure 1(b,c,d) depict a learner moving the laryngoscope closer to the “expert” target position: the scope changes color from red to yellow to green (shown as dark to light in grayscale renditions of the Figure), and the associated visual error metrics decrease. At a user-specified error tolerance both a visual indicator and an auditory cue indicate a successful match with the target.

Discussion We are currently designing a trial to test the effectiveness of this system in aiding instruction in laryngoscope positioning. We plan to acquire expert and novice data [1][2], and to perform cluster analysis on the data to differentiate levels of proficiency. The clustered metrics could also form a database of reference positions that may help

402

D.A. Nelson and J.T. Samosky / The Tool Positioning Tutor

Figure 1. (a) Laryngoscope with attached tracking sensor. A 3D model of the laryngoscope is displayed tracking the pose of the real scope. (b-d) After the target position is fixed by an expert, the learner scope changes color (hue) to indicate proximity (or dark to light in grayscale). Dynamic visual cues indicate the distance error to three corresponding points on the target and learner scopes. When the learner and target scope are aligned within a preset threshold (d) the “goal light” illuminates with concomitant audio feedback.

support automated instruction and self-learning without the need for an expert instructor to be present. We are exploring a variety of visual cues that may help guide the user to successful matching of target positions. For example, preliminary tests offer promise that continuous-gradient, as opposed to discrete, color shading of the learner scope as a function of distance to the target scope may offer benefits in guiding the learner toward the target. We also envision the display incorporating 3D anatomic images in addition to the tracked scope to allow the learner to develop a mental model of the interactions between scope and internal anatomy that may not be visible to the learner during the procedure. We are also exploring applications of similar techniques to tool positioning in other procedures, including regional anesthesia and diagnostic ultrasound.

References [1] [2]

Delson NJ, Koussa N, Hastings RH, Weinger MB. Quantifying Expert vs Novice Skill In Vivo for Development of a Laryngoscopy Simulator. Stud Health Technol Inform, 2003. 94: p. 45-51. Stylopoulos N, Cotin S, Maithel SK, Ottensmeyer M, Jackson PG, Bardsley RS, et al. Computerenhanced laparoscopic training system (CELTS): bridging the gap. Surg Endosc, 2004. 18(5): p. 782-9.


403

A Cost Effective Simulator for Education of Ultrasound Image Interpretation and Probe Manipulation S.A. NICOLAUa,1 A. VEMURIa, H.S. WUa, M.H. HUANGa, Y. HOa, A. CHARNOZ b, A. HOSTETTLER b, C. FOREST b, L. SOLER b and J. MARESCAUX b a IRCAD Taiwan, Medical Imaging Team, 1-6 Lugong Road, Lukang 505 TAIWAN b IRCAD Strasbourg, 1 place de l’hopital 67091 Strasbourg FRANCE

Abstract. Ultrasonography is the lowest cost no risk medical imaging technique. However, reading an ultrasonographic (US) image as well as performing a good US probe positioning remain difficult tasks. Education in this domain is today performed on patients, thus limiting it to the most common cases. In this paper, we present a cost effective simulator that allows US image practice and realistic probe manipulation from CT data. More precisely, we tackle the issue of providing a realistic interface for the probe manipulation with a basic haptic feedback. Keywords. Ultrasound image simulation, training simulator, optical tracking

Introduction Education of young practitioners in most medical specialties, as a first step, is approached using a phantom that simulates the human body. However, most of them are very expensive and provide limited realistic experience to young practitioners. Because of their ease of use and better educative value to young practitioners, software based simulators in medical field have gained more importance in recent years. Indeed, they can reduce the cost and allow education on different kinds of pathology. Our final goal is to provide a US simulator for education that would allow for practice of US abdominal image interpretation and probe manipulation on patient database. The simulator we propose would offer the following advantages. Firstly, the student could work wherever and whenever he wants without needing to go to the hospital. Secondly, the time spent by a medical expert would be reduced, thus decreasing the education cost. Finally, students could practice on rare pathologies. To provide an efficient simulator for US image interpretation and probe manipulation, there are three major constraints to fulfil: realistic US image simulation, realistic probe interaction and a minimal haptic feedback. Vidal, Forest, Ni, Blum and Magee propose a US simulator for practicing needle puncture on patient dependent data [12, 5, 10, 2, 9]. The relative localization of the fake US probe is realized either using Omni © Sensable haptic feedback system, an EM tracking system (Ascension ©) or an optical tracking system (ARTrack ©) which makes the system price prohibitive. Cynydd is the only one to propose a cost effective 1


404

S.A. Nicolau et al. / A Cost Effective Simulator for Education of Ultrasound Image Interpretation

solution to education of US image interpretation [3]. The probe interaction is performed using a wiimote Nintendo © (~30 Euros). However, the system provides no haptic feedback and the probe rotation interaction is limited since the wiimote must be oriented toward the infrared emitter below the visualization screen. In this paper, we present the development of a low cost simulator for US education devoted to abdomen and that can be used on a standard workstation with a webcam. The principle is to load patient CT data (the skin is automatically segmented), and to move interactively a virtual US probe in the virtual scene on the skin reconstruction. The interaction is realized thanks to a 3D optical tracking of a fake probe that the user holds and moves on the surface of a phantom ensuring a haptic feedback (a cardboard box can also be used). From the position of the virtual US probe on the skin model, we compute the CT slice viewed by the probe. The final corresponding US image is provided thanks to an algorithm that transforms the CT image to US modality (already published in a patent [7]). Although the software part of the project is almost finished, this work is still in progress since we still need to demonstrate the clinical benefit of the system for student education. 1 . Methods & Materials The system is composed of a PC (workstation or laptop) equipped with one web camera. The phantom can be either a basic foam phantom or a simple cardboard box on which several optical markers are stuck (cf. Fig 1). To track the probe and the phantom, we use the ARToolkit+ library [1] to extract the 4 corners pixel coordinates of each individual marker and to identify them. Then, we use openCV library to refine the corner extraction [8]. Each set of 4 corners are then used to compute the marker pose in the camera frame. We print 5 markers that we stick on each face of a plastic cube, which is attached to the fake US probe (cf. Fig 1). Patient data is a CT volume (thickness

(3)

where y 4 T G2 G3 G7 Z V W D are the coordinates and are external forces and torques. , c, , and 1 are the kinetic, potential, dissipation, and constraint energies of the suture, respectively. To reduce the computation for the simulation of the suture model, we ignore the kinetic energy terms in Eq. (3) and compute an approximation of the constraint energy t ` @ A@y 4 r u @ A@y 4 due to E< AUE< U d PD K as a potential penalty energy j and put it back into the equation to replace the constraint energy terms [1]. By not adding the constraint directly into the equation, the movements of the suture’s centerlines can be computed separately from the movements of its directors, i.e. the orientation of the centerlines [1]. It means the need for solving the boundary value problem is reduced into solving two separate sets of a chain of semi-rigid bodies – centerlines and orientations. We simplify this model further by ignoring the damping energy term. Instead the previous velocity of each centerline is damped by a velocity damping constant - before the numerical time integration. Our final simplified equation is wz y:

r

w y:

wz y:

r

wz y:

r

w y:

V

Z Q>.

(4)

In discretized form, the computation is separated into two computations; one for forces applied on the centerlines and the other one for torques applied on the orientations: f JsM r JsM JsM and ig JM r i JM i JM.

(5)

The stretch forces f and bending ig torques are computed from centerlines and orientations, respectively. The constraint forces and torques i are computed from both centerlines and orientations. The stretch cf , bending cg , and constraint j energies and their discretized versions are derived same as in [1]. 2.2. The Solver The chain of point masses E4 representing the centerlines and the chain of quaternions l representing the material frames are loosely coupled by penalty forces and torques that constraint their movements towards a valid configuration. Since, the dynamic movement of the point masses is virtually decoupled from the dynamic movement of the quaternions, we can numerically time-integrate the point masses as point masses in a mass-spring system, and the orientation of quaternions as orientation of rigid bodies

470

S. Punak and S. Kurenov / Simplified Cosserat Rod for Interactive Suture Modeling

[1]. A semi-implicit Euler scheme described in [1] is used for the numerical time integration.

3. Results We use C++ with Object-Oriented Programming and OpenGL to implement the simulation of the surgical suture. Since the suture model computation is similar to a computation of two mass-spring chains, we could use GPU for the computation. We tried using NVIDIA CUDA for the computation of the suture simulation loop on GPU. To run the simulation on GPU, the simulation data had to be copied from the CPU memory to the GPU memory. For detecting and resolving the suture’s self intersection (including knot recognition) and interactions with tools and other objects in the scene we had to copy the result data from the GPU back to the CPU. Table 1 shows the average computation time (in ms) for a simulation loop (including collision detection) of our suture implementations on CPU, CPU+GPU via CUDA, and CPU+GPU via CUDA with CUDA Page-Locked Host Memory (PLHM). All simulations were tested on a desktop computer running Windows XP 32-bit OS, which is equipped with an Intel® Core™ i7-940 (2.93 GHz) CPU, 3 GB of DDR3 SDRAM, and an NVIDIA GeForce GTX 285 graphics card. A PHANTOM Omni® haptic device is used to manipulate the laparoscopic surgical tool. The suture collision detection including self intersection and knot recognition is based on a sphere bounding volume hierarchy. To resolve a collision, a penalty force is applied to exactly move the collided suture’s link or point away from the collided object or the other suture’s link due to self intersection. The penalty force is calculated as the penetration distance multiplied by mass and divided by the square of time step, which is similar to an inverse of the position based dynamics [13]. The results show that for small number of links, less than 512 which are used in our case, the computation on the CPU is faster. Therefore, we only continue developing our suture simulation on CPU. In the future, we may use GPU to simulate the suture if we can move all computations related to the suture’s intra- and inter-actions onto GPU. Our simulation gives an improvement in performance compared to the CORDE model [1] (Table 2), where FC and IT stand for force computation and integration time, respectively. The ratios in Table 2 shows the relative computation time based on 50 links. The simulation results show that our model still exhibits bending and twisting similar to a real suture (Figure 2).

Table 1. Average computation time (in ms) for a simulation loop (with collision detection). #links 32 64 128 256 512 … 4096

CPU 0.18 0.31 0.65 1.28 2.51 … 20.28

CPU + GPU 2.13 1.38 2.59 4.71 5.73 … 17.12

CPU + GPU (PLHM) 1.97 2.53 3.11 4.87 7.22 … 48.75


471

Table 2. Computation time (in ms) of the CORDE model and our suture model. #links 50 100 1000

CORDE FC + IT [1] Ratio 0.069 1.000 0.131 1.899 1.24 17.791

Our suture model FC + IT Ratio 0.0354 1.000 0.0594 1.678 0.5448 15.791

Table 3. The parameter values used for the simulations. Suture parameter length p (mm) diameter Q G [ ] (mm) mass n (g) material density + (g/m3) tensile modulus jf (GPa) bending modulus jg (kg/s2) shearing modulus k (GPa) penalty constraint stiffness (unitless) velocity damping constant - (%) time step (s)

Normal suture 100 0.5 +oG W = 10 12.73 10000 2500 200 100 50 0.001

Stiff suture 100 0.5 10 2000 10000 5000 5000 500 50 0.001

Figure 2. Tying a double knot.

To render the suture, a combined chain of point masses and orientations is created from the chain of point masses and the chain of orientations. The combined chain is then subdivided twice by the Chaikin’s Algorithm, similar to [7]. A generalized cylinder is generated and rendered for the subdivision chain. Figure 2 shows the simulations of tying a double knot on a fixed rigid object using a surgical instrument similar to the one used in [14]. Table 3 provides the parameters used for the simulations. It is noticeable that the values in the table are higher than in the real world. This is due to scaling adjustments in the virtual world. A stiff suture can be created by increasing the material density, moduli, and the penalty constraint stiffness. However, increasing the material density has to be compensated by reducing the time step. Therefore, we break the rule of physics by increasing the material density without increasing the mass, so that we do not have to reduce the time step, which keeps the influence of forces on the position of centerlines strong. The stiffness of the suture can be changed by setting the material density, since it directly influences the rotation of the orientations – the higher the material density, the lesser the rotation of the orientations.

472


4. Conclusion We have created a virtual surgical suture model based on our simplification of the CORDE model, which is in turn based on the Cosserat theory of elastic rods. The purpose of the simplification is to make the model run as fast as possible while keeping the model accuracy as much as possible. We tested the model with simple knot tying tasks on a rigid object. The test results show that our simplified model still exhibits bending and twisting similar to a real suture. It uses less computation time compared to the original CORDE model. We are working on adding interactions of the surgical suture model with other virtual objects in an environment designed for real-time surgical simulation.

References [1] J. Spillmann and M. Teschner. CORDE: Cosserat Rod Elements for the Dynamic Simulation of OneDimensional Elastic Objects. In Proceedings of the 2007 ACM SIGGRAPH/ Eurographics symposium on Computer animation, pages 63–72, Aire-la-Ville, Switzerland, Switzerland, 2007. Eurographics Association. [2] J. Brown, J.-C. Latombe, and K. Montgomery. Real-Time Knot-Tying Simulation. In The Visual Computer, volume 20(2-3), pages 165–179. Springer, May 2004. [3] J. Lenoir, P. Meseure, L. Grisoni, and C. Chaillou. A Suture Model for Surgical Simulation. In S. C. et D. Metaxas, editor, 2nd Int. Symp. on Medical Simulation (ISMS), volume 3078, pages 105–113, Cambridge M.A., June 2004. Springer Verlag. [4] D. K. Pai. STRANDS: Interactive Simulation of Thin Solids Using Cosserat Models. Computer Graphics Forum, 21(3), 2002. [5] J. Phillips, A. Ladd, and L. E. Kavraki. Simulated Knot Tying. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 841–846, 2002. [6] M. LeDuc, S. Payandeh, and J. Dill. Toward Modeling of a Suturing Task. In Proceedings of the Graphics Interface, pages 273–279. CIPS, Canadian Human-Computer Commnication Society, A K Peters, June 2003. [7] B. Kubiak, N. Pietroni, F. Ganovelli, and M. Fratarcangeli. A Robust Method for Real-Time Thread Simulation. In Proceedings of the 2007 ACM symposium on Virtual reality software and technology, pages 85–88, New York, NY, 2007. ACM. [8] F. Bertails, B. Audoly, M.-P. Cani, B. Querleux, F. Leroy, and J.-L. Lévêque. Super-Helices for Predicting the Dynamics of Natural Hair. In SIGGRAPH ’06: ACM SIGGRAPH 2006 Papers, pages 1180–1187, New York, NY, 2006. ACM. [9] M. Grégoire and E. Schömer. Interactive Simulation of One-Dimensional Flexible Parts. ComputerAided Design, 39(8):694–707, 2007. [10] L. Duratti, F. Wang, E. Samur, and H. Bleuler. A Real-Time Simulator for Interventional Radiology. In Proceedings of the 2008 ACM symposium on Virtual reality software and technology, pages 105–108, New York, NY, 2008. ACM. [11] M. Bergou, M. Wardetzky, S. Robinson, B. Audoly, and E. Grinspun. Discrete Elastic Rods. In SIGGRAPH ’08: ACM SIGGRAPH 2008 Papers, pages 1–12, New York, NY, USA, 2008. ACM. [12] D. Cao, D. Liu, and C. H.-T.Wang. Three-Dimensional Nonlinear Dynamics of Slender Structures: Cosserat Rod Element Approach. International Journal of Solids and Structures, 43(3–4):760–783, 2006. [13] M. Müller, B. Heidelberger, M. Hennix, and J. Ratcliff. Position Based Dynamics. Journal of Visual Communication and Image Representation, 18(2):109–118, 2007. [14] S. N. Kurenov, S. Punak, M. Kim, J. Peters, and J. C. Cendan. Simulation for Training with the Autosuture Endo Stitch Device. Surgical Innovation, 13(4):1–5, December 2006.


473

A Design for Simulating and Validating the Nuss Procedure for the Minimally Invasive Correction of Pectus Excavatum Krzysztof J. RECHOWICZ a,1 , Robert KELLY b Michael GORETSKY b Frazier W. FRANTZ b Stephen B. KNISLEY c Donald NUSS b and Frederic D. MCKENZIE a a

Modeling, Simulation, and Visualization Department, Old Dominion University b Pediatric Surgery, Children’s Hospital of The King’s Daughters c Mechanical Engineering Department, Old Dominion University Abstract. Surgical planners are used to achieve the optimal outcome for a surgery, especially in procedures where a positive aesthetic outcome is the primary goal, such as the Nuss procedure which is a minimally invasive surgery for correcting pectus excavatum (PE) - a congenital chest wall deformity. Although this procedure is routinely performed, the outcome depends mostly on the correct placement of the bar. It would be beneficial if a surgeon had a chance to practice and review possible strategies for placement of the corrective bar and the associated appearance of the chest. Therefore, we propose a strategy for the development and validation of a Nuss procedure surgical trainer and planner. Keywords. pectus excavatum, surgical planner

Introduction Pectus excavatum (PE) is a congenital chest wall deformity which is typically characterized by a deep depression of the sternum. The minimally invasive technique for the repair of PE (the Nuss procedure) has been proven to have a high success rate and satisfactory aesthetic outcome [1]. Although this procedure is routinely performed, the outcome depends mostly on the correct placement of the bar. It would be beneficial if a surgeon had a chance to practice and review possible strategies for placement of the corrective bar and the associated appearance of the chest. Therefore, we present the design of a Nuss procedure surgical planner and a strategy for its validation, taking into account the biomechanical properties of the PE ribcage, emerging trends in surgical planners, deformable models, and visualization techniques. 1 Corresponding Author: Krzysztof J. Rechowicz, Modeling, Simulation, and Visualization Department, Old Dominion University, Norfolk, VA 23529, USA; E-mail: [email protected].

474

K.J. Rechowicz et al. / A Design for Simulating and Validating the Nuss Procedure

Figure 1. The core design of the Nuss procedure.

1. Methods The core of our Nuss procedure surgical planner is based on a black-box approximation of a finite element model (FEM) of the PE ribcage (fig. 1) in order to ensure real-time performance which is not possible to obtain directly [2,3]. It includes development of a parametric model of the ribcage that can be deformed (based upon individual patient parameters obtained from CT slices to fit the PE ribcage. The core of the system is implemented in a virtual environment so deformation, triggered by the bar, can be visualized in the surgical planner. An average shape is being used for evaluation of the plan developed by the surgeon during training. This average has been developed based on a sample of normal subjects surface scans [4]. We leverage the core model design of the surgical planner to create a Nuss procedure surgical trainer with the interaction forces fed back to the user through a haptic interface. The system is meant to provide intelligent performance feedback based on predicted shape outcomes and comparisons to an averaged normal shape and to known successful post-surgical results for a specific case. The user would utilize this system to pick up a virtual scalpel, make incisions on a virtual PE chest, choose and insert a pectus bar into the PE chest, then receive a performance score. All is to be performed while receiving visual and touch feedback. Evaluation will be performed by experienced surgeons from the pectus clinic at the Children’s Hospital of the King’s Daughters, who regularly practice the Nuss procedure. Validation of the system will also be performed by testing the planner with previously operated cases. A user would recreate a scenario, i.e., the ribcage geometry and location of the bar, and compare a simulated outcome with the actual result. In this way, different cases can be studied in order to prove that the solution accomplishes its intended results.

2. Results We evaluated an average shape by comparing it with a normal chest without PE and post-operative chest shape. Based on differences between two shapes (presented as the

K.J. Rechowicz et al. / A Design for Simulating and Validating the Nuss Procedure

475

Figure 2. Comparison of the pre- and post-operative surface scans.

colormap), it is possible to quantify results. Differences between an average and normal chest shape were small up to 4 mm due to muscle structure which is not typically present in PE. Comparison with a post-operative shape showed overcorrection due to positive difference up to 30 mm. The same approach is used for evaluation of the outcome of the planning process. Additionally, we have already performed a comparison of three pairs of pre- and post-operative scans using a displacement map projected on the surface scan obtained prior to the surgery (fig. 2). For validation purposes, a similar approach will be used to compare simulated shapes with actual outcomes.

3. Conclusions In this paper, the approach for developing a real-time Nuss procedure planner has been presented. The proposed solution will utilize patient specific data, incorporate the biomechanical properties of the PE ribcage, and provide information about a post-operative shape of the chest based on the position of the bar. In addition, we have presented the initial outcome of before and after surface scans analysis as a means to validate results.

References [1]

A.D. Protopapas and T. Athanasiou, Peri-operative data on the Nuss procedure in children with pectus excavatum: independent survey of the first 20 years’ data, Journal of Cardiothoracic Surgery 40 (2008). [2] G. San Vicente, C. Buchartand, D. Borro, J. Celigueta, Maxillofacial surgery simulation using a massspring model derived from continuum and the scaled displacement method, International Journal of Computer Assisted Radiology and Surgery 4 (2009), 89–98. [3] U. Obaidellah, Z. Radzi, N.A. Yahya, N.A.A. Osman, A.F. Merican, The Facial Soft Tissue Simulation of Orthognathic Surgery Using Biomechanical Model, presented at 4th Kuala Lumpur International Conference on Biomedical Engineering 2008, Kuala Lumpur, Malaysia (2008). [4] K.J. Rechowicz, R. Kelly, M. Goretsky, F.W. Frantz, S. Knisley, D. Nuss, F.D. McKenzie, Development of an average chest shape for objective evaluation of the aesthetic outcome in the Nuss procedure planning process, in IFMBE Proceedings: 26th Southern Biomedical Engineering Conference SBEC 2010 32 (2010), 528–531.

476


AISLE: an Automatic Volumetric Segmentation Method for the Study of Lung Allometry Hongliang REN and Peter KAZANZIDES 1 Dept. of Computer Science, Johns Hopkins University, Baltimore, MD USA {hlren,pkaz}@jhu.edu Abstract. We developed a fully automatic segmentation method for volumetric CT (computer tomography) datasets to support construction of a statistical atlas for the study of allometric laws of the lung. The proposed segmentation method, AISLE (Automated ITK-Snap based on Level-set), is based on the level-set implementation from an existing semi-automatic segmentation program, ITK-Snap. AISLE can segment the lung field without human interaction and provide intermediate graphical results as desired. The preliminary experimental results show that the proposed method can achieve accurate segmentation, in terms of volumetric overlap metric, by comparing with the ground-truth segmentation performed by a radiologist. Keywords. AISLE, Automatic volumetric segmentation, Lung allometry

Introduction In order to develop a statistical atlas and study allometric laws of the lung, we need to extract the anatomical structures of the lung field from a population of volumetric CT datasets. Therefore, it is highly desirable to have an accurate segmentation method with minimum human intervention. There are quite a few semi-automatic segmentation algorithms and software toolkits available, such as Analyze [3], MIPAV [2], ITK [4] and Osirix [5]. However, for a large-scale population study, it is still a labor-intensive process for most of the semi-automatic methods. Therefore, it is desirable to have a fully automatic pipeline to get the whole segmentation done without human intervention.

1. Methods We developed a fully automatic segmentation method based on the 3D activecontour-evolution algorithms [1] of ITK-snap [6]. The active contour refers to a 1 Corresponding Author: Peter Kazanzides, Department of Computer Science, Johns Hopkins University.

H. Ren and P. Kazanzides / AISLE: An Automatic Volumetric Segmentation Method

477

Figure 1. Two key steps (located at the purple circles) in the intensity feature based active-contour segmentation: thresholding and bubble placement.

Figure 2. Histogram plots of the selected slices from different study sequences. The upper row is the original slice and the lower row is the corresponding histogram plots, where the deep dark histogram is the log-scale of the grey histogram.

closed contour that can evolve with time and space, and is driven by internal contour geometry and external force from the feature images. Hence, there are two key steps that need to be automated herein: creating binary intensity images after thresholding and initial contour (aka. bubble in ITK-snap) placement, as shown in Figure 1. The automatic thresholding step can be realized by utilizing a histogram-based Otsu filter, as it is good at identifying the threshold in contrast between lung field and background from the statistical histogram in Figure 2. The initial bubble (contour) placement is done by calculating the geometric center of the lobes from a connected-component filter. In practice, we only need to place one bubble for one lobe. The radius of the bubble is arbitrary as long as it does not fall across the other lobe. The whole pipeline is shown in Figure 3.

Figure 3. The pipeline for the AISLE algorithm

478

H. Ren and P. Kazanzides / AISLE: An Automatic Volumetric Segmentation Method

Figure 4. The DICE metric between the AISLE segmentation and manual segmentation

2. Results We have run 13 fully-automatic segmentations, and the results were validated by comparing with the manual segmentation performed by the radiologist. The DICE volumetric overlap metric is used to evaluate the segmentation performance, and the results in Figure 4 demonstrate the proposed algorithm is capable of achieving very accurate segmentation (average DICE >0.99 ).

3. Conclusions The proposed AISLE segmentation method can achieve decent results without human interaction for lung field extraction from volumetric CT datasets. The developed method is deployable in both Windows and Linux operating systems, and has an additional feature for showing the intermediate graphical results. The next step is to realize the edge-feature based automatic segmentation for the bronchial tree from thoracic images.

References [1] [2] [3] [4] [5] [6]

Vicent Caselles, Ron Kimmel, and Guillermo Sapiro. Geodesic active contours. International Journal of Computer Vision, 22(1):61–79, February 1997. http://mipav.cit.nih.gov/. http://www.analyzedirect.com/. http://www.itk.org/. http://www.osirix viewer.com/. P. A. Yushkevich, J. Piven, H. Cody, S. Ho, J. C. Gee, and G. Gerig. User-guided level set segmentation of anatomical structures with ITK-SNAP. Insight Jounral, 1, 2005. Special Issue on ISC/NA-MIC/MICCAI Workshop on Open-Source Software.


479

Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery Hongliang REN a , Denis RANK b , Martin MERDES b , Jan STALLKAMP and Peter KAZANZIDES a,1

b

a

Dept. of Computer Science, Johns Hopkins University, Baltimore, MD USA {hlren,pkaz}@jhu.edu b Fraunhofer Institute for Manufacturing Engineering and Automation, Stuttgart, Germany {rank,martin,stallkamp}@ipa.fhg.de Abstract. Navigation devices have been essential components for ImageGuided Surgeries (IGS) including laparoscopic surgery. We propose a wireless hybrid navigation device that integrates miniature inertial sensors and electromagnetic sensing units, for tracking instruments both inside and outside the human-body. The proposed system is free of the constraints of line-of-sight or entangling sensor wires. The main functional (sensor) part of the hybrid tracker is only about 15 mm by 15 mm. We identify the sensor models and develop sensor fusion algorithms for the proposed system to get optimal estimation of position and orientation (pose). The proof-of-concept experimental results show that the proposed hardware and software system can meet the defined tracking requirements, in terms of tracking accuracy, latency and robustness to environmental interferences. Keywords. Image guided surgery, Surgical navigation, Electromagnetic tracking, Inertial measurement unit, Sensor fusion

Introduction Real-time tracking of surgical instruments inside the human body poses unique challenges in developing tracking devices for minimally invasive surgeries. Optical tracking (OPT), the gold standard for surgical navigation, is bulky and blind when its line-of-sight (LOS) between the cameras and the markers is occluded [7]. Electromagnetic Tracking (EMT) [4] is feasible for laparoscopic surgery but notorious for its susceptibility to surrounding metallic or conductive surgical tools [6] and its reliance on a wired connection to the markers (coils). In addition, both of them have a limited working volume and OPT further has the restriction of angle of view relative to the optical camera. Ultrasound based [9,5] navigation or 1 Corresponding Author: Peter Kazanzides, Department of Computer Science, Johns Hopkins University.

480

H. Ren et al. / Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery

mechanical tracking [1] usually have handling inconveniences. Some prior work, such as [2][7], combined information from an optical tracker and EM tracker, which makes the operating room even more crowded with two bulky tracking systems, and can still suffer from the constraint of line-of-sight.

1. Objective Our objective is to develop a miniature tracking system that is easy-to-use, free of line-of-sight, angle-of-view, or cabling constraints, and with a reasonable working volume, without affecting surgical workflow. The desired technical specification is to achieve 6DOF target tracking accuracy of about 1 mm in position and 1 degree in orientation, with maximum latency of 100 milliseconds, minimum update frequency of 30 Hz, and with robustness against interference due to metallic objects, electrocautery, etc. Towards this end, we combine inexpensive miniature MEMS inertial sensors to compensate the distortions of the EM tracker, and to improve the dynamic behavior of the tracking system.

2. Material We employ a self-contained Inertial Measurement Unit (IMU), including accelerometer, gyroscope and magnetometer, to provide highly dynamic measurements with respect to global coordinates. For example, the accelerometer and magnetometer together can provide roll, pitch and heading measurements. The miniature electromagnetic sensor is used to provide an external reference to compensate the drift of the IMU. The prototype hardware is shown in Figure 1. It consists of two electromagnetic tracking systems (the commercial Aurora system from NDI [3] and a custom EMT system) and a sensor PCB which has integrated inertial sensors and electronics for signal processing. The commercial Aurora EMT system is included for the purpose of both validation and comparison. The rationale for developing a custom EM system is mainly due to the need for time synchronization and wireless operation. The custom EMT consists of a field generator and up to three receiving coils on the sensor PCB, which is collocated with the inertial sensors. Communication between the sensor PCB and a PC is via Bluetooth (for wireless operation) or USB (for debugging or firmware updates). Note that in this paper, the sensor fusion experiments used the commercial NDI Aurora system, as we are still working on the custom EMT to get comparable performance. The three axis accelerometer, used for detecting movements in the x, y and z directions, is the ST331DLH from STMicroelectronics and its measurement range is set to ±2g with an internal cutoff frequency of 780Hz. Two gyroscope sensors, the two-axis IDG300 and the single-axis ISZ300 from InvenSense, are used to measure the 3-axis angular rate. A three-axis AMR magnetometer, the Honeywell HMC1043 with sensitivity of each axis about 0.3mV/μT, serves as an electronic compass.


481

Figure 1. The hybrid tracking system consists of a hybrid tracker inside the handle of an endoscope (left), and an external field generator (right). The hybrid tracker is composed of inertial sensors, electromagnetic coils and other supporting electronic components. The field generator is a coil array with 4x4x3 transmitters.

3. Methods Because the raw measurements are from two different sensor-coordinate systems (EMT and IMU), it is necessary to register these coordinate systems before performing sensor fusion. The coordinate registration includes a body-frame registration solved by an AX=XB formulation [8], and base-frame registration solved by a paired-orientation formulation [11]. Figure 2 defines the coordinate systems and illustrates the two unknown transformations, X and Y. Fcoil Ffg X Field Generator

Fimu Y

Fnav : Navigation frame (North East Down)

Figure 2. Definition of the coordinate systems. X and Y are the two unknown transformations, corresponding to body-frame and base-frame transformations, respectively.

The two streams of registered measurements are subsequently fed to a sensor fusion module, which consists of an orientation estimator (rot) and a position estimator (pos), both based on Kalman filters, as shown in the block diagram of Figure 3. For the orientation estimation in Figure 3, the measurements are obtained from the IMU and EMT, as both subsystems can provide orientation measurements. We are weighting them based on the acceleration of the tracker: when the acceleration is small, the measurements from the IMU are more accurate; otherwise, the measurements from the EMT are more accurate.

482

H. Ren et al. / Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery fg

x

fg

q

Coil(s)

nav

Accel (x3) Magnetic (x3)

Gyro ( 3) (x3)

nav

nav

x g

q

q

Kalman Kalman filter (pos) fgR

~ x

fg

~ x

fg

q~

fg

~ q

nav

Attitude Meas. fgR nav

fggR

fg

fg

qˆ

Kalman filter (rot)

nav

Figure 3. Block diagram of the sensor fusion algorithm; x is position measurement, q is orientation measurement.

The orientation dynamics model is derived in time-derivative quaternion formulation as: q˙ =

1 [ω×] · q, 2

(1)

where q is the quaternion representation of the orientation, and [ω×] is a skew symmetric matrix of the angular velocity, ω, acquired from the gyroscope. We also include the gyroscope bias in the system dynamics, by assuming it is a random walk process. For position estimation, we are using the external position measurements from EMT as the reference and the dynamic model is derived from the kinematic relationship between position, velocity and acceleration, given by, x ¨nav = R · a − R · ω × x˙ nav − g,

(2)

where R is the rotation matrix from IMU frame to navigation frame, xnav , x˙ nav and x ¨nav are the position, velocity and acceleration vectors in the navigation frame, a is the acceleration measured by the accelerometer, ω is the angular velocity measured by the gyroscope, and g is the gravity vector.

4. Results We conducted a series of experiments to validate the proposed hybrid tracking system (HYB). First, we compared the orientation estimate between the hybrid tracker and the commercial Aurora EM tracker. Note that we are using the orientation estimates from the commercial NDI Polaris optical tracker (OPT) as the benchmark. Figure 4 shows the difference in the orientation estimate with respect to the optical tracker (i.e., HYB-OPT vs. EMT-OPT). The overall root-meansquare (RMS) tracking errors of the HYB for the 2 runs were 0.9 degrees, 0.8 degrees, 1.0 degrees, for roll, pitch and yaw, respectively. For EMT, the overall


483

Estimate difference wrt OPT: HYB vs. EMT

4

HYB EMT

3 2

degrees

1 0 -1 -2 -3 -4 roll

pitch

yaw

roll

pitch

yaw

runs

Figure 4. Measured orientation errors for proposed hybrid tracker and EMT, using optical tracking as ground truth.

RMS tracking errors were 1.9 degrees, 2.1 degrees, 2.1 degrees, for roll, pitch and yaw, respectively. We also compared the tracking performance in an earlier experiment [10] between HYB and EMT when the same metallic tool was moved around the tracker. The hybrid tracking method demonstrated its resistance to the environmental interference. Thus, the orientation estimation shows superior performance in terms of accuracy and robustness to metallic disturbance, compared to just the use of EM tracking. A trajectory tracking experiment is shown in Figure 5. The dynamic tracking performance of the hybrid tracker is better than just using the external EMT reference. Note that the hybrid system can obtain position measurements even when the EMT signals are missing for a short duration. In order to present the tracking information graphically, we implemented an OpenIGTLink [12] interface to enable rapid integration with IGT platforms such as 3D Slicer, as shown in Figure 6.

5. Conclusions & Outlook The proposed electromagnetic aided inertial navigation system demonstrated improved tracking performance in terms of tracking accuracy, data update rate, and tracking robustness. The integration of inertial sensing with an external reference, such as electromagnetic tracking, provides a promising solution for tracking surgical instruments during laparoscopic surgery. The external reference tracking system can provide

484

H. Ren et al. / Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery HYB EMT

160 140 120

Y(mm)

100 80 60 40 20 0 −350

−300

−250 X(mm)

−200

Figure 5. Trajectory of position estimation from HYB (blue line) and EMT (red line); HYB demonstrated better dynamic tracking performance.

Figure 6. Graphical representation of the tracking results in 3D Slicer through OpenIGTLink interface

stable correction for inertial sensor drifts and, in turn, the inertial sensor can provide better dynamic tracking results. A important future work is to validate the custom EMT system, including the calibration, localization and integration of the EMT for sensor fusion.

Acknowledgment This project is a joint development between The Johns Hopkins University (JHU) and the Fraunhofer Institute for Manufacturing Engineering and Automation (IPA) and is supported by internal funds from both institutions. The authors gratefully acknowledge the contributions of our colleagues Russell Taylor, Elliot McVeigh, Iulian Iordachita, and Anton Deguet, at JHU.


485

References [1]

[2]

[3] [4] [5]

[6]

[7]

[8] [9]

[10]

[11]

[12]

J. Bax, D. Cool, L. Gardi, K. Knight, D. Smith, J. Montreuil, S. Sherebrin, C. Romagnoli, and A. Fenster. Mechanically assisted 3D ultrasound guided prostate biopsy system. Medical Physics, 35(12):5397–5410, 2008. W. Birkfellner, F. Watzinger, F. Wanschitz, R. Ewers, and H. Bergmann. Calibration of tracking systems in a surgical environment. IEEE Trans. on Medical Imaging, 17(5):737– 742, Oct. 1998. http://www.ndigital.com/medical. V. Kindratenko. A survey of electromagnetic position tracker calibration techniques. Virtual Reality, 5(3):169–182, Sept. 2000. H.-H. Lin, C.-C. Tsai, and J.-C. Hsu. Ultrasonic localization and pose tracking of an autonomous mobile robot via fuzzy adaptive extended information filtering. IEEE Trans. on Instrumentation and Measurement, 57(9):2024–2034, Sept. 2008. C. Nafis, V. Jensen, and R. von Jako. Method for evaluating compatibility of commercial electromagnetic (EM) microsensor tracking systems with surgical and imaging tables. In SPIE Medical Imaging 2008: Visualization, Image-guided Procedures, and Modeling, volume 6918, pages 691820,1–15. SPIE, 2008. M. Nakamoto, Y. Sato, M. Miyamoto, Y. Nakamjima, K. Konishi, M. Shimada, M. Hashizume, and S. Tamura. 3D ultrasound system using a magneto-optic hybrid tracker for augmented reality visualization in laparoscopic liver surgery. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 148–155, London, UK, 2002. F. Park and B. Martin. Robot sensor calibration: solving AX=XB on the Euclidean group. IEEE Trans. on Robotics and Automation, 10(5):717–721, Oct 1994. J. F. Quinlan, H. Mullett, R. Stapleton, D. FitzPatrick, and D. McCormack. The use of the Zebris motion analysis system for measuring cervical spine movements in vivo. Proceedings of the Institution of Mechanical Engineers – Part H – Journal of Engineering in Medicine, 220(8):889 – 896, 2006. H. Ren and P. Kazanzides. Hybrid attitude estimation for laparoscopic surgical tools: A preliminary study. EMBC 2009. IEEE International Conference on EMBS, pages 5583– 5586, 2009. H. Ren and P. Kazanzides. A paired-orientation alignment problem in a hybrid tracking system for computer assisted surgery. Journal of Intelligent and Robotic Systems, accepted, 2010. J. Tokuda, G. S. Fischer, X. Papademetris, Z. Yaniv, L. Ibanez, P. Cheng, H. Liu, J. Blevins, J. Arata, A. J. Golby, T. Kapur, S. Pieper, E. C. Burdette, G. Fichtinger, C. M. Tempany, and N. Hata. OpenIGTLink: an open network protocol for image-guided therapy environment. The International Journal of Medical Robotics and Computer Assisted Surgery, 5(4):423–434, 2009.

486


Visualization of Probabilistic Fiber Tracts in Virtual Reality Tobias RICK

Anette VON KAPRI a,b Svenja CASPERS c Katrin AMUNTS c,b,d Karl ZILLES c,b,e Torsten KUHLEN a,b a Virtual Reality Group – RWTH Aachen University b Jülich-Aachen Research Alliance (JARA-HPC, JARA-BRAIN) c Institute of Neuroscience and Medicine, INM-1, INM-2, Research Center Jülich d Department of Psychiatry and Psychotherapy, RWTH Aachen University e C. and O. Vogt Institute for Brain Research, Heinrich-Heine-University Düsseldorf a,b,1

Abstract. Understanding the connectivity structure of the human brain is a fundamental prerequisite for the treatment of psychiatric or neurological diseases. Probabilistic tractography has become an established method to account for the inherent uncertainties of the actual course of fiber bundles in magnetic resonance imaging data. This paper presents a visualization system that addresses the assessment of fiber probabilities in relation to anatomical landmarks. We employ real-time transparent rendering strategy to display fiber tracts within their structural context in a virtual environment. Thereby, we not only emphasize spatial patterns but furthermore allow an interactive control over the amount of visible anatomical information. Keywords. probabilistic tractography, virtual reality

Introduction Neuroscientific research aims at understanding the structure-function relationship in the brain. Networks of communicating brain areas are required to fulfil motor, sensory as well as all mental and cognitive activities. The structural basis of such networks are nerve fibers connecting the participating brain areas. A profound knowledge about this connectivity structure is therefore necessary for understanding the computational activity of the brain. The mapping of nerve fibers and fiber bundles in the brain is also required to further understand psychiatric or neurological diseases. Currently, diffusion tensor magnetic resonance imaging (DT-MRI) provides the most forward method for the assessment of white matter fiber tracts in the living human brain. Hereby, the course of the fibers is estimated by measuring water diffusion in the brain. Based on their Brownian motion, water molecules prefer to move along directions with lowest resistance which in the brain is provided along the myelin sheaths. By applying magnetic field gradients from different spatial directions, the uncertainty within the diffusion data can be estimated and used for consecutive analysis. From these DT-MRI 1 Corresponding

Author: [email protected]

T. Rick et al. / Visualization of Probabilistic Fiber Tracts in Virtual Reality

487

Figure 1. Solid surface representation of a fiber pathway, image created with FSL [4] (left). Our volumetric rendering shows the probability distribution within the fiber tract (right).

data an effective diffusion tensor can be estimated within each voxel. The quantities as mean diffusivity, principal diffusion direction and anisotropy of the diffusion ellipsoid can be computed from the elements of the diffusion tensor [1]. To reconstruct fiber pathways based on the diffusion data, two main methods are currently used: (1) deterministic tractography, and (2) probabilistic tractography. Deterministic tractography tries to find the path from a seed to a target voxel based on the main diffusion direction within each voxel on the way. Hereby, uncertainty within the course of the fiber pathway cannot reliably be accounted for. In contrast, probabilistic tractography explicitly accounts for the uncertainty of the actual fiber tracts. For each voxel, a local probability distribution of the diffusion direction is calculated. A probabilistic tractography algorithm then tries to find the most probable course of a fiber between a seed and a target voxel by deciding in each voxel which would be the most probable prosecution of the fiber, based on the local probability distribution and its prior course [2]. As a result of probabilistic tractography, no single fiber strand is provided, but a probability distribution of possible fiber pathways between seed and target voxels, ranging from voxels with a large number of passed traces to voxels with only a low number of passes. The visualization of the probability in three dimensions (3D) is an essential step for the registration of the most likely course of a fiber bundle. Furthermore, anatomical information is required to reveal the fiber in its structural context. In this paper we extent the ideas from [3] and address both the visualization of probabilistic fiber tracts with a special focus on how to provide the required degree of anatomical context. We embed our visualization system in a virtual environment which not only improves depth perception due to stereoscopic projections but enables the use of direct interaction techniques such that the user becomes an integral part of the visualization pipeline. We use direct volume rendering for structural as well as fiber information in order to provide semi-transparent renderings in real-time. The amount of visible anatomical context can be controlled by a so-called magic lens interaction metaphor which we will refer to as virtual flashlight. The remainder of this paper is structured as follows. After briefly reviewing previous work in Section 1, we will describe our visualization and interaction approach in Section 2. We present the results in Section 3 and conclude our work in Section 4.

488


1. Related Work Deterministic streamline tractography emphasizes the course of the neuronal fibers using the principal eigenvector of the diffusion tensor [5]. To visualize the 3D large scale structure, Kindlmann [6] applied direct volume rendering strategies to the anisotropy values to map from the diffusion tensor data to color and opacity. Other common visualizations often make use of glyph-based techniques that represent a single tensor as a geometric primitive or via streamline advection among the principal eigenvector of the local tensor. Chen at al. [7] for example, merge ellipsoids to show the connectivity information in the underlying anatomy while characterizing the local tensor in detail. Sherbondy et al. [8] implemented interaction techniques to place and manipulate regions to selectively display deterministic fiber tracts that pass through specific anatomical areas. However, due to the relatively low resolution of DTI data as compared to the diameter of an axon, only the main fiber direction within each voxel is accounted for. Therefore, a main methodological issue are crossing fibers. Qazi et al. [9] successfully trace through regions of crossing fibers deterministically by extracting two tensors at any arbitrary position. Nevertheless, streamline methods only represent a single fiber path between two points without indication of correctness. Current probabilistic tractography algorithms [2] model different courses of fibers within each voxel using priors about the previous course of the estimated fiber tract and anatomical plausibility assumptions [10], thereby addressing the issue of crossing fibers adequately. Conveying uncertainty in the rendering is an inherent requirement for neuroscientists to evaluate probabilistic tractographies. The high interest in uncertainty and fiber crossing is shown in the recent work of Descoteaux et al.[11]. Deterministic and probabilistic tractography are compared with respect to crossing and splitting fiber bundles. In most current visualizations uncertainty is only represented on two-dimensional (2D) slices. 3D representations of probabilistic fiber tracts are often generated by extracting opaque isosurfaces for certain probability ranges (see Figure 1 left). In addition to visualization, the exploration of data also requires interactive manipulation. Therefore, we introduce the so-called magic lens interaction metaphor. It was first discussed by Bier et. al [12] as a 2D see-through user interface that changes the representation of content in a special window. A popular example is for instance a magnifying glass. In [13], Viega et. al extend the concept of magic lenses to virtual environments. They present the implementation of volumetric lenses that uses hardware clipping of geometric primitives to reveal the inner structure of objects. Whereas in [14] a magic box is used to present a higher-resolution of a flow visualization in order to focus attention on these regions and investigate them in more detail. 2. Method A major issue of current 3D visualization techniques in common DTI analysis tools is that no indication of uncertainty in the fiber tracts is contained in the final renderings. For instance in Figure 1 (left), the rendering of tracts is achieved by extracting an isosurface from the fiber tract but with no further clues to anatomical details or probability distribution within the fiber tract. However, anatomical context information is crucial for the registration of the most likely course of a fiber pathway in relation to structural landmarks.


489

Figure 2. Three-dimensional brain with fiber bundle (left). Brain areas provide additional anatomical context (right).

Figure 3. A user defined clipping region relates the fiber pathway with structural information.

2.1. Requirements To overcome such shortcomings, we formulate four conceptual requirements of our visualization system based on an interdisciplinary discussion with DTI domain experts as follows: (1) The visualization should emphasize spatial patterns and present the threedimensional physical structure in an intuitive fashion. (2) The final rendering should convey the uncertainty within each fiber tract. (3) The location of fiber tracts within the human brain should easily be deduced by the anatomical context. (4) None of the above requirements must interfere with the interactivity of the visualization system. 2.2. Visualization Technique We employ a direct volume rendering as the underlying rendering technique for our visualization system. However, our visualization system requires the display of multiple and transparent volumetric (voxel-based) information simultaneously. Here, state-of-the-art techniques for direct volume rendering are no longer sufficient for an interactive visualization in a virtual environment. The main reason for this is that the process of rendering transparent objects, which usually relies on either depth sorting or ray casting, is a complex process in general and becomes even more demanding the more objects are involved. However, we can exploit the fact, that most medical datasets are already registered in a common reference space. Therefore, we adapt classical slice-based volume rendering to efficiently handle multiple co-registered data sets as follows: First, we interleave the data

490


Figure 4. Users in an immersive CAVE virtual environment (left). A spatial input device allows interaction directly in 3D (right).

sets into one vector-valued data field. The proxy geometry (texture slices) is setup such that it represents the shared reference space and is rendered only once which alleviates the problems of depth-sorting multiple proxy geometries. Then, a special shader program handles each integration step of the individual data sets, separately. In a subsequent step the temporary integration values are interpolated according to user setting (e.g. maximum intensity or weighted sum). Hence, fiber probability and structural information can be classified according to separate transfer functions but form a consistent and correctly depth-sorted transparent image (cf. Figure 1 right). 2.3. Anatomical Context The anatomical context is provided by including a standardized reference brain (Figure 1). As illustrated in Figure 2, the opaque rendering of cross sections of the brain resembles the 2D slices the domain expert are familiar with from common DTI tools and provides an unbiased view on the original data. The rendering of functional or cortical defined brain areas is used to give additional clues to the anatomical connection of fiber tracts. Additionally, the reference brain is volume rendered semi-transparently with userdefined clipping regions in order to reduce visual cluttering. The clipping regions can either be simple axis-aligned planes or can directly be controlled by the user via a virtual clipping cone (virtual flashlight) as depicted in Figure 3. 2.4. Emphasizing Spatial Patterns We use the virtual reality toolkit ViSTA [15] as basis for our implementation. This allows the deployment of our visualization system on common desktop computers as well as on immersive virtual environments (cf. Figure 4 left). Depending on the available hardware infrastructure, this also allows the combination of 3D rendering with user-centered projection (head-tracking) which increases the overall depth perception, significantly. In addition to stereoscopic vision, direct interaction where the user takes an active role is an integral part of every interactive virtual reality system. We have incorporated a direct interaction metaphor into our visualization system, the virtual flashlight. Similar to the beam of a flashlight, the user can directly control the amount of visible anatomical structure by a 3D interaction device (cf. Figure 4 right). Interesting parts of the probabilistic fiber tracts can be revealed and referenced with the anatomical landmarks with reduced occlusion or visual clutter. This allows a more accurate inspection of the anatomic structure in the direct vicinity of fiber pathways. The concept is illustrated by the image series in Figure 5.


491

Figure 5. The user can control the clipping region with a virtual flashlight in the CAVE virtual environment.

3. Results The data used for all visualizations were obtained in the Institute of Neuroscience and Medicine of the Research Centre Jülich and the C. and O. Vogt Institute for Brain Research of the Heinrich-Heine-University Düsseldorf. The brain areas shown here were depicted from the Jülich-Düsseldorf cytoarchitectonic atlas [16]. All data were displayed on the standard reference brain of the Montreal Neurological Institute (MNI) as internationally used as common reference space (voxel resolution: 1mm3 ). Domain experts state that by combining anatomical information from the reference brain with overlaying fiber tracking results, the visualization gives first hints to the anatomical context of the fiber tracts. Former visualization software most widely used in DTI tractography research only reconstructed fiber tracts in 3D as solid paths without any information about the uncertainty. Therefore, the coding of different probability values with different colors and transparencies allows a 3D impression of the fiber tract while still revealing its main direction and the uncertainty around it. Furthermore, the new visualization method allows interactive manipulation of the magnitude of anatomical information displayed which was hardly possible in former software packages. 4. Conclusion Our work addresses the visualization of probabilistic fiber tracts in the human brain. Here, the comprehension of the course of the fiber in relation to its confidence is one of the most crucial steps. The interactive 3D visualization of probabilistic fiber tracts referenced with their anatomical landmarks allows the domain scientists to directly interpret their results in 3D. Hereby, reducing the additional mental workload previously required from judging 2D slices or missing uncertainty information in non-interactive 3D plots. We have embedded our visualization in a virtual reality application which increases the depth perception of structural patterns and enables direct interaction metaphors due

492


to tracking of 3D input devices. Furthermore, the degree of anatomical information necessary in order to establish a relationship between nerve fibers and structural landmarks can be controlled by the virtual flashlight metaphor. Here, the user is provided with a fine-grain control which parts of the structural information is cut away while the fiber tracts remain visible in the cone of the virtual flashlight.

References [1] [2]

[3] [4] [5] [6] [7] [8] [9]

[10] [11] [12] [13] [14] [15] [16]

P. J. Basser, J. Mattiello, and D. Lebihan, “MR diffusion tensor spectroscopy and imaging,” Biophysical Journal, vol. 66, pp. 259–267, 1994. T. Behrens, H. Johansen-Berg, M. Woolrich, S. Smith, C. Wheeler-Kingshott, P. Boulby, G. Barker, E. Sillery, K. Sheehan, O. Ciccarelli, A. Thompson, J. Brady, and P. Matthews, “Non-invasive mapping of connections between human thalamus and cortex using diffusion imaging,” Nature Neuroscience, vol. 6, no. 7, pp. 750–757, 2003. [Online]. Available: http://dx.doi.org/10.1038/nn1075 A. von Kapri, T. Rick, S. Caspers, S. B. Eickhoff, K. Zilles, and T. Kuhlen, “Evaluating a visualization of uncertainty in probabilistic tractography,” K. H. Wong and M. I. Miga, Eds., vol. 7625, no. 1. SPIE, 2010, p. 762534. “FSL 4.1,” August 2008. [Online]. Available: http://www.fmrib.ox.ac.uk/fsl/ G. Kindlmann, “Visualization and analysis of diffusion tensor fields,” Ph.D. dissertation, School of Computing, University of Utah, 2004. G. Kindlmann, D. Weinstein, and D. Hart, “Strategies for direct volume rendering of diffusion tensor fields,” IEEE Transactions on Visualization and Computer Graphics, vol. 6, no. 2, pp. 124–138, 2000. W. Chen, S. Zhang, S. Correia, and D. F. Tate, “Visualizing diffusion tensor imaging data with merging ellipsoids,” IEEE Pacific Visualization Symposium, vol. 0, pp. 145–151, 2009. A. Sherbondy, D. Akers, R. Mackenzie, R. Dougherty, and B. Wandell, “Exploring connectivity of the brain’s white matter with dynamic queries,” IEEE Transactions on Visualization and Computer Graphics, vol. 11, no. 4, pp. 419–430, 2005. A. Qazi, G. Kindlmann, L. O’Donnell, S. Peled, A. Radmanesh, S. Whalen, A. Golby, and C.-F. Westin, “Two-tensor streamline tractography through white matter intra-voxel fiber crossings: Assessed by fMRI,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops., 2008, pp. 1–8. T. Behrens, H. Johansen-Berg, S. Jbabdi, M. Rushworth, and M. Woolrich, “Probabilistic diffusion tractography with multiple fibre orientations: What can we gain?” NeuroImage, vol. 34, no. 1, pp. 144–155, 2007. [Online]. Available: http://dx.doi.org/10.1016/j.neuroimage.2006.09.018 M. Descoteaux, R. Deriche, T. Knosche, and A. Anwander, “Deterministic and probabilistic tractography based on complex fibre orientation distributions,” IEEE Transactions on Medical Imaging, vol. 28, no. 2, pp. 269–286, 2009. E. Bier, M. Stone, and K. Pier, “Enhanced illustration using magic lens filters,” IEEE Computer Graphics and Applications, vol. 17, no. 6, pp. 62–70, 1997. J. Viega, M. J. Conway, G. Williams, and R. Pausch, “3d magic lenses,” in UIST ’96: Proceedings of the 9th annual ACM symposium on User interface software and technology. New York, NY, USA: ACM, 1996, pp. 51–58. A. Fuhrmann and E. Gröller, “Real-time techniques for 3d flow visualization,” in VIS ’98: Proceedings of the conference on Visualization ’98. Los Alamitos, CA, USA: IEEE Computer Society Press, 1998, pp. 305–312. I. Assenmacher and T. Kuhlen, “The ViSTA Virtual Reality Toolkit,” The SEARIS Workshop on IEEE VR 2008, Reno, 2008. K. Zilles and K. Amunts, “Receptor mapping: architecture of the human cerebral cortex,” Current Opinion in Neurology, vol. 22, no. 4, pp. 331–339, 2009.


493

NeuroVR 2 - A Free Virtual Reality Platform for the Assessment and Treatment in Behavioral Health Care Giuseppe RIVA 1-3, Andrea GAGGIOLI 1-2, Alessandra GRASSI 1-2, Simona RASPELLI 1, Pietro CIPRESSO 1, Federica PALLAVICINI 1, Cinzia VIGNA1, Andrea GAGLIATI 3 Stefano GASCO 3, Giuseppe DONVITO 3 1 Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, Milan, Italy 2 Psychology Department, Catholic University of Milan, Italy 3 Virtual Reality & Multimedia Park, Turin, Italy

Abstract. At MMVR 2007 we presented NeuroVR (http://www.neurovr.org) a free virtual reality platform based on open-source software. The software allows non-expert users to adapt the content of 14 pre-designed virtual environments to the specific needs of the clinical or experimental setting. Following the feedbacks of the 2000 users who downloaded the first versions (1 and 1.5), we developed a new version – NeuroVR 2 (http://www.neurovr2.org) – that improves the possibility for the therapist to enhance the patient’s feeling of familiarity and intimacy with the virtual scene, by using external sounds, photos or videos. More, when running a simulation, the system offers a set of standard features that contribute to increase the realism of the simulated scene. These include collision detection to control movements in the environment, realistic walk-style motion, advanced lighting techniques for enhanced image quality, and streaming of video textures using alpha channel for transparency. Keywords: Virtual Reality, Assessment, Therapy, NeuroVR, Open Source

1. Introduction The use of virtual reality (VR) in medicine and behavioral neurosciences has become more widespread. This growing interest is also highlighted by the increasing number of scientific articles published each year on this topic: searching Medline with the keyword “virtual reality”, we found that the total number of publications has increased from 45 in 1995 to 3203 in 2010, showing an average annual growth rate of nearly 15%. Although it is undisputable that VR has come of age for clinical and research applications [1-3] the majority of them are still in the laboratory or investigation stage. In a recent review [4], Riva identified four major issues that limit the use of VR in psychotherapy and behavioral neuroscience: • the lack of standardization in VR hardware and software, and the limited possibility of tailoring the virtual environments (VEs); • the low availability of standardized protocols; • the high costs (up to 200,000 US$) required for designing and testing a clinical VR application; • most VEs in use today are not user-friendly.

494

G. Riva et al. / A Free VR Platform for the Assessment and Treatment in Behavioral Health Care

To address these challenges, we presented At MMVR 2007 NeuroVR (http://www.neurovr.org) a free virtual reality platform based on open-source software [5]. The software allows non-expert users to adapt the content of 14 pre-designed virtual environments to the specific needs of the clinical or experimental setting. Following the feedbacks of the 1000 users who downloaded the first version, we developed a new version – NeuroVR 2 (http://www.neurovr2.org) – that improves the possibility for the therapist to enhance the patient’s feeling of familiarity and intimacy with the virtual scene, by using external sounds, photos or videos.

2. NeuroVR 2 Using NeuroVR 2, the user can choose the appropriate psychological stimuli/stressors from a database of objects (both 2D and 3D) and videos, and easily place them into the virtual environment. The edited scene can then be visualized in the Player using either immersive or non-immersive displays. Currently, the NeuroVR library includes 18 different virtual scenes (apartment, office, square, supermarket, park, classroom, etc.), covering some of the most studied clinical applications of VR: specific phobias, cognitive rehabilitation, panic disorders and eating disorders. Specifically, the new version now includes full sound support and the ability of triggering external sounds and videos using three different approaches: the keyboard, timeline or proximity. The VR suite leverages two major open-source projects in the VR field: Delta3D (http://www.delta3d.org) and OpenSceneGraph (http:// www.openscenegraph.org). Both are building components that integrates with ad-hoc code to handle the editing and the simulation.The NeuroVR2 Editor's GUI is now based on the QT cross-platform application and UI framework from Nokia (http://qt.nokia.com/) that grants an higher level of editing and customization over the editor functionalities, while the graphical rendering is done using OpenSceneGraph, an open source high performance 3D graphics toolkit (http://www.openscenegraph.org/projects/osg). All the scenes building can now be done by the therapists using a cleaner and simpler interface, and through a powerful "Action and Trigger" system and an easy to use interface exposed by the editor. The scene creator can now also define how the scene reacts to the patients behavior, when he is using the scene in the VR Player. The NeuroVR2 Player too has been largely rewritten to grant a more efficient workflow for the scenes playback and has a brand new startup interface written in QT. The whole suite is developed in C++ language, targeted for the Microsoft Windows platform but fully portable to other systems if needed. The key characteristics that make NeuroVR suitable for most clinical applications are the high level of control of the interaction with the tool, and the enriched experience provided to the patient. These features transform NeuroVR in an “empowering environment”, a special, sheltered setting where patients can start to explore and act without feeling threatened. Nothing the patient fears can “really” happen to them in VR. With such assurance, they can freely explore, experiment, feel, live, and experience feelings and/or thoughts. NeuroVR thus becomes a very useful intermediate step between the therapist’s office and the real world. Actually, NeuroVR is used in the assessment and treatment of Obesity [6], Alcohol Abuse [7], Anxiety Disorders [1], Generalized Anxiety Disorders [8] and Cognitive Rehabilitation [9; 10].

G. Riva et al. / A Free VR Platform for the Assessment and Treatment in Behavioral Health Care

495

3. Conclusions In this chapter, we introduced NeuroVR 2, the new version of an advanced platform designed for the creation and customization of highly flexible VEs for clinical psychology and behavioral neurosciences. A future goal is to provide software compatibility with instruments that allow collection and analysis of behavioral data, such as eye-tracking devices and sensors for psycho-physiological monitoring. Beyond clinical applications, NeuroVR provides the VR research community with a free “VR lab”, which allows the creation of highly-controlled experimental simulations for different of behavioral, clinical and neuroscience applications

4. Acknowledgments The NeuroVR development was partially supported by the European funded project “Interstress” – Interreality in the management and treatment of stress-related disorders (FP7-247685).

5. References A. Gorini and G. Riva, Virtual reality in anxiety disorders: the past and the future, Expert Review of Neurotherapeutics 8 (2008), 215-233. [2] T.D. Parsons and A.A. Rizzo, Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: A meta-analysis, Journal of Behavior Therapy and Experimental Psychiatry 39 (2008), 250-261. [3] G. Riva and A. Gaggioli, Virtual clinical therapy, Lecture Notes in Computer Sciences 4650 (2008), 90107. [4] G. Riva, Virtual reality in psychotherapy: review, Cyberpsychology & Behavior 8 (2005), 220-230; discussion 231-240. [5] G. Riva, A. Gaggioli, D. Villani, A. Preziosa, F. Morganti, R. Corsi, G. Faletti, and L. Vezzadini, NeuroVR: an open source virtual reality platform for clinical psychology and behavioral neurosciences, Studies in Health Technology and Informatics 125 (2007), 394-399. [6] G. Riva, M. Bacchetta, G. Cesa, S. Conti, G. Castelnuovo, F. Mantovani, and E. Molinari, Is severe obesity a form of addiction? Rationale, clinical approach, and controlled clinical trial, CyberPsychology and Behavior 9 (2006), 457-479. [7] E. Gatti, R. Massari, C. Sacchelli, T. Lops, R. Gatti, and G. Riva, Why do you drink? Virtual reality as an experiential medium for the assessment of alcohol-dependent individuals, Studies in Health Technology and Informatics 132 (2008), 132-137. [8] F. Pallavicini, D. Algeri, C. Repetto, A. Gorini, and G. Riva, Biofeedback, VR and Mobile Phones in the treatment of Generalized Anxiety Disorders: A phase-2 controlled trial, Journal of CyberTherapy & Rehabilitation 2 (2009), 315-328. [9] S. Raspelli, L. Carelli, F. Morganti, B. Poletti, B. Corra, V. Silani, and G. Riva, Implementation of the multiple errands test in a NeuroVR-supermarket: a possible approach, Studies in Health Technology and Informatics 154, 115-119. [10] G. Albani, S. Raspelli, L. Carelli, F. Morganti, P.L. Weiss, R. Kizony, N. Katz, A. Mauro, and G. Riva, Executive functions in a virtual world: a study in Parkinson's disease, Studies in Health Technology and Informatics 154, 92-96. [1]

496


Personal Health Systems for Mental Health: The European Projects Giuseppe RIVA 1-2, Rosa BANOS 3, Cristina BOTELLA 4, Andrea GAGGIOLI 1-2, Brenda K WIEDERHOLD 5 1 Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, Milan, Italy 2 Psychology Department, Catholic University of Milan, Italy 3 University of Valencia, Spain 4 Jaume I University, Spain 5 Virtual Reality Medical Center Europe, Bruxelles, Belgium

Abstract. Since the European funded project VREPAR - Virtual Reality in NeuroPsycho-Physiology (1995) – different European research activities have been using virtual reality and advanced information and communication technologies to improve the quality of care in the treatment of many different mental health disorders including anxiety disorders, eating disorders and obesity. Now the European Commission funding is shifting from the traditional hospital-centred and reactive healthcare delivery model toward a person-centred and preventive one. The main outcome of this shift is the “Personal Health Systems” (PHS) paradigm that aims at offering continuous, quality controlled, and personalized health services to empowered individuals regardless of location. The paper introduces four recently funded projects – Interstress, Monarca, Optimi and Psyche – that aim at using PHS based on virtual reality, biosensors and/or mobile technologies to improve the treatment of bipolar disorders, depression and psychological stress. Keywords: Virtual Reality, Assessment, Therapy, Personal Health Systems, Biosensors, Mobile Technologies, Stress, Depression, Bipolar Disorders.

1. Introduction New Information and Communication Technologies (ICT) offer clinicians significant new abilities to monitor patients’ conditions, thereby enabling them to diagnose problems earlier and treat them more effectively. For this reason, since the European funded project VREPAR - Virtual Reality in Neuro-Psycho-Physiology (1995 – 4th Framework Programme) – different European research activities have been using virtual reality and advanced information and communication technologies to improve the quality of care in the treatment of many different mental health disorders: anxiety disorders, male sexual disorders, eating disorders and obesity. Recently, the European Commission focus shifted from the traditional hospitalcentered and reactive healthcare delivery model toward a person-centered and preventive one. The main outcome of this shift is the “Personal Health Systems” (PHS) paradigm that aims at offering continuous, quality controlled, and personalized health services to empowered individuals regardless of location [1].

G. Riva et al. / Personal Health Systems for Mental Health: The European Projects

497

PHS cover a wide range of systems including wearable, implantable or portable systems, as well as Point-of-Care (PoC) diagnostic devices. Typically, the functioning of PHS is related to three main blocks as shown below [2]:

Figure 1. The structure of a Personal Health System (from [3])

1.

2.

3.

Data Acquisition: Collection of data and information related to the health status of a patient or healthy individual, e.g., through the use of sensors and monitoring devices. Data Analysis: Processing, analysis and interpretation of the acquired data to identify what information is clinically relevant and useful in diagnosis, management or treatment of a condition. This entails processing of data at both ends: locally at the site of acquisition (e.g., with on-body electronics) and remotely at medical centres. Data processing and interpretation takes into account the established medical knowledge and professional expertise where appropriate. Patient/Therapist Communication: Communication and feedback between various actors, in a loop: from patient/individual to medical centre; from medical centre that analyses the acquired data to doctor/hospital; and back to the patient/individual from either the wearable/portable/implantable system itself or the doctor or the medical centre (e.g., in the form of personalised feedback and guidance to the patient, adjusted treatment via closed loop therapy, control of therapy devices).

2. Personal Health Systems for Mental Health The European Commission is supporting research in this area under the Seventh Framework Programme (FP7). FP7 funds are used to support research into monitoring systems for patients with chronic diseases. In particular, such tools should provide improved quality of life for chronically ill patients, enabling them to stay at home rather than have to be admitted to hospitals. With ICT systems able to monitor a range

498


of parameters related to the patient’s condition, medical professionals can take timely decisions on the most effective treatment. Automatic alerts ensure doctors are immediately made aware of changes in the patient’s condition and can respond to prevent severe deteriorations. This approach can also used to improve mental health treatment. While most of us immediately think of either drugs or traditional talk therapy as the primary tools for mental health problems, there is a long history of using technologies for the diagnosis and treatment of psychological disorders. Specifically, PHS help us to connect on a level never seen in history; and for individuals less likely to seek professional help, they provide a confidential self-paced avenue towards change. For these reasons, the FP7 decided to support ICT based research projects providing solutions for persons suffering from stress, depression or bipolar disorders. These projects should address the parallel development of technological solutions, as well as new management or treatment models based on closed-loop approaches. Emphasis will be on the use of multi-parametric monitoring systems, which monitor various metrics related to behavior and to bodily and brain functions (e.g. activity, sleep, physiological and biochemical parameters). More, the required systems should aim at (i) objective and quantitative assessment of symptoms, patient condition, effectiveness of therapy and use of medication; (ii) decision support for treatment planning; and (iii) provision of warnings and motivating feedback. In the cases of depression and bipolar disorders, the systems should also aim at prediction of depressive or manic episodes. The solutions should combine wearable, portable or implantable devices, with appropriate platforms and services. Finally, they should promote the interaction between patients.

3. Personal Health Systems for Mental Health: The Funded Projects After a very demanding selection, the Commission provided financial support to the following four projects– Interstress, Monarca, Optimi and Psyche – that aim at using PHS based on virtual reality, biosensors and/or mobile technologies to improve the treatment of bipolar disorders, depression and psychological stress. Below there is a short description of their contents. 3.1. Interstress “Psychological Stress” occurs when an individual perceives that environmental demands tax or exceed his or her adaptive capacity . According to the Cochrane Database of Systematic Reviews the best validated approach covering both stress management and stress treatment is the Cognitive Behavioural (CBT) approach. Typically, this approach may include both individual and structured group interventions (10 to 15 sessions) interwoven with didactics. It includes in-session didactic material and experiential exercises and out-of-session assignments (practicing relaxation exercises and monitoring stress responses). The intervention focuses on: - Learning to cope better with daily stressors (psychological stress) or traumatic events (post traumatic stress disorder), - and optimizing one's use of personal and social resources.


499

CBT has undergone a very large number of trials in research contexts. However it has been less efficacious in clinical contexts and it has become obvious that CBT has some failings when applied in general practice. INTERSTRESS aims to design, develop and test an advanced ICT based solution for the assessment and treatment of psychological stress that is able to address three critical limitation of CBT: • The therapist is less relevant than the specific protocol used. • The protocol is not customized to the specific characteristics of the patient. • The focus of the therapy is more on the top-down model of change (from cognitions to emotions) than on the bottom-up (from emotions to cognitions). To reach this goal the project will use a totally new paradigm for e-health - Interreality [4; 5] – that integrates assessment and treatment within a hybrid environment, bridging physical and virtual world. Our claim is that bridging virtual experiences – fully controlled by the therapist, used to learn coping skills and emotional regulation - with real experiences – that allows both the identification of any critical stressors and the assessment of what has been learned – using advanced technologies (virtual worlds, advanced sensors and PDA/mobile phones) is the best way to address the above limitations. These devices are integrated around two subsystems - the Clinical Platform (inpatient treatment, fully controlled by the therapist) and the Personal Mobile Platform (real world support, available to the patient and connected to the therapist) – that will be able to provide: (i) Objective and quantitative assessment of symptoms using biosensors and behavioural analysis; (ii) Decision support for treatment planning through data fusion and detection algorithms; and provision of warnings and motivating feedback to improve compliance and long-term outcome. By creating a bridge between virtual and real worlds, Interreality allows a full-time closed-loop approach actually missing in current approaches to the assessment and treatment of psychological stress: • The assessment is conducted continuously throughout the virtual and real experiences: it enables tracking of the individual’s psycho-physiological status over time in the context of a realistic task challenge. • The information is constantly used to improve both the appraisal and the coping skills of the patient: it creates a conditioned association between effective performance state and task execution behaviours.

3.2. Monarca Manic-depression psychosis also known as bipolar disease is a mood disorder characterized by alternating periods of mania and depression. The current methodologies of diagnosis of this disease are based on self-reported experiences, typically done after a crisis episode has elapsed, that intrinsically lack objectivity due to the patients’ depressive or manic condition. The treatment of bipolar disorder is based on pharmacological and psychotherapeutic techniques often characterized by low compliance from patients.

500


In this scenario, MONARCA‘s aim is to develop and validate a closed-loop, multiparametric approach to the treatment, management, and self-treatment of bipolar disorder disease and facilitate effective and efficient therapy that reduces costs and load of the health system while at the same time improving the quality of life of the patients. The main project objectives consist in: • Bipolar disorder events assessment based on objective, measurable data. • Continuous multi-parametric monitoring. • Warnings on “risky” behavior (prevention of crisis). • Increase of patients’ awareness through self- monitoring and timely personalized coaching. To reach these objectives, the MONARCA tools will be designed and tested for the assessment and prediction of episodes of bipolar disorder disease. The design and tests will be carried out with the patients and healthcare professionals involvement. The system will consist of 5 main components: • A sensor enabled mobile phone. • A wrist worn activity monitor. • A novel “sock integrated” physiological (GSR, pulse) sensor. • A stationary EEG system for periodic measurements. • A home gateway. Additionally, GPS location traces, physical motion information, and recognition of complex activities (nutrition habits, household activity, amount and quality of sleep) will be combined into a continuously updated behavioral profile that will be provided to doctors in a meaningful way to support treatment. The system will support both the patients through personalized interfaces, helping them to better manage their disease and the medical professionals to adjust the therapy.

3.3. Optimi Depression and Stress related disorders are the most common mental illnesses and the prevention of depression and suicide is one of the five central focus points in the European Pact for Mental Health and Well Being. Currently the main treatments for mental illness are pharmacological and evidence based Cognitive Behavioral Therapy (CBT). However little is being done to develop effective systems for prevention of the onset of the illnesses. OPTIMI (Online Predictive Tools for Intervention in Mental Illness) is based on the hypothesis that the central issue and starting point of longer-term mental illness depends on the individual’s capacity and ability to cope with stress. Many of us are lucky not to be subject to daily stressful conditions that ultimately will result in changes to our biology and personality. Some are fortunate be able to cope with enormous real pressure. Many however are in high-risk situations where despite their best efforts, they decompensate and develop a depressive disorder. With the aim of detecting the onset of a mental illness, OPTIMI: • will identify the occurrence of high stress in the individual on a daily basis. • will determine the ongoing effect of stress on the individual by studying the behaviour pattern over a longer period


•

501

will also make estimates of the base line changes in the person’s state of mind using measurements that closely link depression with cognitive, motor and verbal behavior.

OPTIMI will use wearable appliances based on EEG, EGG, Cortisol levels, Voice analysis, Physical Activity analysis and a self reporting Electronic Diary in order to identify stress coping behavior patterns. The smart identification sensors that capture stress, specific behaviors and test results, will be enhanced with a knowledge based rule system to interpret the data and provide a diagnostic tool for both pharmacological and CBT based preventative and intervening treatments. OPTIMI will augment two existing computerized CBT systems to use these tools in real time to optimize the treatment cycle. OPTIMI will conduct two phases of trials with volunteers at high -isk situations. The first phase being held in 3 countries (China, Switzerland, Spain) over 6 months will use the tools, develop and fine tune the algorithms against the gold standard of regular therapist interviews. The second phase in 2 countries (UK, Spain) will use the calibrated tools and a computerized CBT preventative treatment system to evaluate effectiveness in reducing the impact of stress to high risk people as well as the relapse after treatment for depression.

3.4. Psyche One of the areas of great demand for the need of continuous monitoring, patient participation and medical prediction is that of mood disorders, more specifically bipolar disorders. Due to the unpredictable and episodic nature of bipolar disorder, it is necessary to take the traditional standard procedures of mood assessment through the administration of rating scales and questionnaires and integrate this with tangible data found in emerging research on central and peripheral changes in brain function that may be associated to the clinical status and response to treatment throughout the course of bipolar disorder. In this scenario, PSYCHE project will develop a personal, cost-effective, multiparametric monitoring system with the aim to treat and predict depressive or manic episodes in patient diagnosed with bipolar disorder by combining wearable and portable devices, with appropriate platforms and services. PSYCHE project will develop a personal, cost-effective, multi-parametric monitoring system based on textile platforms and portable sensing devices for the long term and short term acquisition of data from selected class of patients affected by mood disorders. The project will develop novel portable devices for the monitoring of biochemical markers, voice analysis and a behavioral index correlated to patient state. Additionally, brain functional studies will be performed under specific experimental protocols in order to correlate central measures with the clinical assessment, and the parameters measured by Psyche platform. Specifically, will focus on the following objectives: • Integration of sensors for physiological and behavioral data into a monitoring system for patients affected by bipolar disorders. • Development of novel portable devices for the monitoring of biochemical markers, voice analysis and a behavioral index correlated to mental illness.

502


•

Implementation of an integrated system to collect data from bipolar patients. Bipolar patients in different states of the illness (mania or depression episodes, remission) will be considered.

4. Conclusions PHS is a relatively new concept, introduced in the 1990s, that place the individual citizen in the centre of the healthcare delivery process. PHS can bring significant benefits in terms of improved quality of care and cost reduction in patient management, especially through applications for remote patient monitoring and disease management. The paper introduced four recently funded projects – Interstress, Monarca, Optimi and Psyche – that aim at using PHS based on virtual reality, biosensors and/or mobile technologies to improve the treatment of bipolar disorders, depression and psychological stress. The expected end outcome of these projects are : • Increased mental health practitioners productivity (i.e. reduced patient unit cost through remote monitoring and self care). • Reduced in-patient costs (i.e. due to delay of the time between when a disease becomes complex and chronic and the end of life or to the elimination altogether of the development of pre-morbid conditions into a full-blown disease); • Decreased diagnostic and treatment costs as less visits will be needed as a result of both preventive monitoring and chronic disease management.

5. Acknowledgments This paper was supported by the FP7 European funded projects “Interstress Interreality in the management and treatment of stress-related disorders” and ”Optimi Online Predictive Tools for Intervention in Mental Illness”.

6. References [1]

[2]

[3] [4] [5]

I. Iakovidis, Consultation Workshop on Personal Health Systems, in: PHS 2010 consultation, Brussels, Belgium, 2010, p. online: http://ec.europa.eu/information_society/activities/health/docs/events/phs2010wkshp/phs2010consult_w orkshop_report.pdf. C. Codagnone, D5.1 Consolidated Roadmaps Report - FP7-IST-2007- 215291, in: Roadmapping Personal Health Systems: Scenarios and Research Themes for Framework Programme 7th and beyond, Consorzio per l’innovazione nella gestione delle imprese e della Pubblica Amministrazione (MIP), Milan, Italy, 2009. G. Loukianos, Objective 5.1: “Personal Health Systems”, in: ICT WP 2011-12 - Challenge 5, ICT for Health, DG Information Society & Media, European Commission, Bruxelles, Belgium, 2010. G. Riva, D. Algeri, F. Pallavicini, C. Repetto, A. Gorini, and A. Gaggioli, The use of advanced technologies in the treatment of psychological stress, J CyberTher Rehab 2 (2010), 169-171. G. Riva, Interreality: A New Paradigm for E-health, Stud Health Technol Inform 144 (2009), 3-7.


503

An Intelligent Virtual Human System for Providing Healthcare Information and Support Albert A. RIZZOa1, Belinda LANGEa, John G. BUCKWALTERa, Eric FORBELLa, Julia KIMa, Kenji SAGAEa, Josh WILLIAMSa, Barbara O. ROTHBAUMb, JoAnn DIFEDEc, Greg REGERd, Thomas PARSONSa and Patrick KENNYa a University of Southern California - Institute for Creative Technologies; bEmory University, cWeill Cornell Medical College; dMadigan Army Medical Center Army Abstract. Over the last 15 years, a virtual revolution has taken place in the use of Virtual Reality simulation technology for clinical purposes. Shifts in the social and scientific landscape have now set the stage for the next major movement in Clinical Virtual Reality with the “birth” of intelligent virtual humans. Seminal research and development has appeared in the creation of highly interactive, artificially intelligent and natural language capable virtual human agents that can engage real human users in a credible fashion. No longer at the level of a prop to add context or minimal faux interaction in a virtual world, virtual humans can be designed to perceive and act in a 3D virtual world, engage in spoken dialogues with real users and can be capable of exhibiting human-like emotional reactions. This paper will present an overview of the SimCoach project that aims to develop virtual human support agents to serve as online guides for promoting access to psychological healthcare information and for assisting military personnel and family members in breaking down barriers to initiating care. The SimCoach experience is being designed to attract and engage military Service Members, Veterans and their significant others who might not otherwise seek help with a live healthcare provider. It is expected that this experience will motivate users to take the first step – to empower themselves to seek advice and information regarding their healthcare and general personal welfare and encourage them to take the next step towards seeking more formal resources if needed. Keywords. SimCoach, Virtual Humans, Military Healthcare, Barriers to Care

Introduction Over the last 15 years, a virtual revolution has taken place in the use of simulation technology for clinical purposes. Technological advances in the areas of computation speed and power, graphics and image rendering, display systems, tracking, interface technology, haptic devices, authoring software and artificial intelligence have supported the creation of low-cost and usable PC-based Virtual Reality (VR) systems. At the same time, a determined and expanding cadre of researchers and clinicians have not only recognized the potential impact of VR technology, but have now generated a significant research literature that documents the many clinical targets where VR can add value over traditional assessment and intervention approaches (1-5). To do this, VR scientists have constructed virtual airplanes, skyscrapers, spiders, battlefields, ___________________________ 1 Albert Rizzo, University of Southern California, Institute for Creative Technologies, 12015 Waterfront Dr. Playa Vista, CA. 90064, [email protected]

504

A.A. Rizzo et al. / An Intelligent VH System for Providing Healthcare Information and Support

social settings, beaches, fantasy worlds and the mundane (but highly relevant) functional environments of the schoolroom, office, home, street and supermarket. And this state of affairs now stands to transform the vision of future clinical practice and research in the disciplines of psychology, medicine, neuroscience, physical and occupational therapy, and in the many allied health fields that address the therapeutic needs of those with clinical disorders. This convergence of the exponential advances in underlying VR enabling technologies with a growing body of clinical research and experience has fueled the evolution of the discipline of Clinical Virtual Reality. This paper presents the design vision for a Clinical VR project called SimCoach that aims to create intelligent virtual human agents to serve the role of online healthcare guides/coaches for military Service Members, Veterans and their significant others in an effort to break down barriers to care.

1. Virtual Humans in Clinical VR These shifts in the VR technological and scientific landscape have now set the stage for the next major movement in Clinical VR. With advances in the enabling technologies allowing for the design of ever more believable context-relevant “structural” VR environments (e.g. homes, classrooms, offices, markets, etc.), the next important challenge will involve populating these environments with Virtual Human (VH) representations that are capable of fostering believable interaction with real VR users. This is not to say that representations of human forms have not usefully appeared in Clinical VR scenarios. In fact, since the mid-1990’s, VR applications have routinely employed VHs to serve as stimulus elements to enhance the realism of a virtual world simply by their static presence. More recently, research and development has appeared in the creation of highly interactive, artificially intelligent and natural language capable virtual human agents. No longer at the level of a prop to add context or minimal faux interaction in a virtual world, these VH agents are designed to perceive and act in a 3D virtual world, engage in face-to-face spoken dialogues with real users (and other VHs) and in some cases, they are capable of exhibiting human-like emotional reactions. Previous classic work on virtual humans in the computer graphics community focused on perception and action in 3D worlds, but largely ignored dialogue and emotions. This has now changed. Intelligent VH agents can now be created that control computer generated bodies and can interact with users through speech and gesture in virtual environments (6). Advanced virtual humans can engage in rich conversations (7), recognize nonverbal cues (8), reason about social and emotional factors (9) and synthesize human communication and nonverbal expressions (10). Prototype-level embodied conversational characters have been around since the early 90’s (11) but significant advances have occurred more recently working systems used for training (12), intelligent kiosks (13) and virtual patients for clinical training (14). Both in appearance and behavior, VHs have now evolved to the point where they are usable tools for a variety of clinical and research applications.

2. Breaking Down Barriers to Care in Military Healthcare Research suggests that there is an urgent need to reduce the stigma of seeking mental health treatment in Service Members (SM) and Veteran populations. While US


505

military training methodology has better prepared soldiers for combat in recent years, such hesitancy to seek treatment for difficulties that emerge upon return from combat, especially by those who may need it most, suggests an area of military mental healthcare that is in need of attention. Moreover, the dissemination of healthcare information to military SMs, Veterans and their significant others is a persistent and growing challenge. Although medical information is increasingly available over the web, users can find the process of accessing it to be overwhelming, contradictory and impersonal. At the same time, the need for military-specific health information is growing at an astounding rate. In this regard, the reports over the last few years of a surge in U.S. Army suicide rates have again thrust the challenges of military mental health care into the public spotlight. With annual suicide rates steadily rising since 2004, the month of Jan. 2009 saw 24 suspected suicides, compared to five in Jan. of 2008, six in Jan. of 2007 and 10 in Jan. of 2006 (15). In spite of a Herculean effort on the part of the U.S. Department of Defense (DOD) to produce and disseminate behavioral health programs for military personnel and their families, the complexity of the issues involved continue to challenge the best efforts of military mental health care experts, administrators and providers. Since 2004, numerous blue ribbon panels of experts have attempted to assess the current DOD and Veterans Affairs (VA) healthcare delivery system and provide recommendations for improvement. For example, the American Psychological Association Presidential Task Force on Military Deployment Services for Youth, Families and Service Members (16) poignantly stated that they were, “…not able to find any evidence of a well-coordinated or well-disseminated approach to providing behavioral health care to service members and their families.” The APA report also went on to describe three primary barriers to military mental health treatment: availability, acceptability and accessibility. More specifically: Well-trained mental health specialists are not in adequate supply (availability), the military culture needs to be modified such that mental health services are more accepted and less stigmatized, And even if providers were available and seeking treatment was perceived as more acceptable, appropriate mental health services are often not readily accessible due to a variety of factors (e.g. long waiting lists, limited clinic hours, a poor referral process and geographical location).The overarching goal reported in this and other reports is to provide better awareness and access to existing care while concurrently reducing the complexity and stigma in seeking psychological help. In essence, new methods are needed to reduce such barriers to care.

3. SimCoach Design Approach While advances in technology has begun to show promise for the creation of new and effective clinical assessment and treatment approaches, from Virtual Reality to computerized prosthetics, improvements in the military health care dissemination/delivery system are required to take full advantage of these evolving treatment methodologies, as well as for promoting standard proven intervention options. In response to the clinical health care challenges that the conflicts in Iraq and Afghanistan have placed on the burgeoning population of service members and their families, the U.S. Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE) have recently funded our development of an intelligent, interactive, online Virtual Human (VH) healthcare guide program currently referred to as SimCoach. The SimCoach project that aims to address this need by developing

506


virtual human support agents to serve as online guides for promoting access to psychological healthcare information and for assisting military personnel and family members in breaking down barriers to initiating the healthcare process. The SimCoach experience is being designed to attract and engage military SMs, Veterans and their significant others who might not otherwise seek help. It aims to create an experience that will motivate users to take the first step – to empower themselves to seek information and advice with regard to their healthcare (e.g., psychological health, traumatic brain injury, addiction, etc.) and general personal welfare (i.e., other nonmedical stressors such as economic or transition issues) – and encourage them to take the next step towards seeking more traditional resources that are available, when the need is determined. Rather than being a traditional web portal, SimCoach will allow users to initiate and engage in a dialog about their healthcare concerns with an interactive VH. Generally, these intelligent graphical characters are being designed to use speech, gesture and emotion to introduce the capabilities of the system, solicit basic anonymous background information about the user’s history and clinical/psychosocial concerns, provide advice and support, direct the user to relevant online content and potentially facilitate the process of seeking appropriate care with a live clinical provider. An implicit motive of the SimCoach project is that of supporting users determined to be in need, to make the decision to take the first step toward initiating psychological or medical care with a live provider. It is not the goal of SimCoach to breakdown all of the barriers to care or to provide diagnostic or therapeutic services that are best delivered by a real clinical provider. Rather, SimCoach will foster comfort and confidence by promoting users’ efforts to understand their situations better, to explore available options and initiate treatment when appropriate. Coordinating this experience will be a VH SimCoach, selected by the user from a variety of archetypic character options (See Figures 1-3), who will answer direct questions and/or guide the user through a sequence of user-specific questions, exercises and assessments. This interaction between the VH and the user will provide the system with the information needed to guide them to the appropriate next step of engagement with the system or to initiate contact with a live provider.

Figures 1-3. SimCoach Archetypes – Female Aviator, Battle Buddy, Retired Sergeant Major

The SimCoach project is not conceived to deliver diagnosis or treatment or as a replacement for human providers and experts. Instead, SimCoach will aim to start the process of engaging the user by providing support and encouragement, increasing awareness of their situation and treatment options, and in assisting individuals, who may otherwise be initially uncomfortable talking to a “live” care provider, in their efforts to initiate care.


507

Users can flexibly interact with these VHs by typing text, clicking on character generated menu options and have some limited speech interaction during the initial phases of development. The feasibility of providing the option for full spoken natural language dialog interaction on the part of the user will be explored in the later stages of the project. Since this is designed to be a web-based system that will require no downloadable software, it is felt that voice recognition is not at a state where it could be reliably used at the current time. The options for SimCoach appearance, behavior and dialog is being designed to maximize user comfort and satisfaction, but also to facilitate fluid and truthful disclosure of medically relevant information. Based on the issues delineated in the initial interview, the user will be given access to a variety of general relevant information on psychology, neurology, rehabilitation, When relevant, users will also be directed to experts on specific areas such as stress, brain injury, marriage counseling, suicide, rehabilitation, reintegration and other relevant specialties the military healthcare system, and also to other SMs and Veterans by way of a variety of social networking tools (e.g., 2nd Life, Facebook, etc.). The user can progress through the system at their own pace over days or even weeks as they feel comfortable and the SimCoach will be capable of “remembering” the information acquired from previous visits and build on that information in similar fashion to that of a growing human relationship. The persistence of the SimCoach’s memory for previous sessions will require the user to sign into the system with a user name and password. However, that is optional for use of the system. Interspersed within the program will be the option to allow the user to perform some simple neurocognitive and psychological testing to inform the SimCoach’s creation of a model of the user to enhance the reliability and accuracy of the SimCoach output to the user, to support user selfawareness, and better guide the delivery of initial referral options. Users will also have the option to print out a summary of the computerized sessions to bring with them when seeking clinical care to enhance their comfort level, armed with knowledge, when dealing with the “real” human clinical care providers and experts. Software authoring tools are also being created that will allow other clinical professionals to create SimCoach “content” to enhance the likelihood that the program will evolve based on other care perspectives and emerging needs in the future. A fundamental challenge of the SimCoach project will be to better understand the diverse needs of the user base such that appropriate individual user experiences can be delivered to promote effective healthcare access. At the most basic level, there are immense differences in the needs of service members and their families. Further, there are likely large differences in the level of awareness that users will have of existing resources and in their own need/desire to engage such resources. Within the service member population there is a high likelihood that individual users will have had very diverse combat experiences, help-seeking histories and consequent impact on significant others. The net result of attempting to engage such a diverse user base is that the system will need to be able to employ a variety of general strategies and tactics to be relevant to each individual user. Focus groups and “Wizard of OZ” user studies are currently in progress in order to prepare the SimCoach interaction system for a wide range of potential dialog. In this regard, the SimCoach project is employing a variety of techniques to create the user experience. One relevant clinical model is the PLISSIT therapeutic framework (Permission, Limited Information, Specific Suggestions, and Intensive Therapy) (17), which provides an established model for encouraging help-seeking behaviors in persons who may feel stigma and insecurity regarding a clinical condition. In the

508


SimCoach project, the aim is to address the “PLISS” components, leaving the intensive therapy component to live professionals to which users in need of this level of care can be referred. Another source of knowledge is social work practice. Such models take a case management approach, serving both as an advocate and a guide. The SimCoach development team is also leveraging knowledge from the entertainment/gaming industry. While knowledge from this community is not typically applied towards healthcare, a primary aim by this community is in the explicit attraction and engagement of individuals’ attention. As we work to develop this web-based VH interactive system we are working closely with experts in all three of these models to achieve our goal of engaging and focusing this unique user base on the steps to initiate care as needed. Additionally, all interactions will be consistent with findings that suggest that interventions with individuals with PTSD and other psychosocial difficulties achieve the following: 1) promotion of perceptions of self-efficacy and control 2) encouragement of the acceptance of change; 3) encouragement of positive appraisals; and 4) an increase in the usage of adaptive coping strategies (18). These principles of intervention will be implicit in all of the interactions between the SimCoach and its users.

4. Conclusions The systematic use of artificially intelligent virtual humans in Clinical Virtual Reality applications is still clearly in its infancy. But the days of limited use of VH’s as simple props or static elements to add realism or context to a VR application are clearly in the past. In this paper we have presented our general approach to the design and development of the SimCoach VH project envisioned to serve as an online clinical healthcare guide or coach. This work is focused on breaking down barriers to care (stigma, unawareness, complexity, etc.) by providing military SMs, Veterans, and their significant others with confidential help in exploring and accessing healthcare content and for promoting the initiation of care with a live provider if needed. This work will also afford many research opportunities for investigating the functional and ethical issues involved in the process of creating and interacting with virtual humans in a clinical context. While the ethical challenges may be more intuitively appreciated, the functional technology challenges are also significant. However, although this project represents an early effort in this area, it is our view that the clinical aims selected can still be usefully addressed in spite of the current limits of the technology. As advances in computing power, graphics and animation, artificial intelligence, speech recognition, and natural language processing continue to develop at current rates, the creation of highly interactive, intelligent VHs for such clinical purposes is not only possible, but probable.


M.K. Holden, Virtual Environments for Motor Rehabilitation: Review, CyberPsychology and Behavior 8, 3 (2005), 187-211. T. Parsons & A.A. Rizzo, Affective Outcomes of Virtual Reality Exposure Therapy for Anxiety and Specific Phobias: A Meta-Analysis, Jour. of Behav. Therapy & Exper. Psychiatry 39 (2008), 250-261. M. Powers & P.M.G. Emmelkamp, Virtual reality exposure therapy for anxiety disorders: A metaanalysis, Journal of Anxiety Disorders 22 (2008), 561-569.

A.A. Rizzo et al. / An Intelligent VH System for Providing Healthcare Information and Support [4] [5] [6] [7] [8]

[9] [10] [11]

[12] [13] [14] [15]

[16]

[17] [18]

509

G. Riva, Virtual Reality in Psychotherapy: Review, CyberPsychology and Behavior 8, 3 (2005), 220230. F.D. Rose, B.M. Brooks & A.A. Rizzo, Virtual Reality in Brain Damage Rehabilitation: Review, CyberPsychology and Behavior 8, 3 (2005), 241-262. J. Gratch, et al., Creating Interactive Virtual Humans: Some Assembly Required, IEEE Intelligent Systems July/August (2002), 54-61. D. Traum, J. Gratch, et al., Multi-party, Multi-issue, Multi-strategy Negotiation for Multi-modal Virtual Agents, 8th International Conference on Intelligent Virtual Agents, Springer, Tokyo, Japan, 2008. L.P. Morency, I. de Kok et al., Context-based Recognition during Human Interactions: Automatic Feature Selection and Encoding Dictionary, 10th International Conference on Multimodal Interfaces, IEEE, Chania, Greece, 2008. Gratch & S. Marsella, A domain independent framework for modeling emotion, Journal of Cognitive Systems Research 5, 4 (2004), 269-306. M. Thiebaux, A. Marshall, et al., SmartBody: Behavior Realization for Embodied Conversational Agents. Intern. Conf. on Autonomous Agents and Multi-Agent Systems Porto, Portugal, 2008. T. Bickmore & J. Cassell, Social Dialogue with Embodied Conversational Agents. In Advances in Natural, Multimodal Dialogue Systems (J van Kuppevelt, L Dybkjaer and N Bernsen, Eds.), Kluwer Academic, New York, 2005. D. Evans, M. Hern, M. Uhlemann & A. Lvey, Essential Interviewing: A Programmed Approach to Effective Communication, (3rd Ed), Brooks/Cole Publishing Company, 1989. L. McCauley & S. D’Mello, A Speech Enabled Intelligent Kiosk, In IVA 2006 (J. Gratch et al. Eds), Springer-Verlag, Berlin, Germany, 132-144, 2006. P. Kenny, T. Parsons, G. Reger, C. Pataki & A. Rizzo, Virtual Patients for Future Leaders, Proceedings of the 2008 IITSEC, Orlando, FL., 2008. P. Jelinek & K. Hefling, AP Report: Army suicides at record high, passing civilians, Downloaded on 1/29/2009 at: http://www.google.com/hostednews/ap/article/ALeqM5jrRijfpxg8ZdUbcDpGbmnEpYPH9wD9616BB 80 American Psychological Association Presidential Task Force on Military Deployment Services for Youth, Families and Service Members. The Psychological Needs of U.S. Military Service Members and Their Families: A Preliminary Report. Retrieved 04/18/2007, from: http://www.apa.org/releases/MilitaryDeploymentTaskForceReport.pdf J. Annon, Behavioral Treatment of Sexual Problems, Harper-Collins. NY, NY, 1976. J.M. Whealin, J.I. Ruzek & S. Southwich, Cognitive-behavioral theory and preparations for professionals at risk for trauma exposure, Trauma Violence Abuse, 9, 100-113, 2008.

510


Virtual Reality Applications for Addressing the Needs of those Aging with Disability Albert RIZZOa1, Phil REQUEJOb, Carolee J. WINSTEINc, Belinda LANGEa, Gisele RAGUSAd, Alma MERIANSe, James PATTONf,g, Pat BANERJEEg & Mindy AISENb a University of Southern California - Institute for Creative Technologies; bRancho Los Amigos National Rehabilitation Hospital; cUniversity of Southern California – Division of Biokinesiology and Physical Therapy; dUniversity of Southern California – Rossier School of Education, eUniversity of Medicine and Dentistry of New Jersey, f Northwestern U., gUniversity of Illinois at Chicago Abstract. As persons with disabilities age, progressive declines in health and medical status can challenge the adaptive resources required to maintain functional independence and quality of life [1]. These challenges are further compounded by economic factors, medication side effects, loss of a spouse or caregiver, and psychosocial disorders [1-2]. With the gradual loss of functional independence and increased reliance on others for transportation, access to general medical and rehabilitation care can be jeopardized [2]. The combination of these factors when seen in the context of the average increase in lifespan in industrialized societies has lead to a growing crisis that is truly global in proportion. While research indicates that functional motor capacity can be improved, maintained, or recovered via consistent participation in a motor exercise and rehabilitation regimen [3], independent adherence to such preventative and/or rehabilitative programming outside the clinic setting is notoriously low [1]. This state of affairs has produced a compelling and ethical motivation to address the needs of individuals who are aging with disabilities by promoting home-based access to low-cost, interactive virtual reality (VR) systems designed to engage and motivate individuals to participate with “game”-driven physical activities and rehabilitation programming. The creation of such systems could serve to enhance, maintain and rehabilitate the sensorimotor processes that are needed to maximize independence and quality of life. This is the theme of the research to be presented at this MMVR workshop. Keywords. Virtual Reality, Aging, Disability, Technology

Introduction This MMVR workshop brings together researchers, users and industry partners to present and discuss the issues relevant to advancing the science and practice for using VR applications with those aging with and into disability. The session will commence with a user panel that will discuss the activity limitations experienced by aging adults with disabilities, followed by researchers presenting state of the art work using VR technologies for maintaining and enhancing sensorimotor and cognitive functions across the lifespan. Access to such systems by users will be the theme of the closing panel that will be made up of industry leaders, users and researchers. The objectives of this session include: 1. Promote awareness of the unique challenges and needs of adults with disability for maintaining and enhancing functional independence at this point in the lifespan; 2. Educate the public and professionals about the centers supported by the ___________________________ 1 Albert Rizzo, University of Southern California, Institute for Creative Technologies, 12015 Waterfront Dr. Playa Vista, CA. 90064, [email protected]

A. Rizzo et al. / VR Applications for Addressing the Needs of Those Aging with Disability

511

National Institute of Disability and Rehabilitation Research (NIDRR) that addresses these challenges using VR and associated technologies; 3. Clarify the issues involved in promoting equal access to VR and game-based applications for older adults with disability.

1. Virtual Reality Rehabilitation Over the last 15 years, a virtual revolution has taken place in the use of Virtual Reality (VR) simulation technology for clinical purposes. Technological advances in the areas of computation speed and power, graphics and image rendering, display systems, body tracking, interface technology, haptic devices, authoring software and artificial intelligence have supported the creation of low-cost and usable VR systems capable of running on a commodity level personal computer. At the same time, a determined and expanding cadre of researchers and clinicians have not only recognized the potential impact of VR technology, but have now generated a significant research literature that documents the many clinical targets where VR can add value over traditional assessment and intervention approaches [4-8]. To do this, scientists have constructed virtual airplanes, skyscrapers, spiders, battlefields, social events populated with virtual humans, fantasy worlds and the mundane (but highly relevant) functional environments of the schoolroom, office, home, street and supermarket. This state of affairs now stands to transform the vision of future clinical practice and research in the disciplines of psychology, medicine, neuroscience, physical and occupational therapy, and in the many allied health fields that address the therapeutic needs of children and adults with disabilities. This convergence of the exponential advances in underlying VR enabling technologies with a growing body of clinical research and experience has fueled the evolution of the discipline of Clinical Virtual Reality. This is expected to have significant impact on promoting access to VR technology for addressing the needs of persons aging with disabilities.

2. Theoretical Model for VR Applications for Successful Aging with Disabilities In 2008, the NIDRR made a 5-year award to the University of Southern California (USC) and Rancho Los Amigos National Rehabilitation Center (RLANRC) to establish a unique Rehabilitation Engineering Research Center (RERC)—“Optimizing Participation through Technology” (OPTT). The overall purpose of OPTT RERC is focused on those aging with and into disability. Over the first two years, we built a strong interdisciplinary infrastructure engaged in a set of research and development activities at the nexus of biomedical engineering and technology, sensorimotor systems rehabilitation, and gerontology and aging (see http://www.isi.edu/research/rerc/). Central to our research and development activities is the creation and delivery of VR simulation technologies for enhancing targeted skills (e.g., dexterity, balance) and exercise in those who are aging into and with disability. One of the biggest challenges we face with the research and development of VR applications is in maintaining a proper balance between usability, inherent flexibility to allow adaptation to the user’s needs, and cost, all while assuring the most efficient and appropriate means toward important rehabilitation goals and one that is compatible with an ever-changing technology. There is no doubt that many of the so called off the

512


shelf ‘rehab’ games are enjoyable, engaging and can even foster social interaction among family and friends—all important for supporting healthy minds and bodies, but it is less clear (and one of our major concerns) if important rehabilitation goals that are tailored to the special needs of those aging with a disability are being achieved. To this end, we have attempted to create a theoretically defensible, evidence-based, conceptual model as a means to guide the development of VR simulation technologies for rehabilitation in the context of OPTT-RERC. Our model contains three overlapping elements: Skill Acquisition, Capacity Building, and Motivational Enhancements. VR game-based rehabilitation provides the glue for achieving the aims of the model. A pathway from impairment reduction (i.e. physiological loss) to functional capability (e.g., instrumental activities of daily living, self-care, mobility) to more general function in real-world contexts (e.g., independent living, social participation) is more often implicit but less frequently operationalized in VR games-based therapeutic intervention protocols. Task-specific practice is considered to be the most important element of any behavioral training program, particularly when improved functional skills are sought (e.g., cognitive and physical). In fact, the effects of practice are often underestimated and all too often, programs fail to be effective because either ample practice time was not prescribed or compliance was poor. Recently, our work has shown that in addition to time-on-task practice, the practice structure (e.g., variable or constant) is important for optimizing consolidation and motor learning [9]. The scientific rationale and evidence for impairment mitigation (capacity) comes from a growing body of work showing the importance of fundamental impairments including strength and control for restoration of function (e.g., individuals post-stroke, elderly atrisk for falls) [10]. Similarly, the scientific rationale and evidence for motivational enhancements (intrinsic drive) as well as the pursuit of meaningful goals for sustainable behavioral change (e.g., cognitive behavioral intervention such as ‘Matter of Balance’, comes from a growing body of work showing the importance of selfregulation, self-management, and self-efficacy for behavioral change that supports beneficial outcomes [11-14]. In most cases, the motivational enhancements strengthen self-confidence and support participant control or autonomy (intrinsic motivation). As well, providing choice in the context of effective intervention programs engages the learner and supports adherence. The active ingredients of an effective task-oriented VR game likely consists of interactions that are: 1) Challenging enough to require new learning and active attentional engagement to solve a motor problem; 2) Progressive and optimally adjustable, such that over practice, the task demand is optimally adapted to the user’s capability and the environmental context. Extending the environment outside the laboratory or clinic to the home is an important aspect of an optimal consumer-centered program. 3) Interesting enough to promote active participation to engage a ‘particular type of repetition’ that Bernstein referred to as ‘problem-solving’. For more details, we elaborate and provide examples of intervention programs based on this conceptual model in the context of stroke rehabilitation in two recent publications [15-16]. Once the particular task or set of tasks to-be-trained has been chosen, the VR simulation game can be embedded into the fully-defined, task-oriented training program. VR simulation technology affords certain key design features that map nicely onto the active ingredients for an effective program. These include: 1) Focus on a specific skill and involve data-based and task-specific training (skill/practice); 2) Have adjustable difficulty levels from something simple for the user to accomplish, to a level representing normal or skilled performance (capacity building); 3) Be quantifiable in


513

order to assess progress (assessment, motivation); 4) Be administered repetitively and hierarchically to allow enough practice with the right amount of challenge (motivation, skill acquisition/practice); 5) Provide the user with feedback as to the outcome of performance (builds confidence); 6) Have some relevance to real world function (meaningful, skill/task-based, motivating); 7) Motivate and engage the user (enhances compliance). With this brief overview of our theoretical model, the following sections summarize the rationale and results from NIDRR-supported VR research and development efforts targeting three core rehabilitation domains relevant for persons aging with or into disability: upper extremity sensorimotor function, cognitive processing, and a VR software package for designing customized exercise applications.

3. VR Simulations for Recovery of Upper Extremity Function Sensorimotor impairments and participation restrictions remain a pervasive problem for patients post stroke, with recovery of upper extremity function particularly recalcitrant to intervention. 80% to 95% of persons demonstrate residual upper extremity impairments lasting beyond six months after their strokes (17). One of the issues that may contribute to less than satisfactory outcomes for the upper extremity is the complexity of sensory processing and motor output involved in normal hand function. There is a vital need to develop rehabilitative training strategies that will improve functional abilities and real-world use of the arm and hand in order to increase independence [18-19]. To address this need, we have developed an exercise system that integrates robotic-assisted arm training with complex VR gaming simulations [20]. We are using this system in several innovative ways. First, the system allows us to utilize current neurophysiological findings regarding the importance of repetitive, frequent and intensive practice for skill development and motor recovery to train the hemiparetic upper extremity of people post stroke [21-22]. After a two-week period of VR training, participants were able to more effectively control the upper limb during reaching and hand interaction with a target as demonstrated by improved proximal stability, smoothness and efficiency of the movement path. This improved control was in concert with improvement in the distal kinematic measures of fractionation (ability to move fingers one at a time) and improved speed. These changes in robotic measures were notably accompanied by robust changes in the clinical outcome measures. Because of the systematized, objective nature of this system, it allows us to test hypotheses regarding the most efficacious therapeutic interventions. It is controversial whether training the upper extremity as an integrated unit leads to better outcomes than training the proximal and distal components separately. During recovery from a lesion, the hand and arm are thought to compete with each other for neural territory. Therefore, training them together may actually have deleterious effects on the neuroplasticity and functional recovery of the hand. However, neural control mechanisms of arm transport and hand-object interaction are interdependent. Therefore, complex multi-segmental motor training is thought to be more beneficial for skill retention. We are investigating these competing theories to determine if and how competition among body parts for neural representations stifles functional gains from different types of training regimens. Lastly, we are also exploring how this promising therapeutic strategy may actually change neural connections in the brain as a patient’s motor functions improve. Animal and human research suggests that functional recovery is dependent on neural reorganization. We have developed an innovative MRI-compatible VR system that

514


tracks bilateral hand movement and uses these measurements to drive motion of virtual hand models during an fMRI experiment. Our preliminary data suggest that, indeed, robot-assisted training in VR may be beneficial for functional recovery after chronic stroke. Further, our data suggest that this functional recovery may be attributed to increased functional connectivity in bilateral sensorimotor cortex.

4. VR Simulation for Recovery of Cognitive Function A key aspect of rehabilitation recovery is deliberate, repetitive practice that resembles functional activity in some way. This can involve tool use, and can be accomplished via interactive VEs that incorporate robotics that can render haptic feedback. Our recent work has evaluated several clinically promising uses of these haptic/graphic environments (HGE) in stroke recovery, which often involves exploiting the features of HGE by distorting the visual and mechanical fields in which users operate. Our MMVR talk will present this work as an extension of our neurorehabilitation research [23-24] with inpatients having moderate-severe impairments following a TBI, in which a key deficit is attention. Our approach is believed to be distinctive in that we use the HGE in an extremely minimal environment -- a cursor and target in the dark with no other distractions. This concept emerged from our observations that patients in the earliest stages of recovery exhibit severe impairment of both focused and sustained attention. It was hypothesized that since VR allows for complete control of the task environment and difficulty level, it might lead to short-term gains that in turn might lead to longer-term benefits in these users. Our initial study provided an assessment of the tolerance of a VR intervention for attention remediation in persons with severe TBI [25]. A small sample of patients with severe TBI in the early stages of recovery received acute inpatient rehabilitation along with a minimalistic interactive VR reaching task. Six TBI patients and three healthy controls were tested while reaching for successive visual targets. Initial results showed the system to be well-tolerated and engaging, and users focused on the task without being distracted by environmental stimuli while showing gains in proficiency (increased target acquisitions). Encouraged by this preliminary work, our next study tested a more comprehensive 2-day treatment protocol to evaluate how haptic cues might be beneficial [26]. Users visited the laboratory for two successive days, and on each day they executed 6 blocks of training that included three haptic feedback conditions: 1) no haptic forces, 2) resistive haptic forces (giving subjects a "breakthrough" sensation as they acquired the target) and 3) facilitative forces (giving subjects a 250 ms “nudge” toward the target whenever a 1 second period of near-zero speed was detected). We hypothesized that haptic feedback would refocus the patient's attention as well as increase time-on-task for subsequent movements. Overall, 19 of 22 patients were able to tolerate the task and target acquisition time on a 3D cancellation task showed improved performance across trials. The haptic nudge condition resulted in superior performance compared to the breakthrough condition. Subjects with posttraumatic amnesia demonstrated improved performance across trials, with carryover between day 1 and day 2 indicating improved procedural memory despite the fact that subjects lacked the capacity for declarative learning. We are now developing a prolonged 2-week clinical intervention while moving to transfer this technology to a viable system on a smaller and more affordable scale to promote better patient access. This study will include incremental task difficulty adjustment features so that as


515

subjects improve, the task challenges will engage patients in both the earliest stages of recovery and in those whose progress requires more demanding exercises.

5. VR Simulation Architecture for Remote Exercise The application of VR in rehabilitation engineering can be broadly categorized into two main areas: motor rehabilitation and cardiovascular exercise. Fortunately, both motor rehabilitation and cardiovascular exercise share almost the same technologies and devices. The primary difference being that cardiovascular exercise requires a larger range of movement, higher frequency and larger intensity. The challenge is to harness the technology optimally for efficient utilization for these differing aims. To address this challenge, we have created a software architecture, known as REGAL (Remote Exercise and Game Architecture Language) to design virtual exercise environments for people with lower body disabilities. The software also facilitates the development of new virtual exercises by other developers who are interested in creating fitness and rehabilitation applications. New exercises can be uploaded to the architecture and be played by users similar to the pre-loaded exercises we developed, and the REGAL architecture also supports 3D PC games developed using other SDKs and game engines. The first version of the architecture and a “throwing” demo received feedback via questionnaire from 26 participants [27]. Users reported that they were easily able to experience a 3D perspective in the virtual environment (VE); were satisfied with the consistent and responsive natural VE interaction (but expected more); and users with lower body disabilities showed higher satisfaction with all aspects of the VE than users without disabilities. In the current version of the architecture, we have designed two rowing demonstrations: the first is based on PhysX and Coin3D with a very simple 3D scene, while the second is based on a FarCry demo (http://farcry.us.ubi.com/) to show how to reuse currently available VR games [28]. In the FarCry demonstration, the acceleration data from a Wii remote was used to successfully drive the navigation and avatar movement. We are evolving the software such that users can define their own gestures for particular exercise equipment and the architecture will record user-specific interaction gestures that will be recognized later in the application. Such flexible options built into the REGAL software are expected to promote individualization of the exercise program based on the needs and equipment available to the user.

6. Conclusions In this workshop session, we will present the challenges faced in the creation and delivery of VR simulation technologies for addressing barriers faced by individuals aging with and into disability. The research and development briefly presented here illustrates the use of a range of interaction devices, programming engines, and emerging approaches for addressing the sensorimotor and cognitive challenges that adults with disabilities face throughout the lifespan. Such NIDRR-funded research aims to support the theory-informed development of a range of VR applications focused on improving upper/lower extremity functions, remediating cognitive impairments, and for providing technology to promote access to home-based exercise for persons aging with and into disability. These novel VR exercise and rehabilitation strategies are now providing evidence that indicate improvements in function when used by these groups.

516


References [1] A.W. Heinemann, State of the science of postacute rehabilitation: setting a research agenda and developing an evidence base for practice and public policy. Rehabil Nurs 33,2 (2008), 82-7. [2] C.I. Vincent, I. Deaudelin, L. Robichaud, J. Rousseau, C. Viscogliosi, L.R. Talbot, & J. Desrosiers, Rehabilitation needs for older adults with stroke living at home: perceptions of four populations. BMC Geriatric 7 (2007), 20. [3] R.T. Galvin, T. Cusack & E. Stokes, A randomised controlled trial evaluating family mediated exercise (FAME) therapy following stroke." BMC Neurol 8 (2008), 22. [4] M.K. Holden, Virtual Environments for Motor Rehabilitation: Review, Cyberpsy & Behav. 8,3 (2005), 187-211. [5] T. Parsons & A.A. Rizzo, Affective Outcomes of Virtual Reality Exposure Therapy for Anxiety and Specific Phobias: A Meta-Analysis, Jour. of Behav. Therapy & Exper. Psychiatry 39 (2008), 250-261. [6] G. Riva, Virtual Reality in Psychotherapy: Review, CyberPsy. and Behavior 8, 3 (2005), 220-230. [7] F.D. Rose, B.M. Brooks & A.A. Rizzo, Virtual Reality in Brain Damage Rehabilitation: Review, CyberPsychology and Behavior 8, 3 (2005), 241-262. [8] Rizzo AA, Buckwalter JG, van der Zaag C. Virtual environment applications for neuropsychological assessment and rehabilitation. In Stanney, K, (Ed.), Handbook of Virtual Environments. New York, NY: L.A. Earlbaum; 2002, 1027-64. [9] Kantak SS, Sullivan KJ, Fisher BE, Knowlton BJ, Winstein CJ. Neural substrates of motor memory consolidation depend on practice structure. Nat Neurosci. 2010 Aug 13(8):923-5. [10] Liu C, Latham N. Progressive resistance strength training for improving physical function in older adults. Cochrane Database of Systematic Reviews 2009(3). [11] Bandura A. Health promotion by social cognitive means. Health Educ Behav. 2004 Apr;31(2):143-64. [12] Bodenheimer T, Lorig K, Holman H, Grumbach K. Patient self-management of chronic disease in primary care. JAMA. 2002 Nov 20;288(19):2469-75. [13] Hart T, Evans J. Self-regulation and goal theories in brain injury rehabilitation. J Head Trauma Rehabil. 2006 Mar-Apr;21(2):142-55. [14] Siegert RJ, Taylor WJ. Theoretical aspects of goal-setting and motivation in rehabilitation. Disability and Rehabilitation. 2004 Jan 7;26(1):1-8. [15] Winstein C, Wolf S. Task-oriented training to promote upper extremity recovery. In: Stein J, RL H, Macko R, Winstein C, Zorowitz R, eds. Stroke Recovery & Rehabilitation. New York: Demos Medical 2008: 267-90. [16] Wolf S, Winstein C. Intensive physical therapeutic approaches to stroke recovery. In: Cramer S, Nudo R, eds. Brain Repair After Stroke: Cambridge University Press 2010. [17] Kwakkel G, Kollen BJ, van der Grond J, Prevo AJ. Probability of regaining dexterity in the flaccid upper limb: impact of severity of paresis and time since onset in acute stroke. Stroke.2003;34(9):2181-6. [18] Krebs HI, Volpe BT, Ferraro M, Fasoli S, Palazzolo J, Rohrer B, et al. Robot-aided neurorehabilitation: from evidence-based to science-based rehabilitation. Topics in Stroke Rehabil. 2002;8(4):54-70. [19] Kahn LE, Lum PS, Rymer WZ, Reinkensmeyer DJ. Robot-assisted movement training for the strokeimpaired arm: Does it matter what the robot does? J Rehabil Res Dev. 2006;43(5):619-30. [20] Merians AS, Poizner HP, Boian R, Burdea G, Adamovich SV, Sensorimotor training in a virtual reality environment: does it improve functional recovery post-stroke? Neural Rehabil Neur Repair,2006:20 (2). [21] Jenkins WM, Merzenich MM. Reorganization of neocortical representations after brain injury: a neurophysiological model of the bases of recovery from stroke. Prog Brain Res 1987;71:249-66. [22] Nudo RJ, Wise BM, SiFuentes F, Milliken GW. Neural substrates for the effects of rehabilitative training motor recovery after ischemic infarct. Science 1996; 272:1791-4. [23] Patton JL, Dawe G, Scharver C, Muss-Ivaldi FA, Kenyon R. 2006. Robotics and virtual reality: A perfect marriage for motor control research and rehabilitation. Assistive Technology 18:181-95. [24] Rozario S, Housman S, Kovic M, Kenyon R, Patton J. 2009. Therapist-mediated post-stroke rehabilitation using haptic/graphic error augmentation. IEEE Engin.in Med.& Bio. Conf. Minn., MN [25] Dvorkin A, Zollman F, Beck K, Larson E, Patton J. 2009. A Virtual Environment-Based Paradigm for Improving Attention in TBI. IEEE Intern. Conf. on Rehabilitation Robotics (ICORR). Kyoto, Japan. [26] Dvorkin A, Ramaiya M, Zollman F, Larson E, Pacini S, et al. 2010. A virtual environment-based paradigm for improving attention in severe TBI. Cog. Neurosci. Soc. meeting. Montreal, Canada. [27] Zhang, S., Banerjee, P. P., Luciano, C.: Virtual Exercise Environment for Promoting Active Lifestyle for people with Lower Body Disabilities. Proc. IEEE International Conference on Networking, Sensing and Control, Chicago, 2010, pp. 80-84. [28] Banerjee, P. P., Zhang, S., Luciano, C. and Rizzi, S.: Remote Exercise and Game Architecture Language (REGAL), Proc. Rectech 2nd State of the Science Conference, Arlington, VA, 2010. 53-56.


517

The Validation of an Instrumented Simulator for the Assessment of Performance and Outcome of Knot Tying Skill: A Pilot Study David ROJASa,c, Sayra CRISTANCHOb,1, Claudia RUEDAa, Lawrence GRIERSONc Alex MONCLOUa, and Adam DUBROWSKIc,d a Faculty of Electronics Engineering, Pontificia Bolivariana University, Bucaramanga, Colombia. b Department of Surgery and Centre for Education Research & Innovation, Schulich School of Medicine & Dentistry, University of Western Ontario. London, Canada. c The Hospital for Sick Children Learning Institute. Toronto, Ontario Canada. d Department of Pediatrics, Faculty of Medicine, and The Wilson Centre, University of Toronto. Toronto, Canada.

Abstract. The construct validity of a surgical bench-top simulator with built-in computer acquired assessments was examined. It features two parallel elastic tubes instrumented with flexion sensors that simulate the walls of a wound. Participants from three groups (9 novices, 7 intermediates, 9 experts) performed 10 two-handed, double square knots. The peak tensions at the initiation of the first knot, the completion of the first knot and the completion of the second knot, as well as measures of movement economy indicated technical performance. Product quality was indicated by knot stability defined as the amount of slippage of the knot under the tension. There were significant differences between experts and novices for peak tension on first knot (p=.03), movement economy (p=.02), and knot stability (p=.002). The results support the construct validity of these objective measures. Keywords. Simulation, computer-based assessment, technical skills

Introduction The objective assessment of surgical technical performances and the resulting products are critical for ensuring that standards of practice are met as well as for augmenting the learning process. Methods of objective assessment have been classified broadly as either expert-based [1,3-4] or computer-based [2,7-11]. The expert-based methods are criticized for their objectivity and feasibility [1]. Additionally, because these methods rely on expert presence, they are expensive for formative evaluations [6]. Computerbased assessments rely on the acquisition of information about technical performance such as specific motor processes [10], movement patterns and related information about hand motion efficiency [7], as well as outcomes [2,13]. However, assessments of 1

Corresponding Author.

518

D. Rojas et al. / The Validation of an Instrumented Simulator

the processes leading to the skillful execution of technical surgical skills and the quality of their products are typically performed independently. In this paper we present an integrated alternative that allows for the computerbased assessment to measure both the process and the quality of final products associated with knot-tying skills. Specifically, we examined the construct validity of the variables associated with a newly developed computer-based surgical skills assessment device. We hypothesized that the analyses of the variables derived from the device can discern between novice, intermediate and experts. Thus the construct validity of this device will be supported. Of secondary interest to the study is to establish baseline characteristics of the behaviours of the three groups of performers in order to conduct further, more powerful validation studies.

1. Methods 1.1. Participants The novices (n=9) were undergraduate students from the Universidad Pontificia Bolivariana (Bucaramanga, Colombia). The intermediates (n=7) were senior medical students from Hospital Universitario Santander (Bucaramanga, Colombia). The experts (n=9) were senior surgical residents from Hospital Universitario Santander and practicing surgeons from Toronto General Hospital (Toronto, Ontario, Canada), Sunnybrook Hospital (Toronto, Ontario, Canada) and The Hospital for Sick Children (Toronto, Ontario, Canada). All volunteers provided informed consent in accordance with the guidelines set out by The University of Toronto Research Ethics Board and the 1964 Declaration of Helsinki. 1.2. Apparatus and Procedure A custom bench-top suturing simulator was developed (Fig. 1). It was instrumented with sensors to measure the movement parameters related to motor processes, overall movement patterns, economy of hand motions and the quality of the final products. Specifically, two parallel polyethylene tubes were mounted with two flexion sensors (Flex Sensor, SpectraSymbol, Salt Lake City, Utah), which measured the variation in the distance between the tubes. The participants were asked to initiate the knot tying skill with sutures (SOLFSILK 2.0, TYCO Healthcare, Pointe-Claire, Canada) in a predetermined position, and use the two-handed double square technique to approximate the two tubes as close to each other as possible. All sensor data were transformed and recorded as Voltages (V) and processed by computer (HP Pavilion dv2550, Intel Pentium 4, 2.8 GHz) via LabView software (8.5, Texas, EEUU). All data were filtered through an 8Hz low-pass Butterworth filter (Matlab, R2007b, California, EEUU) prior to analysis.


519

Figure 1. Experimental apparatus and steps in the technique. The first pull, defined as the moment when the initial tension on the apparatus was applied to perform the first knot (a), the first knot (b) and the last knot (c), defined as the peak tensions during the completion of the first and last knots.

1.3. Variables of Interest We examined three distinct knot-tying phases: the first pull (the moment when the initial tension on the apparatus was applied to perform the first knot), the first knot (the peak tension during the completion of the first knot), and the final knot (the peak tension during the completion of the second knot). [10,11] Movement economy was quantified by measuring the number of minor peaks in the signal using a custom algorithm. A minor peak in the signal was considered a variation in the current position of the tube of more than its own diameter (5 mm) with respect to its immediately previous position. These variables are referred to as measures of technical performance. The distance between the tubes at two-seconds following the knot’s completion was used to measure the quality of the final product (i.e., knot stability). This product quality measure is related to the amount of slippage of the knot under tension imposed by the natural elastic properties of the tubes. 1.4. Analysis Each dependent measure was subjected to a one-way analysis of variance with group experience as the only factor (Expert, Intermediate, Novice). All effects significant at p < .05 were further analyzed using Tukey’s Honestly Significant Difference post hoc methodology.

2. Results 2.1. The First Pull The peak tension on the initial pulling of the sutures revealed no significant group effect (grand mean = 0.93, standard deviation = 0.07 Volts (V)). The analysis of time to the initial pulling of the sutures did not reveal any significant effect (grand mean = 0.41, standard deviation = 0.04 seconds (s)).

520


2.2. The First Knot The peak tension of the first knot revealed a significant main effect for group, F (2, 22) = 3.78, p = .038. Post hoc analysis indicated that the first knots of experts were tighter than those of novices. The intermediate group’s knots were centered between and not significantly different than other groups’ (Figure 2a). The analysis of the time to the first knot yielded no significant group effects (grand mean = 1.03, standard deviation = 0.13 s).

Figure 2. The means and standard deviations for the number of dependent variable collected from the instrumented simulator plotted as a function of experience (Expert, Intermediate, Novice), a) The mean of peak tension (expressed in Voltages (V)) of the first knot, b) the mean movement economy measure (expressed as a number of peaks performed), c) the mean of peak tension (expressed in V) of the knot stability.

2.3. The Final Knot The peak tension of the sutures at the final knots revealed no significant differences between the groups (grand mean = 2.20, standard deviation = 0.12 V); however, the experts’ knots trended towards being conventionally tighter than the novices’ (F (2, 22) = 3.17, p = .063). The analysis of time to the final knot did not reveal any significant differences (grand mean = 2.87, standard deviation = 0.55 s).


521

2.4. Movement Economy The movement economy revealed a significant group effect, F (2, 22) = 8.15, p = .002. Post hoc analysis of this effect indicated that the intermediate group moved the tubes more while performing the knot tying skill than the novice or expert groups (Figure 2b). 2.5. Knot Stability (i.e., slippage of the knot under tension imposed by the natural elastic properties of the tube) The knots stability revealed a significant group main effect, F (2, 22) = 4.91, p = .017 Post hoc comparisons indicated that the experts’ knots remained closer together than the novices’ (Figure 2c). The intermediate’s knots were centered between and were not significantly different than the other two groups.

3. Discussion The results offer preliminary evidence that supports construct validity of the device insofar that it was capable of differentiating the technical performance and the quality of the final product of performers of varying levels of experience. The unique feature of this device is that it captures data about the technical performance as well as the quality of the final product or outcomes at the same time. Specifically, we have demonstrated that experts apply lower tensile forces than the novices during placement of the surgical knots. Furthermore, that the intermediate trainees used the same amount of tension as the experts on the first knot and slightly, yet not significantly less, in the second knot may indicate that these parameters are acquired and optimized quickly in the learning process. An interesting finding was that the immediate group performed the skills with more movements than the novices and experts. One possible explanation for this finding is that the intermediates perform the skill with certain comfort and explore variations in practice in order to learn, while the novices perform more consistently because the current technique is the only one they know. On the contrary the experts perform consistently as they have automated the technique [15]. The results are also in agreement with previous reports investigating the validity of methods for monitoring technical performances [1,7-9,12]. However, our data reveal low sensitivity to differentiating levels of trainees. This is in partial agreement with previous findings reported by Datta and colleagues [7]. Finally, the measures of the final outcomes in this study also support construct validity. That is, the knot stability measure differed significantly between the experts and the novice trainees. This is in line with previous reports [13,14]. Poorly constructed knots are unstable and slip under tension exposing the tissues, and potentially leading to increased rates of infections. However, here, the knot stability measures show low sensitivity where the quality of the knots produced by the intermediate group of trainees was not statistically different from those of the experts and the novices. This low sensitivity may be a shortcoming of the sensors.

522


3.1. Implications for Simulation-Based Teaching The ability to quantify the technical performance and the quality of the final products may enable surgical skills educators to provide the trainees with in-depth feedback about their skills performance. Providing such feedback is in accordance with the benefits described by many theoretical models of motor skill acquisition. [15-17]. One important characteristic of these models is that the acquisition of a motor skill progresses through distinct stages, which may require different feedback strategies. Our data suggest that different feedback about different aspects of the overall performance may be necessary for different levels of trainees. That is, it is apparent that the motor process parameters related to peak tension applied during knot tying technique were well optimized by the intermediate trainees, and therefore they will most likely not benefit from augmented feedback about this aspect of their performance. On the contrary the novice trainees may need augmented feedback to optimize this parameter. On the contrary, both novices and intermediates showed poor movement economy and therefore both groups would probably benefit from feedback about movement patterns. Likewise, augmented feedback about the quality of knots may be highly beneficial to both the novice and intermediate groups in order to reach the learning goals. 3.2. Study Limitations A significant limitation of the current study is the small number of participants in each of the experimental groups. This could affect the variability in the data and therefore affect our ability to detect group differences where they truly exist; or, conversely, detect group differences where they do not exist. For example, this lack of statistical power could explain why most variables, except for the economy of motion, were not able to discriminate between the expert and intermediate groups. This is especially significant if considering using this device for the assessment of resident proficiency before proceeding to the clinical arena. In addition, to increase the statistical power, we are currently improving technological aspects of the device, such as the sensor sensitivity and the algorithms, to better determine the variables of interests.

Acknowledgments The authors would like to acknowledge the contributions of the Natural Sciences and Engineering Council of Canada for providing funding for this project. We would also like to acknowledge contributions of BISEMIC group of undergraduate engineering students from the Universidad Pontificia Bolivariana, Bucaramanga, Colombia who built the first prototype of the assessment device.

References [1] Brydges R, Sidhu R, Park J, Dubrowski A. Construct validity of computer-assisted assessment: quantification of movement processes during a vascular anastomosis on a live porcine model. Am J Surg. 2007;193(4):523-9.


523

[2] Martin JA, Reznick RK, Rothman A, Tamblyn RM, Regehr G. Who should rate candidates in an objective structured clinical examination? Acad Med. 1996;71(2):170-5. [3] Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative "bench station" examination. Am J Surg. 1997;173(3):226-30. [4] Reznick RK, MacRae H. Teaching surgical skills –changes in the wind. Nw Eng J Med. 2006;355:2664-2669. [5] Szalay D, MacRae H, Regehr G, Reznick R. Using operative outcome to assess technical skill. Am J Surg. 2000;180(3):234-237. [6] Wanzel KR, Ward M, Reznick RK. Teaching the surgical craft: from selection to certification. Curr Prob Surg. 2002;39:573–660. [7] Datta V, Chang A, Mackay S, Darzi A. The relationship between motion analysis and surgical technical assessments. Am J Surg. 2002;184(1):70-3. [8] Bann SD, Khan MS, Darzi AW. Measurement of surgical dexterity using motion analysis of simple bench tasks. World J Surg. 2003;27(4):390-4. [9] Moorthy, Munz, Dosis, Bello, Darzi. Motion analysis in the training and assessment of minimally invasive surgery. Minim Invas Therap Alli Tech. 2003;12(3):137-42. [10] Dubrowski A, Sidhu R, Park J, Carnahan H. Quantification of motion characteristics and forces applied to tissues during suturing. Am J Surg. 2005;190(1):131-136. [11] Dubrowski A, Larmer JC, Leming JK, Brydges R, Carnahan H, Park J. Quantification of process measures in laparoscopic suturing. Surg Endosc. 2006;20:1862–6. [12] Brydges R, Classen R, Larmer J, Xeroulis G, Dubrowski A. Computer-assisted assessment of onehanded knot tying skills performed within various contexts: a construct validity study. Am J Surg. 2006;192(1):109-13. [13] Hanna GB, Frank TG, Cuschieri A. Objective assessment of endoscopic knot quality. Am J Surg. 1997;174(4): 410-413. [14] Leming K, Dorman K, Brydges R, Carnahan H, Dubrowski A. Tensiometry as a measure of improvement in knot quality in undergraduate medical students. Ad Health Sci Ed. 2006;12(1):331-344. [15] Fitts PM, Posner MI. Human Performance. Belmont (CA): Books Cole; 1967. [16] Gentile AM. A working model of skill acquisition with application to teaching. Quest Monog. 1972;17:3–23. [17] Gentile AM. Skill acquisition: Action, movement, and neuromotor processes. In: Carr JH, Shepherd RB, editors. Movement Science: Foundations for Physical Therapy. Rockville (MD): Aspen; 2000. P. 87111.

524


Manual Accuracy in Comparison with a Miniature Master Slave Device- Preclinical Evaluation for Ear Surgery RUNGE, A.d ;HOFER, M. a,d;DITTRICH, E. a; NEUMUTH, T. a; HAASE, R. e; STRAUSS, M. c; DIETZ, A. a,d; LÜTH, T. c; STRAUSS, G. a,d a BMBF-Innovation Center Computer Assisted Surgery ICCAS, University of Leipzig b MIMED, Department of Micro Technology and Medical Device Engineering, Prof. Dr. Tim C. Lueth, Technische Universität München c IMETUM, Central Institute for Medical Engineering, Technische Universität München d University Hospital, ENT Department / Plastic Surgery, University of Leipzig e Phacon GmbH, Leipzig

Abstract. Manual accuracy in microsurgery is reduced by tremor and limited access. A surgical approach through the middle ear also puts delicate structures at risk, while the surgeon is often working at an unergonomic position. At this point a micromanipulator could have a positive influence. A system was developed to measure “working accuracy”, time and precision during manipulation in the middle ear. 10 ENT- surgeons simulated a perforation of the stapedial footplate on a modified 3D print of a human skull in a mock OR. Each trial was repeated more than 200 times aiming manually and using a micro-manipulator. Data of over 4000 measurements was tested and graphically processed. Work strain was evaluated with a questionnaire. Accuracy for manual and micromanipulator perforation revealed a small difference. Learning curves showed a stronger decrease both in deviation and time when the micromanipulator was used. Also a lower work strain was apparent. The micromanipulator has the potential as an aiding device in ear surgery. Keywords. Accuracy, Precision, Patient Model, Micromanipulator, Learning Curve, Stapedotomy, Master Slave

Introduction Surgery of the middle or inner ear requires accuracy at a sub-millimeter level and must therefore be classified as micro-surgery. A high level of surgical accuracy and precision is required to gain optimal postoperative results. However, there are several factors bearing a risk of inaccuracy: the working area is limited to a radius of approximately 1 cm.The movement range of the surgeon’s hand is further reduced by the use of microsurgical instruments and few possible ways of preparation (mainly anterograde). Ergonomics -the interaction between humans and another complete system, in this case the surgical setup- in an otologic intervention are rather poor: the high shoulder of the patient is an impediment, since it makes direct ergonomic access impossible and the surgeon has to perform with fully extended arms (Fig.1). Under these circumstances physiological tremor (an involuntary oscillatory movement

A. Runge et al. / Manual Accuracy in Comparison with a Miniature Master Slave Device

525

coherent in all human motion) puts another negative influence on microsurgical accuracy and precision. Mürbe et al. proved that its ampitude correlates directly with the muscular tension the weight of the instrument as well as physical exertion. [1] A resulting deviation might cause damage of delicate structures and disturb the regular surgical procedure (such as fixing a protheses or implant properly)[2] Of course the surgical experience, commonly illustrated as a learning curve, also has to be regarded [3]. One can find several studies on physiological tremor and its impact on accuracy [4,5,6,7]. In ear surgery, however, the exactness of manual pointing accuracy with a surgical instrument has not yet been evaluated and in addition has not yet been compared to that of a micro-manipulator. It eliminates increased physiological tremor and could improve ergonomics as it is remote- controlled.[8] It is the goal of this study 1. to create a suitable and realistic model in order to measure positioning accuracy of a surgical instrument regularly used in middle ear surgery. 2. to determine positioning accuracy in manual performance as compared to that of the micromanipulator. The results shall be visualized as learning curves and serve as a basis for an illustration of an individual surgical signature. 3. to determine the time-span of execution for manual performance as compared to that of the micro-manipulator. The results shall be displayed as learning curves. 4. to determine a possible reduction of physical work strain by using the micromanipulator.

1. Materials and Methods Surgical task. In this study, the perforation of the stapedial footplate with a perforator of 0.6 mm in diameter (Karl Storz, Germany) was simulated in a strongly simplified and thus easily repeatable manner. This procedure is a step of a stapedotomy, a routine intervention in the event of otosclerosis. Simulation Model and Set Up. The simulation took place in vitro using a 3D Rapid Prototyping print as a phantom.The module included the bony external hearing canal and the tympanic cavity, the latter being openly accessible. The boundary to the inner ear was replaced by a glass plate. The outline of the stapes footplate (surface area: 3.2mm²) [5] was schematically represented on it and provided with a central target point. In place of the inner ear, a miniature camera was installed and connected to an image processing application, through which any movement in the middle ear could be registered.The phantom was positioned in a clinically realistic environment. The stapedial footplate was viewed through a microscope (Zeiss, OPMI 1, Oberkochen, Germany). An upper-body model simulated the “high shoulder” of the patient. Performance. 10 otosurgeons were assigned to guide the perforator from a defined starting point to the center of the stapedial footplate via a transmeatal approach. The test-surgeons were asked to execute the procedure as precisely as possible and at an efficient pace with more than 200 repetitions. Additionally, the same assignment was carried out with the aid of a miniature masterslave device (TUM Manipulator, Prof. Tim Lueth, Technische Universitaet Muenchen). The manipulator is a surgical assistant device with bimodular structure. It contains a motor, through which an instrument-conveying carriage can be manipulated in three different degrees of freedom (x-, y- and z-directions) (Fig. 3). A movement clearance of about 10mm at each axis is possible. An operating console serves as the

526


interface with two joysticks (Fig. 2 and Fig. 3), whereby the movements of the surgeon’s hand are scaled and translated to the motor via cable. Also postural tremor is not transferred to the instrument’s tip. A large movement of the hand is translated into a small motion of the device (scaling 3.5:1). Rotatable clamps enable regulation of the miniature master-slave device in a relaxed seating position during microscopic viewing (Fig. 2). Within the groups data was divided into experts (n=5, specialists in otosurgery) and novices (n=5, residency in ENT) and again time and accuracy values were compared. Automatic evaluation. With the aid of image-processing software (Phacon, Leipzig, Germany) in combination with workflow-analysis software (ICCAS Surgical Workflow Editor, ICCAS, Leipzig, Germany), the time-span of the experiment, positioning deviation and a movement signature were automatically registered. Departure from the starting point and arriving at the end-point were detected through contact readings,all of them being set off by a touch with the tip of the perforator. The release of the starting point within the external hearing canal immediately started time measurement. The trial was concluded automatically as soon as the surgeon touched the glass plate firmly. Statistical methods. For analysis of the collected data, Microsoft Office Excel 2007 and SPSS 17.0 were used. Plain descriptive analysis provided a general overview. The Wilcoxon test was used to investigate significant differences for the manual and micromanipulator group regarding time and accuracy. Also, differences between experts and novices were considered. In order to determine the learning effect of each test-person, the mean values of the first and last 50 pointing attempts for both manual and manipulator assisted test series were placed in a percentage ratio. Graphic displays such as scatter plots, learning and plateau curves were generated for visual feedback on surgical precision and improvement of measuring results. Assessment of the stress level. After execution of the manual and manipulator assisted measurements, the surgeons had to evaluate perceived physical work-strain (stress levels) using a ratio scale from the “NASA- TLX score for evaluation of physical work strain “[9,10].

Figure 1 Unergonomic position

Figure 2 Ergonomic position


Figure 3 Set Up with model skull, PC with software and micromanipulator

2. Results Table 1 Overall results Distance Time Manual Manual [mm] [s]

Distance Time Micromanipulator Micromanipulator [mm] [s]

Mean value

.22

.65

.29

3.69

Median

.20

1.28

.26

2.59

.10

1.87

.19

4.96

Variance

.01

3.49

.04

24.64

Maximum

1.22

31.38

1.73

88.70

N

2206

2206

2286

2286

Standard

deviation

Table 2 Accuracy and time: Experts vs. novices; all values refer to the mean value of each group Micromanipulator

Manual Experts Accuracy [mm] Time [s]

.21 ± .86

.24± .14

1.14 ± 1.97

2.773 ± 3.66

Novices Accuracy [mm] Time [s]

.20 ± .11

.28 ± .23

1.45 ± 1.76

2.48 ± 5.92

527

528


Table 3 Improvement on accuracy and time depending on manual and micro-manipulator preparation, mean values Avg. 1st 50 Avg. last 50 Avg. 1st 50 Micro- Avg. last 50 Micromanual trials manual trials manipulator trials manipulator trials Accuracy [mm] Relative improvement

Time [s] Relative improvement

.26

.21

.38

21,2%

1.83

1.76 3,9%

.26 32,4%

5.58

2.71 51,5%

Overall. (Table 1) A total of 4492 measurements were carried out. The measuring values did not refer to a Gaussian distribution.The mean accuracy measured during manual performances was 0.22 mm and the application of the micromanipulator lead to deviations smaller than 0.29 mm on average. For manual perforation, 0.65 s were required, and using the micromanipulator, each pointing attempt took 3.69 s on average. Only few values exceed the majority of the measurements considerably explaining the difference between mean values and the median in measurements of accuracy and time. Experts vs. Novices. (Table 2) Comparing mean values of accuracy in manual performance (.21 mm experts vs. .20 mm novices), no significant difference between the two groups were found. It took the more experienced test persons less time to reach the target (1.14 s experts vs. 1.45 s novices). Looking at the micromanipulator performance, the novice group did not meet with the accuracy level of the expert group (.24 mm experts vs. .28 mm novices), but pointed significantly faster (2.77 s experts vs. 2.48 s novices). Learning curves. Learning curves were generated to display results of each test person. These graphs regard accuracy achieved and time required for both manual and manipulator aided pointing (Fig. 4 and Fig. 5). A total of 40 learning curves could thus be generated. (For reasons of capacity, there are only two examples provided at this point.) Comparison of the percentage ratios shows an improvement by 32.4% on average in manipulator aided accuracy while the mean manual accuracy was increased by 21.2% (Table 3). The time required for a targeting attempt at the end of the manipulator-assisted series of measurements was on average 51.5% shorter than in the beginning, the duration for manual simulation decreased by 3.9%. Movement signatures. The measured values around the target point were arranged in scatter plots, again regarding manual performance (Fig. 6) and application of the micro-manipulator separately (Fig. 7). A total of 20 individual movement signatures were thus recorded. In 7/10 cases, a higher density of the measuring points is recognisable when the micromanipulator was applied, indicating a higher precision. (For reasons of capacity, there are only two examples provided at this point.) Stress level. The overall physical exhaustion when using the micro-manipulator reached a score of 8.7 (max. score 21, with 0 showing no exertion and 21 indicating high exerction) compared to 12.4 for the manual pointing.

529

(mm)

Deviation from ref. point


No. of trials

Time

Figure 4 learning curve, comparing manual (dotted line) and master slave (solid line) accuracy

No. of trials

Deviation from ref. point on y-axis (Pixel)

(Pixel)

Deviation from ref. point on y-axis

Figure 5 learning curve, comparing time needed for manual (dotted line) and master slave (solid line) aided pointing

Deviation from ref. point on x-axis (Pixel)

Figure 6 Scatter plot manual accuracy

Deviation from ref. point on x-axis (Pixel)

Figure 7 scatter plot master slave aided accuracy

530


3. Conclusion 1.

It was possible to develop a suitable and realistic phantom for measuring positioning accuracy of a perforator. Thus automatic measuring of the manual accuracy in comparison to that of the micromanipulator was possible. 2. It was possible to display each series of accuracy measurements as a learning curve, showing interindividual differences. Evaluation of the average first and last fifty results showed a clearly more steeply descending course when the micromanipulator was used. Individual movement signatures were visualized. 3. For the time length of pointing attempts, inter-individual differences were evidenced for all 10 test surgeons displayed in the learning curves. Regarding time for master slave supported pointing attempts, the learning curves showed an even stronger decrease. 4. The NASA-TLX score was lower with deployment of the micromanipulator which indicates improved ergonomics. Applying the micromanipulator revealed no improvement in accuracy as compared to that of pure manual performance; here the accuracy was even slightly higher. This might be due to rather poor manipulator assisted accuracy especially in the beginning of the measuring trials. However, the steep learning curves show a strong learning effect in decreasing values as test persons became more familiar with the device. The manipulator used in this study is an approach to provide the surgeon with a compact, remote controlled instrument without limiting his manual performance range. Repeated usage as well as clinical evaluation may eventually show the manipulator’s potential as an assisting device in middle ear surgery or approach to the cochlea as in compensating for poor ergonomics or as an additional steady holding function (e.g. the positioning of a prothesis) without prolonging the standard surgical procedure.

References [1]

Muerbe D, Huettenbrink KB, Zahnert T et al. Tremor in otosurgery: influence of physical strain on hand steadiness. Otol Neurotol 2001; 22:672-7. [2] Babighian GG, Albu S. Failures in stapedotomy for otosclerosis. Otolaryngol Head Neck Surg 2009; 141:395-400. [3] Su, E;Win TL; Ang, WT; Lim TC; Teo, CL; Burdet, E. Micromanipulation accuracy in pointing and tracing investigated with a contact-free measurement system Conference proc: Annual international Conference of the IEEE Engineering in Medicine and Biology Society; 2009 : 3960-3 [4] Wade P, Gresty MA, Findley LJ. A normative study of postural tremor of the hand. Arch Neurol 1982; 39:358-62. [5] Duval C, Jones J. Assessment of the amplitude of oscillations associated with high-frequency components of physiological tremor: impact of loading and signal differentiation. Exp Brain Res 2005; 163:261-6. [6] Riviere CN, Khosla PK. Accuracy in positioning of handheld instruments. Amsterdam: Microsurgical and Robotic Interventions I, 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 1996: 211-213. [7] Calzetti S, Baratti M, Gresty M et al. Frequency/amplitude characteristics of postural tremor of the hands in a population of patients with bilateral essential tremor: implications for the classification and mechanism of essential tremor. J Neurol Neurosurg Psychiatry 1987; 50:561-7. [8] Maier T, Strauss G, Dietz A et al. First clinical use of a new micromanipulator for the middle ear surgery. Laryngorhinootologie 2008; 87:620-2. [9] Samel A, Wegmann HM, Vejvoda M et al. Two-crew operations: stress and fatigue during longhaul night flights. Aviat Space Environ Med 1997; 68:679-87. [10] Hart SA, Staveland LE. Human Mental Workload. Amsterdam: North Holland Press; 1988


531

Are Commercially Available Simulators Durable Enough for Classroom Use? Jonathan C. SALUD, BS, Katherine BLOSSFIELD IANNITELLI, MD, Lawrence H. SALUD, MS and Carla M. PUGH, MD, PhD1 Northwestern University Feinberg School of Medicine, Department of Surgery, 251 East Huron Street, Galter 3-150, Chicago, IL 60611 USA

Abstract. Our efforts show that commercially available simulators can be modified to affect realism and durability. Keywords. Simulators, simulation development, simulator maintenance

Introduction According to the American Cancer Society performing the digital rectal exam (DRE) is a highly effective way to detect prostate cancer [1]. Mannequin-based simulators allow medical students to practice the DRE in safe, controlled, environments [2-3]. However, the durability and reliability of commercial products for classroom use has not been established.

1. Methods There are many commercially available diagnostic prostate simulators. For classroom use and research purposes we chose a simulator with the capacity for interchangeable prostate pathologies, but there were many features that could be improved. We performed a DRE on our chosen simulator and compared it to performing the exam on real patient. The prostate gland of the simulator was not placed in the commonly accepted location and the rectum did not feel realistic. In addition, the model could only be placed in the left lateral decubitus position, Figure 1 (a). US practitioners most commonly perform the DRE with the patient in prone position [4-6].

1 Corresponding Author: Carla M Pugh, MD, PhD, Northwestern University Feinberg School of Medicine, Department of Surgery, 251 East Huron Street, Galter 3-150, Chicago, IL 60611 USA; E-mail: [email protected]

532

J.C. Salud et al. / Are Commercially Available Simulators Durable Enough for Classroom Use?

Figure 1. (a) Image of the diagnostic prostate simulator from Limb and Things. Notice that it is in the left decubitus position. (b) Image of the repositioned diagnostic prostate simulator. Notice that the model is supported by a different piece of fiberglass, as well as a cork wedge.

The model also lacked durability. After undergoing 200 student examinations, it was common for the ano-rectal junction to tear, Figure 2. It is unclear whether this was a problem with the structural design or a limitation of the material that was used.

Figure 2. (a) Image of the ano-rectal junction torn from extensive use in class. (b) Image of an intact anorectal junction.

In an effort to make the mannequin feel realistic, we modified the interior of the prostate simulator using customized foam pieces to provide support and imitate the feel of a rectal cavity. We also used commercially available artificial flesh to mimic the texture and elasticity of the rectal cavity. Our team measured the amount of stress that could be applied to three different materials: Vixskin, Cyberskin, and Fakeskin, to determine which would be most appropriate for our DRE simulator.

2. Results To allow for flexibility in positioning, we replaced the original frame with a fiberglass base that allows for prone position, Figure 1 (b). Next we modified the model’s interior with foam, allowing us to stabilize and correct the prostate gland position. Although we placed the prostate in a more accurate location, we maintained the manufacturer’s intent for interchangeable prostates. After securing the new prostate position with custom foam, we improved the model’s realism, by lining the perineum with memory foam. The foam helped imitate the fleshy feel of the ischiorectal spaces. Lastly, we addressed rectal wall feel and durability. Our tests found Fakeskin/ Dermasol 300 to have favorable elastic properties. Test results showed σ = 0.445MPa for Fakeskin/ Dermasol 300, σ = 0.447MPa for Cyberskin, and σ = 0.25MPa for Vixskin. Although Cyberskin had an advantage of σ = .002, we felt that Fakeskin/ Dermasol 300 was the best fit for our simulator because it is also robust, non-reactive, warm to touch, and moldable. The next problem was the constant tearing of the perineum’s anus. In an effort to strengthen the model we stitched different materials to the rectum base. First, we

J.C. Salud et al. / Are Commercially Available Simulators Durable Enough for Classroom Use?

533

sutured a non-lubricated condom to the models base. This worked, but success was short lived because the condom ripped after some use. Next we stitched plastic strips from a zip-lock bag to the base, which also ripped. We now use Tegaderm, folded in half, secured together by two adhesive undersides to support the base. This works better than the previous alternatives. This mechanism provides more durability to the ano-rectal junction and is a conduit for the newly developed Fakeskin/ Dermasol 300 rectum. The modified model was tested by faculty at Northwestern University’s Feinberg School of Medicine, and was approved for classroom use. Although the model is not perfect, faculty agree that it is more realistic than many commercial products and is outstanding in giving students the chance to practice clinical skills. The model now endures 400 examinations, double the original, before the perineum rips, and must be replaced. We may confirm that it is not only a material problem, but a structural one as well.

3. Conclusion This new model, a heavily modified version of the original, is an effective learning tool. Although it does not feel exactly like a human rectum, it is an enormous improvement on commercially available products and effective teaches medical students how to perform the male DRE. Research shows that our models significantly lower anxiety for first year medical students while learning the digital rectal exam. Our modified mannequin supports prior research, which maintains that companies who produce mannequins for simulation-based learning should have a closer relationship with the professors that use them [7]. A stronger relationship will address problems regarding realism and durability prior to manufacturing and distribution, leading to more reliable simulators, better-prepared medical students, and better healthcare. We have reported some of the above findings to the manufacturer and are happy to note they have incorporated a few of the changes into their product.

4. Acknowledgements The model used in this research was the Diagnostic Prostate Simulator (PART #60364) from Limbs & Things USA. We would like to thank Leslie Gittings, Nurul Hamzah, Sofiah Syed, and Latha Subramaniam for their research on artificial skin. We would also like to thank Jessica Haring, and Abby Kaye, for editing this article.

References [1]

[2] [3]

How is prostate cancer found? American Cancer Society. (2010). < http://ww2.cancer.org/docroot/CRI/content/CRI_2_2_3X_How_is_prostate_cancer_found_36.asp?rnav =cri >. N.J. Maran, and R.J. Glavin, Low- to high-fidelity simulation - a continuum of medical education?, Medical Education 37 (2003), 22-28. C.M. Pugh, K.M Blossfield-Iannitelli, D.M. Rooney, and L.H. Salud, Use of mannequin-based simulation to decrease student anxiety prior to interacting with male teaching associates, Teaching and learning in medicine (In press).

534 [4] [5] [6] [7]

J.C. Salud et al. / Are Commercially Available Simulators Durable Enough for Classroom Use? J.Y. Gillenwater, Adult and pediatric urology. Lippincott Williams & Wilkins, Philadelphia, 2002. M.F. Campbell, and P.C. Walsh, Campbell’s urology, W.B. Saunders, Philadelphia, 1998 J.L Willms, H. Schneiderman, and P.S. Algranati, Phisical diagnosis: Bedside evaluation of diagnosis and function, Williams & Wilkins, Baltimore, 1994 J.B. Cooper, and V.R. Taqueti, A brief history of the development of mannequin simulators for clinical education and training, Quality and safety in healthcare 13 (2004), i11-i14


535

Toward a Simulation and Assessment Method for the Practice of Camera-Guided Rigid Bronchoscopy Lawrence H. SALUD1, Alec R. PENICHE, Jonathan C. SALUD, Alberto L. DE HOYOS, and Carla M. PUGH Northwestern University, Department of Surgery, Feinberg School of Medicine

Abstract. We have developed a way to measure performance during a cameraguided rigid bronchoscopy using manikin-based simulation. In an effort to measure contact pressures within the airway during a rigid bronchoscopy, we instrumented pressure sensors in a commercially available bronchoscopy task trainer. Participants were divided into two groups based on self-reported levels of expertise: novice (none to minimal experience in rigid bronchoscopy) and experts (moderate to extensive experience). There was no significant difference between experts and novices in the time taken to complete the rigid bronchoscopy. However, novices touched a greater number of areas than experts, showing that novices induce a higher number of unnecessary soft-tissue contact compared to experts. Moreover, our results show that experts exert significantly less soft tissue pressure compared to novices. Keywords. Bronchoscopy Simulation, Surgical Simulation, Sensors, Learning Technologies, Simulation Support Systems, Evaluation/methodology, Humancentered computing

Introduction Rigid bronchoscopy requires exceptional technical skill to avoid causing trauma to the upper airway and to the bronchi. In addition, clinicians must avoid damaging the teeth and soft tissues, or lacerating the larynx or the bronchial mucosa [1]. Performing the proper direct insertion technique demands a considerable amount of expertise [2]. In a statement released by the European Respiratory Society, a trainee should have performed at least 20 supervised rigid bronchoscopy procedures before attempting it alone. Pulmonary fellowship training programs in the United States currently require at least 50 bronchoscopies for pulmonary trainees to achieve competency in flexible bronchoscopy [3]. While the most dangerous complications associated with rigid bronchoscopy are those related to the use of general anesthesia, the traditional apprenticeship model is not the best way to train this skill as most patients in need of the procedure are in distress and facing life or death circumstances [2]. The development of novel, manikin-based simulators that can capture and present performance metrics has provided an opportunity to define and validate assessment 1

Corresponding Author: Research Engineer, Northwestern Center for Advanced Surgical Education, Northwestern University, 303 E. Chicago Avenue Chicago, IL 60611; E-mail: [email protected]

536

L.H. Salud et al. / Toward a Simulation and Assessment Method

measures in greater depth, and to further refine training needs. To date, however, effective performance criteria for minimally invasive procedures, such as the cameraguided rigid bronchoscopy, has not been defined.

1. Objective In this article, we look at a sensor-enabled simulator that captures rigid bronchoscopy performance from novice and expert clinicians. Our goal is to determine if electronically captured bronchoscopic maneuvers can be analyzed to reveal insightful differences between expert and novice performance. By combining a commercially available bronchoscopy-teaching manikin with force-monitoring sensors, we aim to identify potentially critical assessment criteria for establishing competency in rigid bronchoscopy.

2. Methods 2.1. Participants The participants in this study were attendees at the 44th Annual Meeting of the Society of Thoracic Surgeons, held in Fort Lauderdale, FL. This meeting provides a forum for over 2,000 experts from around the world to discuss and present laboratory and clinical research concerning care of patients in need of heart, lung, esophageal, and other surgery for the chest. The conference is for individuals involved clinically or scientifically in thoracic and cardiovascular surgery. Clinician data were collected over a three-day period during which 38 clinicians visited a booth stationed in the exhibit hall and volunteered to perform a simulated rigid bronchoscopy with an endoscopic camera. The institutional review board approved this study. 2.2. The Rigid Bronchoscopy Simulator The rigid bronchoscopy simulator, as shown in Figure 1, is a newly developed task trainer that is instrumented internally with electronic sensors that can monitor and report forces on contact. A commercially available bronchoscopy model with normal anatomy of the mediastinum is typically used as a task trainer for various introductory procedures. For our study, we instrumented this model with sensors to detect possible abrasions from a rigid bronchoscope. Sensor locations were determined based on feedback from a preliminary pilot study in which both experts and novices performed the rigid bronchoscopy on the simulator. The six sensors were placed along the mucosal walls of the airway and esophageal, specifically, the tongue, the hypopharynx, the vocal cords, the upper trachea-esophageal junction, and the anterior trachea, Figure 2. The sensors are placed on the outside luminal structures to allow for smooth passage of the rigid bronchoscope. Despite their locations, the sensors are sensitive enough to pick up abrasion forces or contact pressures caused by bronchoscope maneuvers.


537

Figure 1. Clinicians perform a rigid bronchoscopy on a sensor-instrumented simulator.

Figure 2. Force-monitoring sensors are placed in six locations of the airway and esophageal, specifically, the tongue, hypopharynx, vocal cords, the upper trachea-esophageal junction, the anterior trachea, and the esophageal.

The sensor-enabled device allows learners and instructors to visualize in real-time how much contact force is being applied on surrounding tissue while advancing and withdrawing the rigid bronchoscope. In the assessment mode, the computer screen is turned away from the user. While the bronchoscopy is being performed, individual performance data are collected. Sensor inputs are sampled at a rate of 30 Hz, and the outputs are captured and stored in named data files for off-line analysis. The device used in this study is a prototype and is not yet commercially available. The computer was used in assessment mode in this study.

2.3. Experimental Protocol Before conducting a camera-guided rigid bronchoscopy, each participant completed a demographics survey indicating years in practice, gender, and specialty. They were also asked to rate their experience in performing a rigid bronchoscopy on a 4-point Likert scale. The level of expertise was then divided into two groups: novice (none to minimal) and experienced (moderate to extensive). Using our sensor-enabled task trainer, electronic data were collected from the simulator as the participants performed a complete rigid bronchoscopy including, advancing the rigid bronchoscope through

538


the airway, inspecting the carina and withdrawing the bronchoscope. Each participant was told in advance that there was a 90 percent occlusion of the airway with the intent to set time sensitive conditions. 2.4. Data Analysis In our previous work, we developed algorithms to extract key variables from our sensors to assess palpation skills [4]. Applying the same parameter extraction technique for rigid bronchoscopy, the simulator data were analyzed by using MATLAB version 7.10 data mining software (The Mathworks, Inc., Natick, MA, USA) to extract the following variables: (1) time to perform a complete rigid bronchoscopy, (2) maximum wall abrasion forces, and (3) number of areas (sensors) contacted during the procedure (an indirect measure of economy of motion and efficiency). Data analysis was performed by using SPSS version 18 (SPSS, Inc, Chicago, IL). Chi-square and analysis of variance were used to compare clinician performance between experts and novices. A p-value less than .05 was considered to be significant.

3. Results The participant group (N=38) included fourteen experts and twenty-four novices. The participants were primarily from thoracic and cardiothoracic specialties with an average of fifteen years in practice. All participants performed the procedure with the aid of the endoscopic camera and light source for visual guidance. We hypothesized that level of expertise would influence the time taken to complete the procedure and that experts would perform the task in a significantly shorter time than novices. Contrary to our expectations, results showed there were no significant differences between experts and novices in the time taken to complete the rigid bronchoscopy (45 seconds). However, the number of areas touched was higher on average for the novices (3.37 versus 2.93), although the difference was not significant. Figures 3 shows two distinct performance characteristics of one expert participant and one novice participant who performed rigid bronchoscopies on the same simulator. Each line graph, with its unique shade of grey, represents a specific anatomical location where a force-monitoring sensor is placed. The six areas are shown in the keys on the right of each line graph. The vertical axis denotes pressure units where 1 unit is equivalent to about 1.25 in pounds per square inch. The horizontal axis depicts time in seconds. In essence, the line graphs show that novices touch more area with greater pressures compared to experts. As shown in Table 1, there were no significant differences between the groups in maximum contact pressures from the bronchoscope at the tongue (experts: 2.18, novices: 3.71, p=.096), hypopharynx (experts: 6.67, novices: 6.64, p=.764), vocal cords (experts: .58, novices: .22, p= .319), anterior trachea (experts: .26, novices:.25, p=.964), or esophagus (experts: 2.04, novices: 2.51, p=.598). However, when comparing expert and novice maximum pressures on the trachea-esophageal junction, novice performers exerted significantly higher pressures with the bronchoscope at this junction than experts (experts: 1.71, novices: 3.31, p=.003). The sensor was located on the anterior portion of the esophageal wall, about five centimeters from the junction joining the trachea and the esophagus. Contact with this sensor is associated with a higher chance of an esophageal intubation.

539


Figure 3. The novice waveform exhibits significantly more rigid bronchoscope manipulation activity about the tongue and the upper trachea-esophageal (TE) junction where the latter forces are repeated from the 10 to about 26-second mark. Entry into the airway may have occurred soon after the 16-second mark where there is a one-pressure unit abrasion at the vocal cords during this time. In contrast, close inspection of the expert reveals only two instances of a less than 3 pressure units of force applied on the tongue. Two seconds later, about one half a pressure unit is exerted about the upper TE junction. Both line graphs show an initial 3.5 pressure units on the hypopharynx. This is caused by the weight of the simulated patient’s head during bronchoscopy procedure.

Table 1. Data analysis results comparing experts and novice performers of the rigid bronchoscopy procedure. *The p-value comparing maximum pressures at the upper trachea-esophageal (TE) junction is less than .05. Performance Variables Time (seconds) Number of Areas Touched Maximum Pressure – Tongue Maximum Pressure – Hypopharynx Maximum Pressure – Vocal Cords Maximum Pressure – Upper TE Junction* Maximum Pressure – Ant. Trachea Maximum Pressure – Esophageal

Experts (N=14) 45.3 2.93 2.18 6.77 .58 1.71 .26 2.04

SD 19.4 1.07 2.26 1.38 1.33 1.31 .60 2.59

Novice (N=24) 45.9 3.37 3.71 6.64 .22 3.31 .25 2.51

SD 22.16 .92 2.87 1.31 .85 1.61 1.11 2.67

p .934 .184 .096 .764 .319 .003 .964 .598

540


4. Discussion Comparisons between expert and novice performances reveal visually apparent differences during the rigid bronchoscopy procedure. The novice waveform exhibits significantly more rigid bronchoscope manipulation activity about the tongue and the upper trachea-esophageal (TE) junction where the latter forces are repeated from the ten to about the twenty six-second mark. Entry into the airway may have occurred soon after the sixteen-second mark where there is a pressure unit of abrasion force at the vocal cords during this time. In contrast, close inspection of the expert performances reveals only two instances of less than three pressure units of force applied to the tongue. Two seconds later, about one half a pressure unit is exerted about the upper TE junction. From this point forward, no contact pressures are reported. Both line graphs show an initial 3.5 pressure units on the hypopharynx. This is caused by the weight of the simulator head and its movement during the rigid bronchoscopy procedure. A correct rigid brocchoscopy entails finding the epiglottis and gently lifting it with the bronchoscope to visualize the vocal cords. The bronchoscope is then advanced through the cords and positioned in the airway [5]. Because vision through the bronchoscope is limited through its small beveled opening, it was difficult in our study for a novice to find the epiglottis. The novice users who could not find the epiglottis tended to advance the bronchoscope into the esophagus. These novices would realize that they were not in the airway. We observed that this realization occurred at varying depths within the esophagus. As such, novices would stop and reverse direction, repeatedly performing the lifting maneuver while slowly withdrawing the bronchoscope. This movement caused the bronchoscope to make contact with the sensor located at the trachea-esophageal junction. Some novices completely withdrew and reinserted the bronchoscope. A few novices were able to detect the epiglottis but proceeded to advance the instrument into the esophagus. One participant communicated that it was difficult to visualize the vocal chords and requested assistance. It is feasible to quantify differences between expert and novice performers of the rigid bronchoscopy procedure through the use of simulation that can capture electronics-based, sensor performance. Our pilot study suggests that bronchoscope abrasion forces applied to specific areas of the chest cavity can be quantified and that experts and novices may differ significantly. The fact that a significant difference in contact forces between expert and novice participants may be found at the upper trachea-esophageal junction suggests that novices may be finding difficulty setting themselves up for executing the critical step of visualizing the vocal cords, hence preventing a successful intubation. Task completion times did not differ significantly between experts and novices. The experts in our study were more accustomed to using a direct line of sight through the bronchoscope and revealed that they found the camera-guided technique a little more difficult.

5. Limitations This pilot study was intended to test the simulator’s ability to familiarize untrained learners with the basics of rigid bronchoscopy and investigate whether distinct novice performance characteristics are discoverable in relation to their expert guides.


541

Additional sensor variables need to be defined in order to extract all possible measures of performance.

6. Conclusion Training for complex procedural skills is the main goal of residency training programs. The technical mastery of procedures is best achieved outside of the operating room. After completing training, a resident in the operating room should be more equipped to focus on higher-order, cognitive tasks such as surgical decision-making and situational awareness. By providing a validated, data-driven feedback mechanism for reporting learner performance against objective standards, a simulator may provide the ideal platform for practice opportunity. This kind of expert guidance—a specialized type of learning construction at the level of the anatomy—could pave the necessary pathway to skills transfer of the rigid bronchoscopy procedure. Validated outcomes from our proposed method could be used to establish critical assessment criteria for credentialing experts in rigid bronchoscopy.

7. Acknowledgement We wish to thank Abby Kaye her assistance in editing the manuscript. The data collection venue was supported by Karl Storz, Inc. We thank them for this support.

References [1] Kvale, P.A. and A.C. Mehta, Training bronchoscopists for the new era. Clin Chest Med, 2001. 22(2): p. 365-72, ix. [2] Ayers, M.L. and J.F. Beamis, Jr., Rigid bronchoscopy in the twenty-first century. Clin Chest Med, 2001. 22(2): p. 355-64. [3] Torrington, K.G., Bronchoscopy training and competency: how many are enough? Chest, 2000. 118(3): p. 572-3. [4] C.M. Pugh and P. Youngblood. “Development and Validation of Assessment Measures for a Newly Developed Physical Examination Simulator,” J Am Med Inform Assoc, vol. 9, no. 5, pp. 448-460, 2002. [5] Ernst, A., G.A. Silvestri, and D. Johnstone, Interventional pulmonary procedures: Guidelines from the American College of Chest Physicians. Chest, 2003. 123(5): p. 1693-717.

542


Use of Sensor Technology to Explore the Science of Touch Lawrence H. SALUD1. MS and Carla M. PUGH, MD, PhD Northwestern University, Department of Surgery, Feinberg School of Medicine Abstract. Two, world-renown researchers in the science of touch (Klatzky and Lederman) have shown that there are a set of reproducible and subconscious maneuvers that humans use to explore objects. Force measuring sensors may be used to electronically identify and quantify these maneuvers. Two sensored silicone breast models were configured to represent two different clinical presentations. One-hundred clinicians attending a local breast cancer meeting performed clinical breast examinations on the models, and their performance was captured using sensor-based data acquisition technology. We have found that Klatzy and Lederman’s previously defined touch maneuvers are used during the clinical breast examination and can be identified and quantified for the first time using sensor technology. Keywords. Haptics, Medical Simulation, Sensors, Learning Technologies, Simulation Support Systems, Evaluation/methodology, Human-centered computing

1. Introduction Extensive research by Klatzky and Lederman has shown that there is a set of reproducible and subconscious maneuvers that humans use to manually explore objects [1, 2]. The authors have established links between desired knowledge about objects and hand movements during haptic object exploration. Named “exploratory procedures (EPs)”, the hand movements are stereotyped movement patterns having certain characteristics that are invariant and highly typical when exploring various properties of an object, such as temperature and hardness. The EPs are comprised of lateral motion, pressure, static, unsupported holding, and enclosure, Figure 1(a). The lateral motion EP is used to explore an object that has texture. The pressure EP, which is defined as applying a normal force to an object, is used to explore hardness. The static EP is used to discover an object’s temperature and occurs when the hand drapes over the object’s surface, maximizing skin contact without an obvious effort. The unsupported holding EP occurs when the object is lifted away from any supporting surface and maintained in the hand without any effort to mold the hand to the object. This maneuver is associated with assessing an objects weight. The enclosure EP, which is used to judge an object’s global shape and size, involves molding the hand to object contours. Finally, the contour following EP is used to encode precise shape information and generally takes the form of traversing along edges of an object with the fingertips. 1

Corresponding Author: Research Engineer, Northwestern Center for Advanced Surgical Education, Northwestern University, 303 E. Chicago Avenue Chicago, IL 60611; E-mail: [email protected]

L.H. Salud and C.M. Pugh / Use of Sensor Technology to Explore the Science of Touch

543

Sensor-enabled task trainers and virtual reality devices have been designed to study palpation techniques and have been used to discover important clinical performance characteristics [3, 4]. For example, Pugh et al. have shown that using force sensors embedded in task trainers make it possible to capture and analyze handson performance [5]. In a 2006 publication, eleven sensors were embedded in a simulated breast model to capture clinician performance during clinical breast examinations [6]. Sensor measurements were analyzed to quantify an individual’s clinical breast examination (CBE) time, the number of sensors touched, the maximum palpation pressure and the mean palpation frequency. Furthermore, palpation was measured from a single sensor by plotting touch forces over time. Figures 1 (b) and (c) reveal differences between “tapping” and “rubbing” – two terms used by Pugh et al. to describe the palpation maneuvers being observed. Differences between two classes of tapping and rubbing are shown as measurable. One class is tapping or rubbing in a single area of the sensor. The other class is tapping or rubbing in multiple areas of the sensor. It is now clear that these maneuvers may be closely related to Klatzky and Lederman’s EPs.

Figure 1. Klatzky and Lederman have shown that there are six reproducible and subconscious maneuvers that humans use to detect objects (a). Force measuring sensors may be used to electronically identify these movements. Forces plotted over time illustrate differences between tapping (b) and rubbing (c).

In addition, there is an emerging body of work in medicine and engineering using technology to better understand touch [7-10]. Remote sensing and manipulation in teleoperated robotics, haptic-based mobile devices, and human-computer interaction have been the predominant focus areas in this body of work. While a by-product of many of the technologies developed may provide mechanisms to measure EPs, no one has yet to establish an association between Klatzky and Lederman’s video recorded observations and electronically measured palpation techniques.

544


2. Theoretical Framework and Hypothesis We hypothesize that electronic capture of hands-on clinical skills in performing theduring a CBE may be used to identify Klatzky and Lederman’s EPs. By embedding sensors on two simulated breast models, we expect to map at least two of the EPs, lateral motion and pressure, to our sensor measurements. Additionally, we hypothesize that contour following and static pressure are observed when clinicians are palpating a mass or abnormality. The remaining two EPs are not applicable to the breast examination as only the fingertips are used. Moreover, we propose that the presence of “tapping” and/or “rubbing” during a CBE could be characteristic of at least two of the six EPs, namely lateral motion and pressure. By finding single and multiple area tapping and rubbing, we suggest a starting point into electronically deciphering the pressure EP (tapping) and lateral motion EP (rubbing).

3. Methods and Materials 3.1. Participants The participants in this study were 102 attendees at the 7th Annual Lynn Sage Breast Cancer Symposium, held in Chicago, IL. The Lynn Sage Breast Cancer Symposium provides a forum for discussing and presenting laboratory and clinical research concerning the care of patients with breast cancer. The conference is for individuals involved clinically or scientifically in diagnostic and therapeutic radiology, oncology, surgery, gynecology, family practice, and genetics. Each year, this conference brings together over 1,200 experts from around the world to exchange ideas and present research in an effort to further the knowledge about breast cancer within the medical community. After obtaining institutional review board approval, clinician data were collected over a 4-day period. One hundred two clinicians visited a booth stationed in the exhibit hall and volunteered to perform a full CBE on each of 2 silicone breast simulators. 3.2. The Breast Examination Simulators The breast examination simulator is a newly developed task trainer that is instrumented internally with several electronic sensors. The breast models that are part of the simulator can be reconfigured to represent various clinical presentations. Two different presentations were used for this study. Simulator A consisted of a dense breast with a firm 2-cm mass in the upper inner quadrant (UIQ), Figure 2 (a). Simulator B was a dense, premenopausal breast with thickened breast tissue in the upper outer quadrant (UOQ) and the inframammary ridge area, Figure 2 (b). Sensor inputs are sampled at a rate of 30 Hz, and the outputs are captured and stored in named data files for off-line analysis. Figure 2 (c) shows a diagram of sensor placements for the breast models. One of the eleven sensors in Simulator A is located in the UIQ just below the firm mass. While the examination is being performed, individual performance data are collected. The simulators used in this study are prototypes and are not yet commercially available.


545

Figure 2. Simulator A contains a 2 cm firm mass (a) at the upper inner quadrant (UIQ) while Simulator B is configured as a dense breast with thickening (b). Both simulators are instrumented with eleven force measuring sensors on a solid planar support platform. Sensor locations are shown in (c).

3.3. Experimental Protocol Before examining the two simulators, clinicians completed a background survey indicating specialty. Clinicians were asked to perform complete CBEs and were told this was an annual examination with no complaints. Each of the 102 clinicians evaluated 2 simulators for a total of 204 examinations. Electronic data were collected directly from each simulator as the participants performed simulated CBEs. After examining each simulator, participants were asked to document their clinical findings on an assessment form. Specifically, the participants were asked to describe probable diagnoses of any discovered pathologies. Force measurements of “tapping” and “rubbing” from a single sensor was plotted over time and graphed. Further, a representative sample of one participant’s CBE was graphed for both the firm mass sensor of Simulator A and a similarly located sensor on Simulator B. Qualitative, visual interpretations were performed on the graphs to search for the presence of pressure and lateral motion EPs using the “tapping” and “rubbing” waveforms of Figures 1 (a) and (c) as comparative guidance. 3.4. Data Analysis Surveys were coded and analyzed using descriptive and comparative inferential statistics. The simulator data were analyzed using MATLAB 5.1 data mining software (The Mathworks, Inc., Natick, MA, USA) to extract the following variables: (1) time to perform a complete examination, (2) frequency each anatomic area was palpated, (3) maximum pressure used when palpating each area, and (4) number of areas (sensors) palpated during the examination (an indirect measure of thoroughness). Data analysis was performed by using SPSS version 10 (SPSS, Inc, Chicago, IL). A t-test was used to compare clinician performance on the 2 clinical presentations. A p-value less than .05 was considered to be significant.

4. Results The maximum pressure was higher for Simulator A than it was for Simulator B, Table 1. In contrast, the number of sensors touched and the mean palpation frequency were significantly higher for Simulator B than for Simulator A.

546


As shown in Figure 3, multiple area tapping was used mostly on the sensor just below the firm mass in Simulator A. Klatzky and Lederman’s work show that this is the preferred technique (the pressure EP) to assess hardness, as in a solitary breast mass. This correlates with the clinical presentation of Simulator A. The same participant performed only multiple area rubbing on Simulator B. Klatzky and Lederman’s work show that this is the preferred technique (the lateral motion EP) to assess texture, as in thickened breast tissue. This correlates with the clinical presentation of Simulator B.

Figure 3. A participant’s CBE palpation pressure is recorded electronically, and one of the eleven sensors of both simulators A and B is plotted as shown. The sensor is located at the UIQ on both simulators, and under the mass of simulator A as shown in Figure 2(a) and (c). After 12 seconds, the participant applies multiple area tapping with increasing pressure on simulator A. For simulator B the participant performs what looks to be rubbing at about the 4 and 23 second mark.

Table 1. Mean Clinician (N=102) Breast Exam Performance Characteristics between Simulator A and B Performance Variable

Simulator A (Dense breast 2cm firm mass)

Simulator B (Dense breast w/ thickening)

Time (seconds)

40.5

42.28

No. of sensors touched

4.54*

6.16

5.35

5.12

17.03*

24.01

87%

68%

Maximum pressure

1

Mean palpation frequency

2

Percent correct 1 – 1 PU (Pressure Unit) is about 1.25psi 2 – measure for palpations 30 times/second * – p < .01 between simulators

5. Discussion The results suggest that when comparing Simulator A to Simulator B, the exploratory procedures appear to correlate with the clinical findings of the breast model. When there is a solitary, firm nodule (hardness objects property), the maximum pressure is


547

higher and correlates with use of the pressure EP to detect hardness. Not only with the pressure higher, but it was used in multiple areas within a small radius, which may hint to contour following to detect the size of the mass. For Simulator B where there was a denser breast model with thickened breast tissue (texture object property) more rubbing occurs as evidenced by comparatively higher frequencies and higher numbers of sensors touched. This correlates with the use of the lateral motion EP to detect texture.

6. Future Work The data in this study was gathered using force sensors that are limited to reporting direct, normal forces. We aim to develop and use shear force sensors to help us delineate not only between pressure and lateral motion, but also between sub-classes of motion such as circular versus static contact. Moreover, we aim to develop the ability to delineate between experts and novice performers such that we may be able to understand how clinical exam maneuvers effect diagnostic accuracy at this more granular and electronically measured level.

7. Conclusion Pressures are higher in the presence of a firm mass while wide area lateral palpation movements correlate more closely with a breast model with texture. This is consistent with Klatzky and Lederman’s findings. Moreover, by graphically interpreting single and multiple area tapping and rubbing, we have proposed a window into quantitatively defining the pressure and lateral motion EPs.

8. Acknowledgement We wish to thank Abby Kaye and Jonathan Salud for their assistance in editing the manuscript. We also wish to thank Dr. Eric Volkman, Dr. Timothy Rankin and Alec Peniche for assisting in the collection and preparation of the dataset. This research was supported by the Augusta Webster Educational Innovation Grants Program at Northwestern University’s Feinberg School of Medicine. Figure 1 (a) was revised from Lederman, S.J., & Klatzky, R.L. (1987). Hand movements: A window into haptic object recognition. Cognitive Psychology, 19(3), 342-368.

References [1] Lederman, S.J. and R.L. Klatzky, Hand movements: a window into haptic object recognition. Cogn Psychol, 1987. 19(3): p. 342-68. [2] Klatzky, R.L. and S.J. Lederman, Stages of manual exploration in haptic object identification. Percept Psychophys, 1992. 52(6): p. 661-70. [3] Pugh, C.M., et al., Use of a mechanical simulator to assess pelvic examination skills. JAMA, 2001. 286(9): p. 1021-3. [4] Forrest, N., S. Baillie, and H.Z. Tan, Haptic Stiffness Identification by Veterinarians and Novices: A Comparison. World Haptics 2009: Third Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Proceedings, 2009: p. 646-651, 656.

548


[5] Pugh, C.M. and P. Youngblood, Development and validation of assessment measures for a newly developed physical examination simulator. J Am Med Inform Assoc, 2002. 9(5): p. 448-60. [6] Pugh, C.M., et al., A simulation-based assessment of clinical breast examination technique: do patient and clinician factors affect clinician approach? Am J Surg, 2008. 195(6): p. 874-80. [7] Feller, R.L., et al., The effect of force feedback on remote palpation. 2004 Ieee International Conference on Robotics and Automation, Vols 1- 5, Proceedings, 2004: p. 782-788, 5306. [8] Okamura, A.M., Haptic feedback in robot-assisted minimally invasive surgery. Curr Opin Urol, 2009. 19(1): p. 102-7. [9] Mahvash, M., et al., Force-feedback surgical teleoperator: Controller design and palpation experiments. Symposium on Haptics Interfaces for Virtual Environment and Teleoperator Systems 2008, Proceedings, 2008: p. 465-471, 480. [10] Gerling, G.J., et al., The Design and Evaluation of a Computerized and Physical Simulator for Training Clinical Prostate Exams. Ieee Transactions on Systems Man and Cybernetics Part a-Systems and Humans, 2009. 39(2): p. 388-403.


549

Real-Time “X-Ray Vision” for Healthcare Simulation: An Interactive Projective Overlay System to Enhance Intubation Training and Other Procedural Training Joseph T. SAMOSKYa,b,c,1, Emma BAILLARGEONb, Russell BREGMANb,c, Andrew BROWNb, Amy CHAYAb, Leah ENDERSb, Douglas A. NELSONb,c, Evan ROBINSONc, Alison L. SUKITSb and Robert A. WEAVERb,c a Department of Anesthesiology, University of Pittsburgh b Department of Bioengineering, University of Pittsburgh c Simulation and Medical Technology R&D Center, University of Pittsburgh

Abstract. We have developed a prototype of a real-time, interactive projective overlay (IPO) system that creates augmented reality display of a medical procedure directly on the surface of a full-body mannequin human simulator. These images approximate the appearance of both anatomic structures and instrument activity occurring within the body. The key innovation of the current work is sensing the position and motion of an actual device (such as an endotracheal tube) inserted into the mannequin and using the sensed position to control projected video images portraying the internal appearance of the same devices and relevant anatomic structures. The images are projected in correct registration onto the surface of the simulated body. As an initial practical prototype to test this technique we have developed a system permitting real-time visualization of the intra-airway position of an endotracheal tube during simulated intubation training. Keywords. Projective augmented-reality display, visualization, non-contact position sensing, intubation training, human-computer interaction, user interfaces.

Introduction A basic but significant limitation of training on real patients is that human bodies are mainly visually opaque. For many procedures—endotracheal tube insertion, Foley catheter placement, bronchoscopy—there may be advantages if a trainee could see what is happening inside the body as he or she manipulates a tool or device outside the body. This could enhance the trainee’s development of accurate and memorable cognitive and spatial models of the internal consequences of their external actions. We can accomplish this in simulation-based training environments by means of augmented reality techniques: via computer-generated visual displays we can make seen what is normally hidden from view in the real world of clinical medicine. We developed this approach in a prototype augmented-reality display of endotracheal tube (ET) position. 1 Corresponding author: Joseph T. Samosky, Director, Simulation and Medical Technology R&D Center, University of Pittsburgh, 230 McKee Place, Suite 401, Pittsburgh, PA 15213; E-mail: [email protected].

550

J.T. Samosky et al. / Real-Time "X-Ray Vision" for Healthcare Simulation

Figure 2 Neodymium disk magnet fixed transversely in lumen near tip of endotracheal tube.

Figure 1 A linear array of Hall effect sensors is affixed to the trachea and right mainstem bronchus of the fullbody simulator. A single Hall effect sensor detects esophageal intubation.

Figure 3 One frame from digital video of ET tube being inserted into coronally hemisected goat trachea with chromakeyed background converted to black.

Our approach aims to avoid the potential encumbrance of head-mounted displays or mirrors interposed between user and workspace, and differs from prior work in projecting anatomy onto a mannequin (such as [1, 2]) in that the IPO system senses the real-time position of a medical device inserted into or manipulated within the simulated body and the projected video images include representations of the device correlated with the position and motion of the actual device. We employ a sensed physical variable to control the playback position of a digital video to generate dynamic simulated images of the 1-dimensional motion of a device, tool or object.

Methods and Materials The IPO system augments a standard full-body human simulator (SimMan®, Laerdal Medical AS) with: (1) a DLP projector (Dell M209X) mounted above it on an adjustable support arm, and (2) a non-contact position sensing system (Figures 1 and 2) that measures the depth of insertion of an endotracheal tube. Control software written in LabVIEW (National Instruments) uses the measured position of the ET tube to select the playback frame of a pre-recorded digital video (Figure 3) that presents a top-down view of an endotracheal tube being inserted into a partially cut-away trachea. For the prototype, a hemisected goat trachea was filmed against a green chroma-keyed background. The background was then post-processed to black. A non-contact method for measuring ET tube depth was desired to enable free manipulation of the ET tube. We designed an array of 7 Hall effect sensors, affixed to the mannequin’s trachea, to sense the position of a small neodymium disk magnet fixed in the lumen near the tip of a standard ET tube. A custom algorithm converts the sensor outputs to a measurement of linear position. An additional sensor attached to the esophagus detects the clinically important error of esophageal intubation. Software controls playback of the digital video via an ActiveX interface to Windows Media Player. Playback position is calibrated and synchronized to the measured tip position of the real ET tube. User control of system functions is performed via a Wiimote.

J.T. Samosky et al. / Real-Time "X-Ray Vision" for Healthcare Simulation

551

Figure 4 Real-time augmented reality display of endotracheal tube position within trachea. An overhead DLP projector projects image correlated to actual tube position onto neck and thorax of simulator. Trainee performance metrics such as intubation depth and time are projected for objective assessment and feedback. System functions are controlled by a Wiimote, employed as a simple, compact wireless control interface.

Results The system successfully creates the illusion of a “see through” view through the anterior chest wall during simulated intubation training (Figure 4). The non-contact intubation depth sensing system exhibited an average absolute accuracy of 1.1 mm +/1.9 mm (s.d.). The position of the projected ET tube matched the actual ET tube position to within 2 mm (average across 13 depths from 14 to 26 cm). The system also accurately detected esophageal intubation, alerting the user via an auditory alarm.

Discussion Using the IPO system trainees can clearly and immediately see important errors such as mainstem bronchus intubation in a way that would otherwise be obscured. We hypothesize that such augmented visual feedback can enhance the development of memorable mental models of procedures, and that proximate feedback on errors is superior to delayed debriefing in developing accurate psychomotor skills efficiently. Extensions of the technique to other procedures are in development (including Foley catheterization and pulmonary artery catheterization). Experiments with clinical trainees are planned to examine the ultimate training effectiveness of the system. References [1] [2]

Kondo D, and Kijima R. Poster: Free Form Projection Display: Virtual Image Located Inside Real Object. IEEE Symposium on 3D User Interfaces. Reno, Nevada. March 2008. Kondo D, Kijima R, and Takahashi Y. Dynamic Anatomical Model for Medical Education using Free Form Projection Display. Proc. 13th International Conference on Virtual Systems and Multimedia. Brisbane, Australia. September 2007: 142-149.

552


Toward a Comprehensive Hybrid Physical-Virtual Reality Simulator of Peripheral Anesthesia with Ultrasound and Neurostimulator Guidance Joseph T. SAMOSKYa,b,d,1, Pete ALLENb, Steve BORONYAKb, Barton BRANSTETTERc, Steven HEINb, Mark JUHASb, Douglas A. NELSONb,d, Steven OREBAUGHa, Rohan PINTOb, Adam SMELKOb, Mitch THOMPSONb and Robert A. WEAVERb,d a Department of Anesthesiology, University of Pittsburgh b Department of Bioengineering, University of Pittsburgh c Department of Radiology, University of Pittsburgh d Simulation and Medical Technology R&D Center, University of Pittsburgh

Abstract. We are developing a simulator of peripheral nerve block utilizing a mixed-reality approach: the combination of a physical model, an MRI-derived virtual model, mechatronics and spatial tracking. Our design uses tangible (physical) interfaces to simulate surface anatomy, haptic feedback during needle insertion, mechatronic display of muscle twitch corresponding to the specific nerve stimulated, and visual and haptic feedback for the injection syringe. The twitch response is calculated incorporating the sensed output of a real neurostimulator. The virtual model is isomorphic with the physical model and is derived from segmented MRI data. This model provides the subsurface anatomy and, combined with electromagnetic tracking of a sham ultrasound probe and a standard nerve block needle, supports simulated ultrasound display and measurement of needle location and proximity to nerves and vessels. The needle tracking and virtual model also support objective performance metrics of needle targeting technique. Keywords. Peripheral nerve block, mixed reality, hybrid reality, virtual reality, human computer interaction, 3D segmentation, MRI, stereolithography

Introduction A variety of approaches have been pursued to simulate the multiple and varied aspects of peripheral nerve block procedures, including physical models, models with embedded conductive “nerves” [1], computer-simulated ultrasound [2], VR systems with commercial haptic interfaces [3], and novel haptic devices to simulate needle force profiles during insertion [4]. A fundamental design decision for simulation systems is the choice of physical or virtual models. We believe each has merits and limitations for perceptual display and 1 Corresponding author: Joseph T. Samosky, Director, Simulation and Medical Technology R&D Center, University of Pittsburgh, 230 McKee Place, Suite 401, Pittsburgh, PA 15213; E-mail: [email protected].

J.T. Samosky et al. / Toward a Comprehensive Hybrid Physical-Virtual Reality Simulator

553

interaction. In this work we are investigating a hybrid approach to emulate the salient aspects of peripheral anesthesia. Our initial prototype focuses on development of a training system for brachial plexus blockade.

Materials & Methods Isomorphic Physical and Virtual Models. A high-resolution 3D MRI scan was obtained of a subject’s arm while encased in a silicone mold and fiberglass support shell. The mold was used to create a silicone model of the skin of the arm and axilla. Nerves, blood vessels and other relevant anatomy were segmented from the MRI scan (using MIMICS, Materialise NV) to create a virtual model that was registered with the physical model. The virtual model was also used to fabricate physical models of fascial planes via stereolithography. These components of the physical model provide the characteristic “pop” felt during needle passage through the fascia. Needle and US Probe Tracking. A 3D electromagnetic tracking system (Aurora, NDI Inc.) was employed to measure the pose of a standard block needle (with attached miniature 5 DOF sensor) and sham ultrasound probe (with a 6 DOF sensor). Anesthetic Syringe Simulator. A subsystem consisting of a flow sensor, solenoid valves and fluid reservoirs enable the ability to draw back blood simulant if the needle tip is determined to be within a (virtual) blood vessel, indicating the need to reposition the needle prior to injection. Fluid resistance can be controlled during injection and plunger pullback depending on the sensed location of the needle tip with respect to structures in the virtual model such as blood vessels, soft tissue or nerve fascicles. Simulated Ultrasound (US). We explored initial capability to track the pose of the US probe, specify an imaging plane through the virtual model and display an approximation to an US image via reformatting of 3D model data. Twitch Display. A set of mechatronic actuators and a cable-drive system was incorporated into the physical arm model to provide visual feedback of muscle twitch to the user when any of the four principal nerves of the brachial plexus are stimulated. Neurostimulator Interface. Any standard clinical neurostimulator can be used with the system. An electronic interface senses the electrical output of the neurostimulator. The measured rate and intensity of the current pulses are then combined with needle-tonerve distance data to compute twitch output quality and intensity.

Results A translucent cylindrical silicone phantom and corresponding virtual model were constructed to test the system (Figure 1 and Figure 1 inset). We verified physicalvirtual registration and that needle tip contact with soft tissue, nerve or vessels was correctly identified. The syringe simulator provided blood simulant during pull-back when the needle tip contacted a virtual vessel and varied flow resistance depending on tip location. Metrics computed on needle tip trajectories included total path length and instantaneous and average velocity and acceleration. The full extremity mechatronic model (Figure 2) displayed four- and two-finger twitch, elbow extension and flexion, and wrist adduction as well as graded response to intensity. Figure 3 shows the corresponding isomorphic virtual model derived from the 3D MRI data, with segmented skin, fascia, brachial artery and vein, and nerves of the brachial plexus.

554

J.T. Samosky et al. / Toward a Comprehensive Hybrid Physical-Virtual Reality Simulator

Sham ultrasound probe and block needle, each with tracking sensor Cylindrical phantom “limb” model

Monitor displaying instructor interface Monitor displaying learner interface Syringe simulation subsystem

Aurora electromagnetic field generator Anesthetic syringe

Figure 1 Peripheral nerve block simulator test system: tracking, display and syringe. Inset: Detail of test phantom with tracked simulated US probe and tracked block needle.

Physical arm model fabricated via lifecasting and stereolithography Standard commercial neurostimulator Actuators and driver electronics

Figure 2 Mechatronic arm with twitch actuators and neurostimulator interface.

Figure 3 MRI-derived virtual model isomorphic to physical model, with segmentation of the brachial artery and vein, and the nerves of brachial plexus.

Discussion We are currently evaluating the fidelity of needle insertion haptics and further developing the US simulation algorithms. We are also exploring augmented visual feedback to the trainee and automated proficiency metrics based on tracked needle trajectories, needle targeting accuracy, and sensed injection rate and volume. References [1] [2] [3] [4]

Eastwood, CB and Moore, DL. A Simple, Near-Ideal Simulation Model for Ultrasound-Guided Regional Anesthesia. SPA/AAP Pediatric Anesthesiology 2010 Winter Meeting (poster P106). Zhu Y, Magee D, Ratnalingam R, and Kessel D (2007). A Training System for Ultrasound-Guided Needle Insertion Procedures. MICCAI 2007: 566-574 Ullrich S, Frommen T, Rossaint R, and Kuhlen T (2009). Virtual Reality-based Regional Anaesthesia Simulator for Axillary Nerve Block. Medicine Meets Virtual Reality 17, 2009: 392-394 Lim YJ, Valdivia P, Chang C, and Tardella N (2008). MR Fluid Haptic System for Regional Anesthesia Training Simulation System. Medicine Meets Virtual Reality 16, 2008: 248-253.


555

A Fixed Point Proximity Method for Extended Contact Manipulation of Deformable Bodies with Pivoted Tools in Multimodal Virtual Environments Ganesh SANKARANARAYANAN a, Zhonghua LU b and Suvranu DE a,1 a Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute, USA b Intelligent Manufacturing and Control Institute, Wuhan University of Technology, China

Abstract. In the real world, tools used for manipulation are pivoted with specialized tips for specific functions including grasping and cutting. Manipulating deformable virtual objects with them involves maintaining extended contact, which is difficult due to the variations in applied force. Our method consists in selecting a fixed set of points on the jaws of a pivoted tool, and placing them either equidistant or according to the geometry of the tool. Vertex and triangle proximities are calculated for each of the interacting deformable objects for collision detection. This method was successfully tested in a surgical simulation scenario where a deformable omental fat model was grasped and retracted while maintaining full contact with the pivoted tool tip at all times. .Keywords. Collision detection, Haptics, Virtual environments

Introduction Haptic interaction of deformable bodies is a difficult problem because of the computational time requirements for collision detection and deformation calculations. For realistic contact and response it is often desired to model the tool as a line (“ray”) at the expense of computation time for collision. In real world interactions the tools are made for special purposes and they often have pivoted jaws for grasping. Such tools often have extended contact along their jaws due to squeezing force applied to the tool handles for tighter contact. Interaction of such tools can still be modeled as a line segment for rigid contacts whereas the same is not true for deformable objects. The major problem in such interactions is the extended contact of the tool with the model, and the collision detection and response between the pivoted jaws and the deforming objects. Collision detection is a well known problem in the field of interactive computer graphics and haptics. In collision detection, interpenetration of source and target 1

Suvranu De, Associate Professor, Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer, Polytechnic Institute, Troy, NY, USA. Email: [email protected]

556

G. Sankaranarayanan et al. / A Fixed Point Proximity Method

objects is constantly checked and reported back for subsequent action. Various types of collision checking methods exist and we refer to surveys [1] and [2] for more details regarding collision detection. In [3] a ray-based haptic rendering method was introduced in which the haptic probe was modeled as a line and collision with convex objects was performed using hierarchical bounding boxes. In [4] a fast and efficient line based collision detection method known as dynamic point was introduced for haptic based applications. For deformable objects, modeling of tool tissue interactions is modeled using a variety of techniques. We refer to [5] for a detailed review in this area. In this paper, we introduce a fixed point proximity (closest distance) algorithm specifically for the interactions between a pivoted tool and triangular meshes of deformable objects in haptic enabled virtual environments. This method is based on a fixed set of carefully sampled points along the pivoted tool jaws whose proximity to the nearby triangular meshes are constantly updated in real-time for haptic interactions. The vertex and triangular neighborhood information in a mesh and the spatial and temporal coherence is used to update the proximities, which gives a near constant time complexity for arbitrary mesh sizes for a fixed set of samples points. We show the effectiveness of this method in a surgical simulation scenario as a case study.

1. Materials and Methods The fixed point proximity method consists of a preprocessing step and an online computation step during which the proximities are updated at near constant time. The preprocessing step consists of selection of fixed points on the pivoted tool tip for collision detection and subsequent application of constraints to the deformable model and haptic response to the user. The selection criteria include the shape of the pivoted tool jaws and the mesh density of the interacting deformable objects. For coarser meshes, fewer points would be sufficient. Once the points are chosen, an initial distance check is performed from each of the points in the tool jaw to the deformable meshes to compute proximity and updated in a database. Two types of proximities can be computed for each mesh – a proximity to the closest triangle surface PijT (Figure 1a) and a proximity to the closest vertex PijV (Figure 1b). For coarser mesh, the triangle proximity would provide a better approximation of the interaction than the vertex proximity and for denser meshes; vertex proximity would itself be enough for realistic interactions. The vertex proximity can also be used for non triangular meshes (example surface voxels of volumetric models). For each of the meshes in the scene, the vertex and triangular neighborhoods are computed and stored in a database. It is reasonable to assume [4, 6] that at haptic sampling rates, the vertex and triangle proximities vary only minimally between two successive frames. Therefore, the precomputed triangle and vertex neighborhood information is used to update the proximities by checking the nearest neighbors at runtime. Collision is detected when the distance between the proximity and the fixed points are less than a predetermined small tolerance value. Once collision is reported, constraints are applied to the deformable mesh at the vertices of the contact triangle or at the contact vertex when using vertex proximities. Reaction force is then applied to the user based on the computed force acting at the contact vertex or the triangle of the model.


(a)

557

(b)

Figure 1. Pivoted tool jaws with four fixed points for each jaw. (a) Triangle proximity and triangle neighbors (b) Vertex proximity and vertex neighbors.

2. Results The fixed point proximity method was applied to a surgical simulation scenario where a surgical grasper was used to interact with an omental fat in the upper region. Both grasping and scooping motion was achieved successfully with our method. The testing scenario consisted of organs of the upper peritoneal region created for a Laparoscopic Adjustable Gastric Banding (LAGB) simulator. In addition to the organ geometry, volumetric models of omental fat were created and simulated using position based dynamics [7]. The implementation is robust and utililizes graphical processing units (GPUs) for computational efficiency. The PHANToM Omni haptic interface device was used for force feedback. The buttons in the Omni controlled the opening/closing of the tool jaws. Triangle proximity was used for collision detection while interacting with the fat. There were a total of 1581 triangles faces in the mesh. A total of four evenly spaced fixed points were attached to each of the jaws of the surgical tool (Figure 2a). Whenever the proximity distance was less than the tolerance value, contact was established between the fixed point and the closest triangle. The opening/closing of the jaws provided two states for simulation. In the closing state, squeezing and manipulation of the fat tissue was made possible by directly applying position constraints to the tetrahedral vertex nearest to the contacted surface triangle. Subsequently, the position and orientation changes in the rest length of the interacting tetrahedra were computed and the corresponding force vector was applied to the haptic device. In the opening state of the tool, the position constraints applied to the fat were removed allowing them to return to their original configuration. Figure 2b shows a snapshot from the simulator where the fat was scooped by the surgical grasper. The fixed point based proximity method enabled the fat to be scooped by the tool in a realistic manner, similar to what would be observed in a real surgical video. In figure 2b, one can observe the bulging of the fat as it is squeezed by the grasper tool.

558


(a)

(b)

Figure 2. (a) Surgical grasper tool (rendered in wireframe) with four equally spaced points for the pivoted tip and the corresponding proximity lines. (b) Grasping and pulling up the omental fat using a surgical grasper.

Figure 3 shows a plot of collision detection time for various mesh sizes and three sets of fixed points half of which were on each jaw. The scene consisted of three meshes, a liver, stomach and omental fat. Only the fat mesh was tessellated with increasing number of triangles. The total number of triangles after tessellation was 19404, 26656, 45472 and 67182 respectively. The timing plot clearly shows that for a fixed set of points, the fixed point proximity method has a constant time of computational complexity. During the timing trials, the vertex and triangle neighborhood information along with initial distance check was computed and it ranged from a minimum of 24 to a maximum of 40 seconds, which is quite low for collision detection preprocessing times.

Figure 3. Computation time for collision detection for three different sets of fixed points with various number of triangles in the scene.

3. Conclusions In this work we proposed a new method called the fixed point proximity to specifically address the problem of extended contact manipulation of deformable objects with


559

pivoted tools. By using vertex and triangle neighborhood information, the collision detection between the points on the pivoted tool jaws and the model was computed at haptic rates. Position constraints were used as the collision response and applied directly to the tetrahedral nodes nearest to the contact surface of the tool to the deformable model. We also showed a successful implementation of this method in a realistic surgical simulation scenario that involved manipulation of highly deformable fat tissues. Since our method requires local neighborhood information for fast updates of the proximities, any change in topology would require some computational time to update the neighborhood structure. We plan to use more efficient methods for updating the neighborhood structure to enable cutting and other changes to topology during the interactions. We also plan to test this method on more complex pivoted tool structure which cannot be approximated by straight line.

Acknowledgements The authors gratefully acknowledge the support of this NIH/NIBIB through grant # R01EB005807.

References [1] [2]

[3]

[4]

[5] [6]

[7]

P. Jimnez, F. Thomas, and C. Torras. 3D Collision Detection: A Survey.Computers and Graphics, 25(2):269–285, Apr 2001. M. Teschner, S. Kimmerle, B. Heidelberger, G. Zachmann, L. Raghupathi, A. Fuhrmann, M.-P. Cani, F. Faure, N. Magnenat-Thalmann, W. Strasser, and P. Volino. Collision detection for deformable objects. Computer Graphics forum, 24(1):61–81, mar 2005. C.-H. Ho, C. Basdogan, and M. A. Srinivasan. Ray-based haptic rendering:Force and torque interactions between a line probe and 3d objects in virtual environments. I. J. Robotic Res., 19(7):668– 683, 2000. Maciel A, De S. An efficient dynamic point TM algorithm for line-based collision detection in real time virtual environments involving haptic. In Computer Animation and Virtual Worlds, volume 19(2), pages 151-163. Misra, S., Ramesh K. T., Okamura A. M., "Modeling of Tool-Tissue Interactions for Computer-Based Surgical Simulation: A Literature Review" Presence 17(5), 463-491, 2008. J. D. Cohen, M. C. Lin, D. Manocha, and M. K. Ponamgi. ICOLLIDE: An interactive and exact collision detection system for large-scale environments. In Proceedings of the 1995 Symposium on Interactive 3D Graphics, pages 189–196, 1995. M. Müller, B. Heidelberger, M. Hennix, and J. Ratcliff, Position Based Dynamics, 3rd Workshop in Virtual Reality Interaction and Physical Simulation, VRIPHYS, 2006.

560


Collision and Containment Detection between Biomechanically Based Eye Muscle Volumes a

Graciela SANTANA SOSAa and Thomas KALTOFEN a,1 Research Unit Medical-Informatics, RISC Software GmbH, Hagenberg, Austria

Abstract. Collision and containment detection between three-dimensional objects is a common requirement in simulation systems. However, few solutions exist when exclusively working with deformable bodies. In our ophthalmologic diagnostic software system, the extraocular eye muscles are represented by surface models, which have been reconstructed from magnetic resonance images. Those models are projected onto the muscle paths calculated by the system’s biomechanical model. Due to this projection collisions occur. For their detection, three approaches have been implemented, which we present in this paper: one based on image-space techniques using OpenGL, one based on the Bullet physics library and one using an optimized space-array data structure together with software rendering. Finally, an outlook on a possible response to the detected collisions is given. Keywords. Collision detection, containment detection, ophthalmology, biomechanical model, deformable surface models, 3D muscle, software system

Introduction The main goal of the work described in this paper is the implementation of a collision and containment detection engine for SEE++, an ophthalmologic diagnostic tool developed within the research project SEE-KID (Software Engineering Environment for Knowledge-based Interactive eye motility Diagnostics, www.see-kid.at) [1]. The SEE++ software system implements a biomechanical model of the human eye, with the possibilities of simulating pathologies in the field of strabismus and performing a corrective surgery with a "virtual patient" so the results and effects of this surgery can be approximated. In SEE++, the path of each extraocular eye muscle is defined by several geometrical properties: the muscle's insertion on the surface of the globe (eye ball), the point of tangency, which is the last area where the muscle touches the globe, the pulley and the anatomical origin in the back of the orbit. The three-dimensional (3D) location data of insertions, pulleys and anatomical origins of all muscles are taken from averaged statistical data measured by Volkmann [2] and Miller [3]. Since the insertion is fixed to the globe and the anatomical origin is fixed to the bony orbit, eye muscles cannot move freely. Moreover, the pulley, an anatomical structure, which stabilizes the 1

Corresponding Author: Thomas Kaltofen, RISC Software GmbH, Research Unit Medical-Informatics, Softwarepark 35, 4232 Hagenberg, Austria; E-mail: [email protected].

G. Santana Sosa and T. Kaltofen / Collision and Containment Detection

561

muscle path in the area behind the globe, additionally restricts a muscle's freedom of movement. Apart from these properties, the path of a muscle also depends on the biomechanical model's influence on the behavior of the muscle in different gaze positions. Several approaches exist for modeling the muscle paths, especially regarding point of tangency and pulley, as described in [4]. SEE++ uses the approach of a static as well as an active pulley model based on the active pulley hypothesis [5]. For modeling the 3D volumes of the eye muscles, each muscle was reconstructed individually from coronal magnetic resonance (MR) images [6] for specific gaze positions. The resulting 3D surface models were projected onto the previously described muscle paths. For gaze positions, where no 3D models were reconstructed, the available models were interpolated. Due to that projection and interpolation, our problem is that 3D muscle volumes intersect, creating the need for a collision detection module. The main task of such a module is to detect collisions and containment of all eye muscles in all gaze positions in order to provide, in a subsequent step, a proper response to the intersections between the eye muscles. The response to the detected collisions in form of a deformation of the muscle volumes is not covered in this paper. For our problem, the definition of collision detection is more a definition of containment, since we have to find out which area of a muscle is contained by another one. Detection of containment with surface models is usually more difficult than just detecting collisions since algorithms that only perform surface intersection tests do not support checking for containment. In our case, the muscles are deformable bodies colliding already in their initial state. As indicated before, a muscle's shape, force distribution and path are solely defined by the biomechanical model which means that the surfaces of the muscle volumes change in every gaze position as determined by the biomechanical model, which prevents for example the usage of continuous collision detection. Another major requirement is that the detection of containment is computed almost in real time, because one of the future goals is building a response, which influences not only the muscle volumes but the biomechanical model itself. Three different solutions were implemented and tested within the SEE++ system. Details about each solution, including our current implementation, which uses a new approach based on existing techniques, as well as performance benchmarks are presented in this paper. The benchmarks were performed on an Intel Core i7 @ 2.66GHz with 6 GB DDR3 RAM and an ATI Radeon HD 4850 with 512 MB GDDR3 RAM.

Methods and Materials Many previous works address the need for a collision detection engine and propose a solution for rigid [7] [8] and for deformable (soft) bodies [9]. One characteristic of our problem is that there are no objects in motion, but the shape of the objects changes in every gaze position. Therefore, the situation is similar to a discrete collision detection (in contrast to a continuous collision detection [7]), with the difference that no information about the first contact time can be retrieved. The reason is the previously mentioned projection of the muscle volumes onto the muscle paths and the resulting colliding state even in primary position when both eyes are looking straight ahead. Given our need for a fast collision and containment detection, we started implementing a module using common image-space techniques based on GPU (Graphics Processing Unit) computation using OpenGL. The main idea behind this image-based approach is doing a rendering of the scene and analyze whether the vertices forming a muscle are

562


contained in another volume or not by checking their visibility. The result of this image-space approach is a list of all the vertices contained in another volume for all the volumes in the scene, in other words: a volume containment instead of a surface intersection. One drawback of this approach is that the whole scene is treated in pairs of objects while looking for possible containment, which means that all volumes in the scene have to be checked against all other volumes. Obviously, this slows down the collision detection process if the number of objects is increased. The different steps of the algorithm are as follows: 1.

For each object in the scene, an Oriented Bounding Box (OBB) structure [10] is built, which completely contains it. 2. The Separating Axis Theorem (SAT) test [10] is carried out between all boxes of all objects, except between boxes belonging to the same object (no self collision supported). 3. The OBBs that were found colliding in step 2 and all the vertices they contain are marked as possibly contained. 4. A pair of objects (in our case, a pair of muscles) is selected from the scene for the collision detection. 5. The surface of one of the muscles is rendered into the color frame buffer using the background color. This makes the muscle invisible in the color frame buffer, but not in the depth buffer (with depth testing enabled). 6. A color value is calculated for each of the vertices in the other muscle, which allows to uniquely identify each vertex in the color frame buffer. 7. All those vertices are rendered with different colors in the color frame buffer with depth testing enabled. 8. The information of the rendering is retrieved by reading the color frame buffer and getting all the pixel data as an array. 9. All the pixels in the array are analyzed. If an RGB (red, green, blue) color value different from the background color is found, the color represents the ID of the corresponding vertex. Since the vertex is visible, it is marked as not being contained. 10. The invisible vertices have to be further analyzed, because they can either be inside the other volume or behind it. 11. Since containment of only one muscle was detected so far (the muscle which vertices have been rendered with different colors), steps 5 to 10 have to be repeated with the other muscle's vertices being rendered with different colors. 12. Steps 5 to 11 are reiterated with the next pair of muscles. To solve the problem stated in step 10, an Axis Aligned Bounding Box (AABB) containing the two selected muscles was calculated for defining the OpenGL viewing frustum for rendering. The viewing frustum was then rotated several times and steps 5 to 11 were carried out again for each pair of muscles so more information about the scene could be retrieved. The timings for the collision and containment detection of six muscles in primary position comprising 44.500 triangles defined by 30.000 vertices with different resolutions of the color frame buffer can be seen in Table 1. However, with this approach there were still vertices wrongly detected as being contained, because even with very high resolutions of the color frame buffer vertices lying extremely close to each other were always drawn to the same location making one of them invisible.


563

Table 1. Timings for the collision and containment detection with the image-space approach (44.500 triangles, 30.000 vertices). Resolution (pixels) 100 x 100 200 x 200 250 x 250

Depth Buffer (bits) 24 24 24

Time (milliseconds) 766 1837 2682

Therefore, the vertices never visible during rendering were tested with the standard OpenGL picking test [11]. For ensuring if the vertices were really contained or not, the quads of the other muscle were determined that were closest to the vertex the picking test was performed with. The normal vectors of those quads were then used to determine whether the vertex was inside or outside the other volume. The main drawback of the described algorithm is that, for every picking test, parts of the other volume have to be rendered. The result is a very time consuming computation, even if done on the GPU, especially when the rendering steps do not detect enough visible vertices (a large amount of vertices remains for the picking test), which makes the approach not suitable for fast collision and containment detection. While working on our image-space approach, a version of the physics library Bullet [12] supporting collision and containment detection between deformable objects was released (version 2.75). Thus, the collision detection for deformable bodies was analyzed, because it supported concave objects (one of our requirements) due to the convex decomposition it does by using clustering and then applying the GilbertJohnson-Keerthi distance (GJK) algorithm [13]. In order to achieve the required precision, Bullet had to be configured to use one cluster per triangle [12] and since the muscle paths were defined by the biomechanical model as previously explained, those clusters had to be updated every time the paths changed. The time needed for collision and containment detection with Bullet (including the updating of the clusters) for six muscles in primary position comprising 44.500 triangles defined by 30.000 vertices is 134 milliseconds. Although Bullet performs way better than our image-space approach regarding speed, results were still not satisfying. Based on the ideas of our image-space approach and Bullet, we tried to combine the advantages of both to develop a new approach. One of the main disadvantages of the image-space approach was that the depth buffer in OpenGL only stored information about the object closest to the near clipping plane [11] and no information about the depth values of objects behind it. This also was the reason why the scene had to be rendered from different angles and why our algorithm could only be applied to a pair of muscles at a time. In order to overcome these limitations, a data structure similar to a combination of OpenGL's color frame buffer and depth buffer was defined, which allowed storing the depth information of all objects in the scene at the same time with just one rendering step. However, such a structure could not be used when rendering with OpenGL and therefore, we decided to switch from rendering on the GPU to software rendering [14]. Our data structure can be seen as a 3D array representing the world space (space-array), where the information stored in every cell is not a color or depth value (like in OpenGL) but a reference to a complex data structure (PositionInfo). This structure allows, on the one hand, to keep track of more than one object per array cell (position in space) and, on the other hand, to store more information about the primitive (quad, triangle) which the rendered vertex belongs to, such as the normal vector of the primitive. By directly storing the normal vectors of all rendered vertices in the cells of the array, it is now easy to determine which vertices are contained in other

564


volumes by simply carrying out a fast picking test similar to the OpenGL picking test (like explained before). In order to be able to properly detect containment by doing such a test, one requirement is that all analyzed volumes must have a closed surface. The workflow is the following: 1. 2. 3. 4.

5.

6.

7.

For each object in the scene, an OBB structure is built, which completely contains it. The SAT test is carried out between all boxes of all objects, except between boxes belonging to the same object (no self collision supported). With the OBB structure, an AABB covering all muscles is built and used for the projection of the real vertices of the muscles into the 3D array. The surface of the objects, defined by quads, is triangulated. Every triangle, which is contained in one of the OBBs detected to be colliding in step 2, is rendered by software rendering. The position of the rendered vertices in space correlates with their actual location in the 3D array. In every position of the 3D array in which a vertex was rendered, a reference to the PositionInfo data structure is stored. Moreover, the data structure is filled with information about the rendered vertex such as the normal vector of the primitive the vertex belongs to. While performing the rendering, it may happen that objects intersect and therefore, are rendered into the same position in space (in the 3D array). Whenever that happens, we have detected a collision between two primitives. In order to detect vertices being contained in another volume, the depth test is carried out for all relevant vertices along the Z-axis of the 3D array. The vertices suspected to be contained are tested the same way as explained in the image-space approach's picking test (standard OpenGL picking test) but without any additional rendering involved.

After carrying out all described steps, collisions and containment of all objects in the scene have been detected and no further processing is required. The timings for the collision and containment detection of six muscles in primary position comprising 44.500 triangles defined by 30.000 vertices with different resolutions of the 3D array can be seen in Table 2. Benchmarks in other eye positions with more contained vertices than in primary position (where 3,7 % of the vertices are detected as being contained) only show logarithmic growth of the calculation time.

Results We have implemented a new collision and containment detection algorithm by combining existing techniques, which can be seen as a mixture of voxeling, 3D rendering and spatial hashing [15]. The main advantage of the space-array approach presented in this paper is that after just one rendering into a special 3D array, all information needed for the detection of collisions and containment is available. This results in a very fast collision detection for convex and concave objects with no previous decomposition. Figure 1 shows the detected contained vertices after applying the space-array approach to a left eye with all six muscles. One parameter that has to be chosen with care is the resolution of the 3D array into which the rendering is done. This parameter has a direct influence on the balance between accuracy and performance.


565

Table 2. Timings for the collision and containment detection with the space-array approach (44.500 triangles, 30.000 vertices). Resolution (array cells) 100 x 100 200 x 200 250 x 250

Depth Buffer (array cells) 50 100 100

Time (milliseconds) 28 41 45

The algorithm of the space-array approach has proven to be very fast and stable and it fulfills all the requirements presented before. Figure 2 shows a direct comparison between the timings of the different approaches with the data taken from Table 1 and Table 2. In case of Bullet, the accuracy of the collision and containment detection is only comparable when having one triangle per cluster. Therefore, equal timing for Bullet is shown in case of all three resolutions in Figure 2. Although even the lowest resolution chosen for the performance benchmarks still provides full accuracy with our volumes, it is recommended to increase the resolution of the 3D array if the vertices of any volume in the scene are not equally distributed throughout the volume. Due to the way the data structure is built, the space-array approach currently has a limitation of 64 objects (muscles) per collision and containment detection and 1.666.425 vertices per object. These limitations can be overcome by reorganizing the way each vertex is identified in the 3D array.

Conclusion Our new approach for collision and containment detection presented in this paper uses a combination of existing technologies and is particularly suitable when exclusively dealing with deformable bodies. The algorithm fulfills all our previously described requirements and its main advantage, compared to other approaches, is that there is no need for creating a hierarchical structure for the objects in the scene. Consequently, there is no need for preprocessing the objects and no time consuming updates of the hierarchical structure are required.

Figure 1. Top view (A) and front view (B) of a left eye with the collision areas marked.

566


Figure 2. Time comparison between the three approaches in logarithmic scale.

The future work will concentrate on the implementation of a response to the detected collisions. The response will deform the surface of the muscle volumes depending on the amount and location of interpenetration, which can be easily determined since all contained vertices are known. Moreover, in a subsequent step the incorporation of the response into the biomechanical model itself is also planned, meaning that the muscle paths calculated by the model will be influenced by the collision detection and vice versa.

References [1]

[2] [3] [4] [5] [6]

[7] [8] [9]

[10] [11] [12] [13] [14] [15]

Buchberger M, Kaltofen T, Priglinger S, Hörantner R. Construction and application of an objectoriented computer model for simulating ocular positioning defects. Spektrum Augenheilkd, 17(4):151157, 2003. Volkmann AW. Zur Mechanik der Augenmuskeln. Berichte Sächsische Gesellschaft der Wissenschaften, Mathematisch-physikalische Klasse, 1869. Clark RA, Miller JM, Demer JL. Threedimensional location of human rectus pulleys by path inflections in secondary gaze positions. Invest Ophthalmol Vis Sci, 41(12):3787-97, 2000. Miller JM. Understanding and misunderstanding extraocular muscle pulleys. J Vis, 7(11):10-15, 2007. Kono R, Clark RA, Demer JL. Active pulleys: magnetic resonance imaging of rectus muscle paths in tertiary gazes. Invest Ophthalmol Vis Sci, 43(7):2179-88, 2002. Buchberger M, Kaltofen T. Ophthalmologic diagnostic tool using MR images for biomechanicallybased muscle volume deformation. In Proceedings of SPIE, Vol. 5032, eds. Sonka M, Fitzpatrick M, pages 60-71, 2003. Redon S, Kheddar A, Coquillart S. Fast Continuous Collision Detection between Rigid Bodies. Proc. of Eurographics (Computer Graphics Forum), 21(3):279-288, 2002. García-Alonso A, Serrano N, Flaquer J. Solving the Collision Detection Problem. IEEE Comput Graph Appl, 14(3):36-43, 1994. Teschner M, Kimmerle S, Heidelberger B, Zachmann G, Raghupathi L, Fuhrmann A, Cani MP, Faure F, Magnenat-Thalmann N, Strasser W, Volino P. Collision Detection for Deformable Objects. Eurographics State-of-the-Art Report (EG-STAR), Eurographics Association, pages 119-139. 2004. Gottschalk S, Lin MC, Manocha D. OBBTree: A Hierarchical Structure for Rapid Interference Detection. Comput Graphics, 30:171-180, 1996. Woo M, Neider J, Davis T, Shreiner D. OpenGL Programming Guide, Version 1.2. Addison-Wesley Longman Publishing, Boston, USA, 1999. Bullet Physics Library. http://www.bulletphysics.org. Gilbert EG, Johnson DW, Keerthi SS. A fast procedure for computing the distance between objects in three-dimensional space. IEEE J Robotic Autom, 4(2):193-203, 1988. Kaufman A, Shimony E. 3D scan-conversion algorithms for voxel-based graphics. Proceedings of the 1986 workshop on Interactive 3D graphics, pages 45-75, ACM, New York, USA, 1987. Turk G. Interactive Collision Detection for Molecular Graphics. Technical report, Chapel Hill, USA, 1990.


567

Visualization of 3D Volumetric Lung Dynamics for Real-Time External Beam Lung Radiotherapy Anand P SANTHANAMa , Harini NEELAKKANTANb, Yugang MINc, Nicolene PAPPd, Akash BHARGAVAd, Kevin ERHARTe, Xiang LONGe, Rebecca MITCHELLf, Eduardo DIVOe, Alain KASSABe, Olusegun ILEGBUSIe, Bari H RUDDYf, Jannick P. ROLLANDg, Sanford L. MEEKSb and Patrick A. KUPELIANa a Department of Radiation Oncology, University of California, Los Angeles 200 UCLA Medical Plaza, Suite B265 Los Angeles, CA 90095 b Department of Radiation Oncology, M.D. Anderson Cancer Center Orlando 1400 S. Orange Ave Orlando FL 32806 c School of Computer Science, University of Central Florida d College of Optics/FPCE, University of Central Florida e Mechanical, Material and Aerospace Engineering, University of Central Florida f Health and Public Affairs, University of Central Florida 4000 Central Florida Blvd Orlando FL 32826 g Institute of Optics, University of Rochester 275 Hutchison Rd. Rochester, NY 14627-0186

Abstract. This paper reports on the usage of physics-based 3D volumetric lung dynamic models for visualizing and monitoring the radiation dose deposited on the lung of a human subject during lung radiotherapy. The dynamic model of each subject is computed from a 4D Computed Tomography (4DCT) imaging acquired before the treatment. The 3D lung deformation and the radiation dose deposited are computed using Graphics Processing Units (GPU). Additionally, using the dynamic lung model, the airflow inside the lungs during the treatment is also investigated. Results show the radiation dose deposited on the lung tumor as well as the surrounding tissues, the combination of which is patient-specific and varies from one treatment fraction to another. Keywords. Lung Radiotherapy, 3D Lung Dynamics, Computational Fluid Dynamics.

1. Introduction Medical simulation and visualization is a critical component in planning procedural interventions and predicting patient outcomes [1,2]. One of the key application domains is the visualization of lung radiotherapy for patients with Non-Small Cell Lung Cancer (NSCLC) [3]. Lung anatomy moves during breathing, which can reduce the amount of radiation dose deposited on the lung during radiotherapy. Such dose reduction lowers the overall treatment efficacy. The ability to predict the lung motion during radiotherapy

568

A.P. Santhanam et al. / Visualization of 3D Volumetric Lung Dynamics

coupled with the ability to calculate the radiation dose in real-time facilitates monitoring and visualizing the actual lung radiation delivery and investigating treatment optimizations that account for the lung motion. The success of such medical simulations is evidenced by the fact that over a third of all medical schools in the United States augment their teaching curricula using patient simulators [4]. Physics-based surface lung models have been previously shown to be effective for monitoring the radiation dose delivered on the tumor [3]. Such physics base models address the issue of predicting the lung tumor motion during the radiotherapy treatment. Results show variations in the dose delivered to the tumor when the tumor motion is taken into account. In this paper, we describe a research effort that focuses on developing methods for creating patient-specific physics-based volumetric lung models. Such volumetric lung models will account for the radiation delivered not only to the tumor but also to the surrounding lung tissues. We hypothesize that GPU-based deformation methods and dose computations are necessary to adequately visualize, monitor, model, and characterize the treatment efficacy of the lung radiotherapy for a given NSCLC patient. The reported results summarize the volumetric model development and the treatment simulation and its real-time nature. Variations observed for a patient from one treatment fraction to another and also from the actual planned treatment are discussed.

2. Proposed Method for 3D Lung Deformation In this section we discuss the methodology adopted for the dynamic simulation of 3D volumetric lungs deformation. 2.1 Patient Data Acquisition The patient imaging data used for the proposed framework includes the 4D Computed Tomography (4DCT) imaging during the diagnostic stage. Additionally, the flowvolume breathing signal is also collected during the imaging stage. The patient’s lung compliance is measured using Impulse Oscillometry (IOS) measurements. IOS is a relatively new noninvasive method for measurement of respiratory impedance, (i.e. airway resistance and reactance) at different oscillation frequencies. Oscillometry uses external forcing signals, which can be mono- or multi-frequency, and applied either continuously or in a time-discrete manner [5]. For the current study, the forced oscillations are performed using a MasterScreenIOS device [6]. A volume-controlled loudspeaker generates a multi-frequency impulse

(a)

(b)

(c)

(d)

Figure 1. The YM estimated for each lung voxel at 100 % ((a) and (b)) and 30% ((c) and (d)) inhalation is shown, for left and right lungs 4DCT dataset.


569

signal lasting 45 milliseconds. Each impulse produces a volume shift consisting of approximately 40 ml either in the inspiratory or the expiratory direction. The power spectrum of the impulse covers a range from above 0 to 100 Hz. The loudspeaker unit is coupled with a tube of 35 cm length and 3 cm diameter to the measuring head based on a Y-connector. The Y-connector is terminated with a mesh screen resistor of 0.1 kPa/(l/s). On the front side, a Lilly-type Pneumotachograph [6] provides differential pressure to sense flow [5]. The patient is instructed to sit upright during tidal volume breathing with a solid lip seal around the mouthpiece flange. Cheeks are held with light pressure using the patient’s hands during the breathing task. The measured impedance is represented for the range of frequencies, which represents the internal airway levels. 2.2 Estimating the YM Value of Each Lung Voxel The YM value of each voxel is estimated for known values of airflow and the volumetric lung displacement estimated from the 4DCT lung dataset using a modified optical flow approach [7]. A Hyper-spherical Harmonic (HSH) transformation is employed to compute the deformation operator. The HSH coordinated transformation method facilitates accounting for the heterogeneity of the deformation operator using a finite number of frequency coefficients. Spirometry measurements are used to provide values for the airflow inside the lung. Using a 3D optical flow-based method, the 3D volumetric displacement of the left and right lungs, which represents the local anatomy and the deformation of a human subject, was estimated from the 4DCT dataset. Results from an implementation of the method show the estimation of the deformation operator for the left and the right lungs of a human subject with NSCLC [8]. Figure 1 represents the color-coded representation of the YM values for the left and right lungs at 30% and 100% tidal inhalation. The ranges of values are represented by 0-400 Pa (black), 400800 Pa (red), 800-1200 Pa (yellow), 1200-1600 Pa (green) and 1600-2000 Pa (white). It can be seen that differences in the elasticity can be observed in different regions of the lung as well as from one sub-anatomy to another. 2.3 Tracheobronchial Fluid Dynamics Simulation The lungs are modeled as poro-elastic media, where the flow field satisfies Darcy’s law and the elasticity field is solved using the non-homogeneous Navier’s equation. The tissue properties, i.e., the porosity, permeability, and shear modulus, are considered non-homogenous as they vary throughout the lung as illustrated by the elasticity. In simulating the fluid flow inside the lungs, two different approaches are considered. The two approaches cross-verify each other’s simulation. In the first approach, a meshless modeling technique is investigated in order to reduce the dependence of the solution accuracy on the discretization of the lung model itself. For stability of the solution, radial basis functions are coupled with moving least squares [9]. For the second approach, the 3D lung anatomy was discretized into finite elements for computation using commercial software ADINA. The FEM model considered for the airflow within the lung to occur through a poro-elastic medium and the structural dynamics were resolved using a flow-structure interaction model. The tissue properties in this case are estimated from the patient’s 4DCT as discussed in section 2.2. The spatial deformation is predicted over a complete breathing cycle. The volumetric lung deformation obtained using this simulation is then employed for refining the YM value estimated for each lung voxel.

570


2.4 3D Volumetric Lung Dynamics Simulation A GPU is used for computing the 3D anatomy deformation using the 3D lung anatomy taken during the diagnostic stage and the YM parameter associated with each voxel (estimated in section 2.2) coupled with the spirometry signal that gives the air volume of the lung. A 3D convolution is performed between the YM elasticity distribution inside the lung and the applied force computed using the spirometry air volume and the airway compliance estimated from the IOS measurements. The result of this convolution provides the displacement for each voxel during the breathing. The 3Dlung anatomy is deformed as follows: The deformed 3D lung volume is first initialized in such a way that voxel positions which represent the lung in the un-deformed lung volume are set to the Hounsfield number representing the air. This initialization is done in order to make sure that the voxel, which will not have the deformed lung anatomy, will be filled with air. For each voxel in the un-deformed 3D lung anatomy, the position of the voxel during the deformation is first computed. The Hounsfield number is then copied into that location [10]. Results show that using GTX 480, the volumetric lung dynamics is achieved at the rate of once every 60 milliseconds. 2.5 3D GPU Based Dose Calculation The 3D dose convolution is performed using a 3D separable dose convolution approach. In this method, a 3D dose convolution is split into row-wise, column-wise and planewise 1D convolutions. A 10 cm3 lung data has been taken from a CT dataset with 128 slices and a 3D data with 1283 voxels is created. The 1D convolutions are computed for each voxel using its 127 neighbors along the row, column, and plane respectively. For real-time purposes, we employed a shared-memory based data access for performing the convolutions. Specifically, the 3DCT data representing the patient anatomy, the displacement vector associated with each voxel, and the voxels representing the 3D dose accumulated on each image voxel are copied into the shared memory of the GPU. The row and column 1D convolutions for each voxel initiate bulk data transfer between the GPU processor and the shared memory thereby increasing the memory bandwidth usage. However, the hardware architecture of the GPU does not directly allow initiating bulk data transfer from the processor. Thus in the proposed method, we employ an optimization where a 3D matrix transpose is performed after the row and column 1D convolution. Such a matrix transpose re-arranges the voxels from neighboring planes to be placed next each other thereby facilitating bulk data transfer. An initial flux of 6 MV photons is assumed and a 3D dose is computed for each lung voxel. To calculate first scattering dose components, the 3D dose is convolved with a kernel of size 33. The kernel is computed using the image voxels surrounding each point to be convolved [3,10,11]. Results show that the dose calculation is achieved at the fastest rate of 130 milliseconds using the Nvidia GTX 480 graphics card. 3D GPU based volume visualization is employed for visualizing the 3D lung volume together with the 3D lung radiation dose accumulated on it. The method is


571

implemented as follows. The intensity of each voxel is used to determine the opaqueness of each voxel. The 3D lung anatomy is represented as a grey scale image, while the 3D dose is represented in a set of colors (blue 0-85% dose, green 85-90% dose, red 90-95% dose and white 95%-100% dose).

2.6 OpenCV Based Spirometry Tracking Clinical spirometry systems that track the patient’s flow volume perform tracking at 20-30 Hz. For our experiment, we use PowerLab 6.0 for measuring the spirometry in real-time. However to incorporate the flow volume signal into the proposed framework, we developed an OpenCV- based spirometry-tracking system. An external HD camera is mounted and calibrated to track the spirometry display screen. The tracking system tracked the flow volume signal displayed on the spirometry equipment and communicates with the proposed framework using inter-process communication techniques. The advantage of using such a spirometry tracking is that the radiotherapymonitoring framework is ensured to work synchronously with the spirometry signal. Based on the computational capability of the simulation system, the flow volume signal is obtained from the spirometry tracking system and the volumetric lung deformation is performed. Additionally, the spirometry tracking enables real-time data acquisition by keeping the spirometry interface independent of the deformation model framework. The interface works as follows: The spirometry display system displays the flow signal as a graph with the air volume along the vertical axis and the time along the horizontal axis. We ensure that the signal displayed by the spirometry has a unique color in the screen. The user sets the breathing time lag, which is the time lag after which the patient’s breathing is reflected in the visualization. This is done by appropriately setting the monitoring vertical axis in the HD camera image. Once the patient starts breathing, the HD camera image is acquired in real-time and the unique color representing the flow signal is tracked along the vertical axis representing the breathing time lag. 2.7 The Integrated Framework We now describe the integrated framework for visualizing the radiation therapy delivery. For a given patient, a 4DCT image of the lung is acquired during the diagnostic stage as discussed in section 2.1. The 3D deformable model is developed by estimating the YM of each lung voxel as discussed in section 2.2. The YM estimation is then refined using the simulation of the tracheobronchial airflow modeling and the overall lung deformation. Finally, during the treatment fraction, the patient breathing is acquired in real-time using the spirometry tracking system as discussed in section 2.6. The 3D volumetric lung is deformed as discussed in section 2.4 and the dose deposited in the volumetric lung is visualized as discussed in section 2.5.

572


Figure 2. The real-time radiation dose delivered for a patient lung with (a) no motion, (b) sinusoidal tumor motion, (c) breathing curve obtained on day 1, (d) day 2, (e) day 3, and (f) day 4.

(a) (b) (c) Figure 3. The DVH of the patient for the different treatment fractions with sinusoidal breathing volume considered. The difference among the DVH is significant.

(a) (b) (c) Figure 4. The DVH of the patient for the different treatment fractions with patient-specific breathing volume considered. The difference among the DVH is significant.

3. Results We now present the patient results obtained using the proposed framework. Figure 2 represents the dose calculated on the 3D deforming volumetric lung for a NSCLC patients. It can be seen that at each fraction the dose deposited on the tumor and the surrounding tissues vary from one another. Figure 2a represents the 3D dose deposited on the volumetric lung without taking into account the lung motion. When the volumetric lung motion is taken into account using sinusoidal breathing (Figure 2b), it can be seen that the radiation dose deposited on the lung varies from Figure 3a, which


573

shows the changes in the treatment efficacy from the actual treatment plan. When the actual patient breathing is considered, Figure 2c-f shows the variations in the radiation dose delivered during every treatment fraction. The difference in the dose accumulation between the sinusoidal breathing and subject-specific breathing can be observed. Such variations can be correlated with the treatment outcomes to further optimize the treatment plan. Figure 3-4 shows the Dose Volume Histogram (DVH) for three different fractions and using two different breathing variations of the same patient. When the lung motion is modeled using a sinusoidal breathing volume (Figure 3), the change in the dose delivered to the tumor changes from one day to another. Such changes in the DVH are attributed to the changes in the breathing lung model developed from each of the 4DCT. When the lung motion is modeled using the patientspecific breathing collected using spirometry (Figure 4), the DVHs significantly varied both from one day to another and from using sinusoidal motion analysis. Such variations further quantify the need for accurately accounting for the patient lung volume changes and the need for real-time adaptive lung radiotherapy, where the treatment is monitored and modified in real-time. To conclude, the usage of physicsbased lung deformation and real-time dose calculation can be used to monitor the subject specific dose delivery and make critical changes to the treatment plan.

4. Acknowledgement This work is funded by the James & Esther King Foundation.

5. References Nye, L.S., The minds' eye, Biochemistry and Molecular Biology Education. 32 (2) 123-131, (2004). Robb, R.A., Three-dimensional visualization in medicine and Biology, Handbook of medical Imaging: Processing and Analysis, I.N. Bankman, Editor Academic Press: San Diego,CA, 2000. [3] Santhanam, A.P., T. Willoughby, I. Kaya, A. Shah, S.L. Meeks, J.P.Rolland, and P. Kupelian, A Display Framework for Visualizing Real-time 3D Lung Tumor Radiotherapy, IEEE Journal of Display Technology “Special issue on Medical Displays” 4 (4) 473-482, (2008). [4] Good, M.I., Patient simulation for training basic and advanced clinical skills, Medical Education. 37 14, (2003). [5] Gerhard, K., Thorsten, B, Ute, R.,Alwin, G. Hans-Juergen, S. and Klaus, P, Measurement of respiratory impedance by Impulse Oscillometry – effects of endotracheal tubes, Research in Experimental Medicine, 200 17-26, (2000). [6] H.J. Smith, H.J., Reinhold, P. and Goldman, M.D., Forced oscillation technique and impulse oscillometry, European Respiratory Monographs, 31 72–105, (2005). [7] Min, Y., N. Papp, A. Shah, S. Meeks, and A.P. Santhanam. 2010. 4D-CT lung registration using anatomy based multi-level multi-resolution optical flow analysis and thin plate splines. Physics in Medicine and Biology (in review). [8] Santhanam, A., Y. Min, S. Mudur, E. Divo, A. Kassab, B. H. Ruddy, J. Rolland and P. Kupelian, An inverse hyper-spherical harmonics-based formulation for reconstructing 3D volumetric lung deformations. Comptes Rendus Mechanique 338 (7-8) 461-473, (2010). [9] V. Huayamave, A. Vidal, A. Kassab, E. Divo, A. Santhanam, and P. Kupelian, A meshless approach to solve the fluid poro-elastic interaction problem between the tracheo-bronchial tree and the lungs, Int. Conf. on Computational Methods for Coupled Problems in Science and Engineering, Ischia Island, Italy, June 8-10, 2009. [10] Min, Y., A. Santhanam, A., Y. Min, H. Neelakkantan, B.H. Ruddy, S. Meeks, P. Kupelian, A GPU based framework for modeling real-time 3D lung tumor conformal dosimetry with subject-specific lung tumor motion, Physics in Medicine and Biology 55 5137, (2010). [11] Santhanam, A.P., T. Willoughby, S.L.Meeks, and P. Kupelian, Modeling simulation and visualization of 3D lung conformal dosimetry, Physics in Medicine and Biology 54 6165-6180, (2009). [1] [2]

574


Laser Surgery Simulation Platform: Toward Full-Procedure Training and Rehearsal for Benign Prostatic Hyperplasia (BPH) Therapy Yunhe SHEN a,b,1, Vamsi KONCHADA a,c, Nan ZHANG a,b, Saurabh JAIN a,c, Xiangmin ZHOU a,d, Daniel BURKE a,b, Carson WONG e, Culley CARSON f, Claus ROEHRBORN g and Robert SWEET a,b a Center for Research in Education and Simulation Technologies, Univ. of Minnesota b Department of Urologic Surgery, University of Minnesota c Department of Computer Science and Engineering, University of Minnesota d Department of Mechanical Engineering, University of Minnesota e Department of Urology, University of Oklahoma Health Sciences Center f Department of Surgery, University of North Carolina g Department of Urology, University of Texas Southwestern Medical Center at Dallas

Abstract. Recently, photo-selective vaporization of the prostate (PVP) has been a popular alternative to the standard electrocautery - transurethral resection of prostate (TURP). Here we introduce a new training system for practicing the laser therapy by using a virtual reality (VR) simulator. To interactively and realistically simulate PVP on a virtual organ with an order of a quarter million elements, a few novel and practical solutions have been applied to handle the challenges in modeling tissue ablation, contact/collision and deformation; endoscopic instruments tracking, haptic rendering and a web/database curriculum management module are integrated into the system. Over 40 urologists and surgical experts have been invited nationally and participated in the system verification. Keywords. BPH, PVP, surgery simulation, virtual reality.

Introduction Benign prostatic hyperplasia [1] or prostate enlargement commonly occurs to men as an aging problem, with a direct symptom of lower urinary tract blockage or occlusion caused by the enlargement, as shown in Figure 1(a). Since 1990’s, a series of minimally invasive therapies and transurethral endoscopic surgeries have been routinely applied to BPH patients in clinics. These treatments generally deliver external energies in terms of electrical current (in TURP), radiofrequency or microwave, thermal current or laser beam (in PVP) into prostate for tissue resection or ablation.

1

Corresponding Author: Yunhe Shen, University of Minnesota, 420 Delaware St SE, Mayo Building A584, Minneapolis, MN 55455; E-mail: [email protected].

575

Y. Shen et al. / Laser Surgery Simulation Platform

Photo-selective vaporization of the prostate is an endoscopic laser surgery. In PVP, laser energy is delivered to a patient’s prostate gland through a fiberoptic glass/probe inside a cystoscope, as shown in Figure 1(b) and 1(c). After inserting the scope through urethra into the prostate or bladder filled with irrigation fluid, a urologist can translate and rotate the scope and the probe to precisely aim the laser beam at certain local treatment area, where a vaporization mechanism can be developed by sweeping the high energy laser beam along the local tissue surface while maintaining appropriate distance between the probe and the tissue. By repeating this operation pattern inside the prostate, most part of the prostate tissue is gradually vaporized and the urinary tract is recovered. Digital video is captured inside the prostate by a cystoscopic camera and displayed in the operating room as visual clue to the surgeon during this operation, as shown in Figure 2(a). The laser used in PVP usually seals bleeding vessels as it ablates the prostate; occasionally as few bleeders do occur, the laser power can be lowered in order to coagulate the bleeders.

1.8mm

(a)

(b)

(c)

Figure 1. (a) BPH [15]. (b) PVP therapy. (c) Cystoscope tip, endoscopic lenses, and laser fiber [8].

The PVP example we choose to simulate is a GreenLight™ laser surgery system [2], which uses a side-firing fiber/probe emitting a 532 nm wavelength lithium triborate (LBO) laser with maximum power of 120W in HPS version and 180W in XPS version. A few technologies and algorithms in the related fields are worth mentioning for the research and development of the real time PVP simulator. Visualization methods such as isosurface extraction by marching tetrahedrons [3] or marching cubes [4], the level set method [5-6], and various physically based modeling approaches are reviewed for deriving the most suitable solutions for this medial VR application. Experimental data of laser-tissue properties [7] provide quantitative estimation in calibrating tissue vaporization rates. The objective of this work is to design and build a VR simulator and provide the most consistent, standardized, modular training curriculum to develop and improve users' skills in performing PVP Therapy, and to achieve rapid clinical proficiency leading to excellent clinical outcomes.

1. Requirements Analysis and Training Modules Design 1.1. Interdisciplinary Research We have formed up an interdisciplinary team nationally including a clinical advisory panel consisting of four key opinion leaders for BPH who collaborate with our engineering group throughout the research and development process from requirements

576


analysis to training curricula development. More than 40 board-certified urologists participated in the verification testing of the training platform at the American Urological Association conference, San Francisco, 2010. The GreenLight™ simulator clinical advisory panel defines desired outcomes, learning objectives and desired metrics which led to the development of desired exercises. These exercises are divided into a full-procedure laser operation on 6 variations of virtual BPH models, as well as several subtasks specifically designed by task deconstruction for part-task training. 1.2. System Framework Design This simulation framework consists of the following major components: • Anatomical models and variations • Volumetric ablation and visualization • Collision detection and contact modeling • Tissue and fiber deformation • Graphical rendering and special effects - bleeding, vaporization, etc. • Motion tracking and haptic rendering • Web/database module for integrated learning management • Subtasks, quizzes and tutoring The main framework and core VR modules are implemented with C++ and objectoriented programming (OOP). Multithreading is used to allow several computationintensive modules run in parallel, whereas hardware acceleration or extensive threading on GPU/CPU cores can be a potential reinforcement for the computation needs in future versions. 1.3. Anatomical Models and Variations We model the anatomy of prostate, urethra and bladder with surface or volumetric meshes generated from medical imaging data, and refine these mesh models in graphical design software, which is also used to model surface textures, materials as well as surgical instruments involved in PVP. Our virtual BPH prostate models cover a mass/volume range from 30 to 95 grams or cm3. The 6 key common clinical variation prostate cases defined by the clinical advisory panel and represented in the curriculum consist of (1) small normal, (2) large median lobe, (3) high median bar, (4) small fibrous, (5) tri-lobar enlargement, and (6) prominent apex. Figure 1(c) shows a close view of the cystoscope tip structure, from which we realize that although a prostate is not considered a large organ, its mesh model does require us to spend a large amount of elements to ensure its resolution and quality, because the scale of this laser-tissue ablation is precisely controlled in an order of a few millimeters at a given time during PVP treatment, and then cystoscopy magnifies it to a full screen size; tissue ablation causes geometrical or topological modifications applicable to the entire prostate volume in the treatment, which requires the modeling to main the level-of-details accordingly.


577

2. Ablation and Visualization Isosurfaces of the ablation are derived from the initial tetrahedral mesh of the virtual prostate model by implicit surface construction. A volumetric ablation method using standard constructive solid geometry (CSG) algorithm is improved with our volumecontrolled CSG approach [10] to maintain the tissue ablation or vaporization rates within the reasonable range estimated by the study in [12] according to the in vitro bovine experimental data [7]. In addition, we have advanced the CSG concept further to a new ablation algorithm, which generates realistic visual effects such as tissue melting or shrinking in PVP treatment, and is applicable to relatively lower resolution models. By using this method, we are able to reduce the tetrahedral elements of a prostate model from more than a million [10] to a quarter million, thus the computation load in deformation and collision handling are substantially relieved.

3. Collision Detection and Contact Modeling A fast and reliable solution for collision or contact handling is critical to the success of PVP simulation, because a cysctosope is tightly and constantly surrounded by largely deformed or stretched prostate tissue; in most time during the treatment, percentage of contact area is also high – both the scope and fiber are deeply operated posterior to the verumontanum of the prostate to avoid damages to the urinary sphincter or the apex zone. “Popping through” or penetration problem was a challenge to an earlier prototype, in a couple of cases when users tried to create a narrow channel by ablation and push the rigid scope against the soft tissue all the way to the prostate boundary. Several analytical bounding shapes are applied to the cystoscope and the fiber probe models, to correct or constrain the movements of those elements that belong to a deformable prostate model but are detected inside or on the surface of a rigid PVP instrument model in the virtual surgery environment. This fast processing is able to address the contact or collision problems at interactive rates. This module also calculates a few PVP metrics such as sweep speed and treatment distance, which is defined as the estimated distance from an emitting point of a laser fiber to the intersection spot where a laser beam meets the surface of a prostate. It detects weather certain predefined anatomical structures such as external urinary sphincter, verumontanum and prostate capsule have been damaged in the treatment, in that case, quantitative results can be calculated and logged in the learning management database.

4. Tissue and Fiber Deformation Deforming the prostate model is another challenging problem to solve for this application. In PVP, a laser beam is arbitrarily manipulated within the volume of a prostate and constantly modifies tissue geometry and topology. Simulating this procedure requires that the deformable model is applicable to and synchronized with the updated prostate ablation model per simulation frame. Concerns in computation size and algorithm stability require a swift and robust deformation model. Starting from our earlier volumetric approach [11], we simplify this module to such a degree that its

578


computation complexity is reduce to minimum yet it handles complicated shapes and presents realistic visual response to users input at local treatment area. The fiber probe is thin and flexible; it may be deflected by or penetrates into soft tissue as being pushed against the inner surface of a prostate. It’s important to enable this fiber motion in PVP simulation, or it would feel quite different from the real operation. We simplify this phenomenon to rigid transformation of the fiber without spending processing power in solving fiber glass deformation.

5. Graphical Rendering and Special Effects Particle systems and texture animations are applied to render bubbles, laser beam, and bleeding in an environment filled with irrigation fluid. Ogre [9] is integrated in the current system for its graphics/GPU rendering features.

6. Motion Tracking and Haptic Rendering Rigid body of cystoscope has 6 Degrees of freedom (DoF) motion; scope camera has 1 DoF independent rotation; fiber has 2 independent DoF (translation in and along the cystoscope plus axial rotation). Real GreenLight™ foot pedals have been connected to the system for 3 control signals - vaporization, coagulation and standby. Some peripheral devices such as motion tracking sensors and hardware assemblies are designed and manufactured by working with two industrial partners. A low-cost Novint Falcon™ [13] force rendering device provides haptic feedback through the cystoscope model, which is a 3-D scan and remodeling of an actual cystoscope. In haptic rendering, resultant Ft is directly calculated from the original deformation or collision force Fd; here we add a center-line enhancement Fc to Ft: (1) Where wd and wc are two scalar values determined by an attenuation function. Our study shows that haptic feedback is important in this VR application. Without being assisted by the haptic guidance existing in real PVP practices, users could lever the cystoscope out of the narrow space or the deformation limits inside the prostate model.

7. Web and Database for Integrated Learning Management A web-based interface and a SQL database connection module have been proposed in our early design phase and now integrated with the real time simulation system. This learning management platform stores and maintains training curriculum including performance metrics, scores and medical knowledge base together with system configurations in the SQL database server. As a powerful multi-institutional validation study tool, this management platform organizes all participants and curricula data at global, local site and local group levels. A web-based interface provides remote users

579


online access to the database server. This unique PVP training platform design can be further extended in the next version to include uploading function allowing certain patient-specific features be built into customized training sessions.

8. Subtasks, Quizzes and Tutoring Similar to the previous task deconstruction approach [14], we divide the skill set in the PVP procedure into several subtasks training laser sweep speed, sweep distance, coagulation and anatomy identification. Other BPH knowledge base or quizzes can also be added in the simulation platform as individual training modules. In a tutoring mode, users will have instant guidance or error warning feedback during self-learning of the PVP procedure on the simulator. In other exercise or skill assessment mode, performance metrics including surgical errors are also logged into the learning management database by the virtual tutor.

9. Results In Figure 2(a), real surgical video captures are shown in the upper row as comparison to the simulation results shown in the lower row. The left column shows the narrow urinary tract being dilated by a cystoscope tip which encapsulates the lenses but is not visible in the view; the middle column shows the scene of laser ablation and part of the fiber/probe extending along and ahead of the cystoscope case; the right column shows the reopening of the urinary track and the fiber optics with laser power off. VR simulation modules are successfully implemented for the simulated PVP procedure. Figure 2(b) shows the integrated system including the VR interface. All processing threads have achieved a refreshing rate above 30 frames per seconds on a mid-range Intel® Core™ i7-860 workstation with Nvidia GTS240 graphic card, as testing the PVP procedure over a set of prostate models with resolutions vary from 220k to 480k tetrahedral elements. This is mainly achieved by algorithm optimization and simplification.

(a)

(b)

Figure 2. (a) PVP surgical video captures and simulation results. (b) The PVP simulator.

580


10. Verification and Validation In 2010 American Urology Association meeting at San Francisco, CA, over 40 urologists tested our prototype PVP BPH simulator. In the recent version we have demonstrated several newly-updated methods in the training system including the new ablation model, the haptic/force-feedback feature and full-motion-tracking interface for the endoscope operation, as well as the integrated learning platform to collect and manage performance data. Two prototype simulators have been running smoothly and sustained 5 day exercise without raising noticeable reliability issues. Encouraged by the valuable feedback from the tests, we are refining the system and developing the next version training platform as well as planning a formal multi-institutional validation study for this VR-based curriculum. Details of the new updates in the functional modules and the results of the validation study will appear in our future report.

Acknowledgement This research and development was supported in part by American Medical Systems®. Visualization facilities supported by Minnesota Supercomputer Institute (MSI) are acknowledged.

References [1] American Urological Association. Guideline on the management of benign prostatic hyperplasia, Chapter 1: Diagnosis and treatment recommendations. J. of Urology, 170(2), pages 530–537, 2003. [2] American Medical Systems. GreenLight system: http://www.greenlighthps.com/lasersystems.html [3] A. Guéziec and R. Hummel. Exploiting triangulated surface extraction using tetrahedral decomposition. IEEE Trans. Visualization and Computer Graphics, 1(4), pages 328-342, 1995. [4] W. Lorensen and H. Cline. Marching cubes: A high resolution 3D surface construction algorithm. Computer Graphics, Vol. 21(4), pages 163-169, July 1987 [5] S.Osher and J. Sethian. Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations, J. Comput. Phys. 79, pages 12–49, 1988. [6] J. Sethian. Level set methods and fast marching methods: Evolving interfaces in computational geometry, fluid mechanics, computer vision, and materials science. Cambridge University Press. 1999. [7] H. Kang, D. Jebens, R. Malek, et. al. Laser vaporization of bovine prostate: A quantitative comparison of potassium-titanyl-phosphate and lithium triborate lasers. J. Urology, 180, pages 2675-2680, 2008. [8] American Medical Systems. GreenLight Standardized Training, CD media, 2009. [9] Ogre rendering engine: http://www.ogre3d.org/ [10] N. Zhang, X. Zhou, Y. Shen and R. Sweet. Volumetric modeling in laser BPH therapy simulation. IEEE Visualization, special issue of IEEE Trans. on Visualization and Computer Graphics, to appear. 2010. [11] Y. Shen, X. Zhou, N. Zhang, K. Tamma, and R. Sweet. Realistic Soft Tissue Deformation Strategies for Real Time Surgery Simulation. Stud. Health Tech. Inform, 132, pages 457-459. 2008. [12] X. Zhou, N. Zhang Y. Shen, et. al. Phenomenologial model of laser-tissue interaction with application to benign prostatic hyperplasia (BPH) simulation. Submitted to MMVR’18. [13] Novint Falcon: http://home.novint.com/products/novint_falcon.php [14] K. Adiyat, R. Beddingfield, T. Holden, Y. Shen, T. Reihsen and R. Sweet, "Task deconstruction facilitates acquisition of TURP skills on a virtual reality trainer," J. Endourology, 23(4), pages 665-668, 2009. [15] National Cancer Institute, AV: CDR462221, 2004. http://visuals.nci.nih.gov/details.cfm?imageid=7137


581

3D Tracking of Surgical Instruments Using a Single Camera for Laparoscopic Surgery Simulation Sangkyun SHIN1, Youngjun KIM, Hyunsoo KWAK, Deukhee LEE and Sehyung PARK Intelligence and Interaction Center, Korea Institute of Science and Technology, Korea

Abstract. Most laparoscopic surgery simulation systems are expensive and complex. To overcome these problems, this study presents a novel threedimensional tracking method for laparoscopic surgical instruments that uses only a single camera and fiducial markers. The proposed method does not require any mechanical parts to measure the three-dimensional positions/orientations of surgical instruments and the opening angle of graspers. We implemented simple and cost-effective hardware using the proposed method and successfully combined it with virtual simulation software for laparoscopic surgery. Keywords. Medical simulation, laparoscopic surgery, computer vision, virtual reality.

Introduction The practice of medical simulation was developed to guarantee patient safety. Medical students or novice surgeons can be efficiently trained using these simulation systems. Compared with patient or animal training environments, medical simulation provides a typical and uniform training condition. One area of potential application is laparoscopic surgery. This type of surgery involves difficulties due to the narrow scope of vision, a lack of perspective, weak contact feelings, and the pivoting movement of the surgical instruments, all of which are in contrast to traditional invasive surgery. Hence, laparoscopic surgery requires systematic and continuous learning of the type provided by virtual surgery simulation. Laparoscopic surgery simulation has been improved by many investigators in many countries using advanced medical technology. However, most systems continue to be associated with problems that arise related to cost-effectiveness. Hence, most medical students scarcely have the opportunity to access training in this type of surgery. To tackle the problem, a 3D tracking technique of surgical instrumentation was recently introduced by means of applied computer vision technology [1-3]. Doignon [1, 2] proposed an algorithm that estimates the 3D pose of the surgical instrument via collinear markers on the instrument. This computer vision-based tracking system has a

1

Corresponding Author: Intelligence and Interaction Center, Korea Institute of Science and Technology(KIST); E-mail: [email protected]

582

S. Shin et al. / 3D Tracking of Surgical Instruments Using a Single Camera

simple and cost-effective structure, as it does not rely on mechanical parts to measure the positions and orientation of the surgical instruments. In the present paper, this earlier method was applied to estimate the 3D positions and orientations of laparoscopic surgical instruments. Furthermore, we expanded the previous method by proposing a novel method of measuring the angle of laparoscopic surgical graspers using a “single” camera. The proposed method is simple and costeffective. We successfully combined the proposed computer vision-based tracking module with a training simulation system for laparoscopic surgery.

1. Method To estimate the pose of the instrument and measure the opening angle of the graspers, a camera unit and four band-type fiducial markers were used.

Figure 1. Photos of the grasper tip (P0, P1, P2: static marker, P3: dynamic marker, upper: opened, lower: closed)

As shown in Figure 1, three markers were static markers; these were used to estimate the instrument’s pose. The other marker was a dynamic marker which measured the opening angle of the grasper. While a user manipulates the graspers, the dynamic marker moves according to the opening angle of the tip. The procedures for the 3D tracking of the laparoscopic surgical instrument are given below. 1. Sequential grey images are obtained from the camera. 2. The image coordinates of the markers are calculated via image processing by OpenCV [4]. 3. The 3D pose of the surgical instrument is estimated from three static markers using Haralick’s algorithm [3]. 4. The grasper’s opening angle is computed from the dynamic marker. 1.1. Hardware Setup To represent the laparoscopic surgery environment and create the simulator on a computer screen, a test case was produced. To simulate skin tissue, we attached rubber onto the top surface of the case. We also considered the movement of the surgical instruments and field of view of the


583

camera to determine the magnitude of the case. To maintain an even illumination level, we attached a light emitting diode (LED).

Figure 2. Hardware set-up of the computer vision-based laparoscopic surgical simulator

As shown in Figure 2, we used the surgical instrument and a trocar as the training equipment in practice. The surgical instrument has labeled markers which contain four white signs so as to be easily recognized on the black background. The marker located on the tip of the instrument was built to move vertically according to the angle of the gripers. 1.2. 3D Pose Estimation of Laparoscopic Surgical Instrument The image processing procedures for the 3D pose estimation of the laparoscopic surgical instrument are shown in Figure 3. First, Zhang’s single camera calibration method was utilized to obtain the camera’s intrinsic parameters [5]. While tracking the laparoscopic surgical instrument, sequential images are captured by the camera. The images are then binarized with a specific threshold value for the markers (Figure 4). From a binarized image, blobs corresponding to the markers are detected. The image coordinates of each marker are set as the center point of each blob. The center point is calculated to sub-pixel accuracy.

Figure 3. Image processing procedures

In previous work by Doignon, the 3D positions and orientations of the laparoscopic surgical instrument could be estimated using collinear markers [4-5]. With Haralick’s algorithm, if more than 3 points are lying on a common line and the

584


distances between n collinear points are given, the position vector t and the relative orientation vector r of a set of n collinear points can be recovered. The input parameters for the estimation are (1) the camera’s intrinsic parameters, (2) the actual distances between the three static fiducial markers, and (3) the markers’ image coordinates. The actual gaps between the markers are measured by a caliper. The markers’ image coordinates (u, v) are taken from the image pixel coordinates. From the input parameters, t and r are obtained as follows: Let P0 = t , P1 = t + λ1r , … , Pi = t + λi r (i=0, 1, 2, 3) be collinear points where λi is the distance between P0 and Pi. The position of point P0 is chosen as the origin, the perspective projection Qi for Pi with homogeneous coordinates (ui, vi, 1) is expressed as Eq. (1)

ªui º [0 0 1][t + λi r ]««vi »» = K c (t + λi r ) «¬ 1 »¼

(1)

where K c is a (3 × 3) upper diagonal matrix whose components are the camera parameters. From Eq. (1), Haralick proposed a homogeneous linear system with a univariate matrix K c = diag( f , f , 1) .

[Ar

ªr º At ]« » = 0 ¬t ¼

(2)

A is a matrix of (2n × 6). It can be solved with n ≥ 3 distinct points. This linear system is reformulated as a classical optimization problem as Eq. (3). Τ (3) min Ar r + At t subject to r r = 1 where Ar and At are two (2n × 3) matrices containing the camera parameters and values of λi . The solution for Eq. (3) is obtained via singular value decomposition of the following symmetric matrix E. (4) E = ArΤ ( I − At ( AtΤ At ) −1 AtΤ ) Ar The eigenvector corresponding to the smallest eigenvalue of E is the relative orientation vector r, and the position vector t is given by t = −( AtΤ At )−1 AtΤ Ar r .

Figure 4. Image processing (left up: original image, left down: binary image)

1.3. Opening Angle Measurement of the Laparoscopic Surgical Grasper The angle of the grasper is correlated with the distance between static marker (P0) and dynamic marker (P3), because the angle and distance are represented as a straight line. This distance is solved by Eq. (2). Eq. (2) is then reinterpreted into Eq. (5) by λ3 .


§ λ3 f ¨¨ © 0

0

− λ3u3

f

0

λ3 f

− λ3v3

0

f

§ rx · ¨ ¸ ¨ ry ¸ − u3 ·¨ rz ¸ ¸¨ ¸ = 0 − v3 ¸¹¨ t x ¸ ¨t ¸ ¨ y¸ ¨t ¸ © z¹

585

(5)

Here, λ3 is the distance between the static marker and the dynamic marker, ( rx , ry , rz ) is the r vector and ( t x , t y , t z ) is the t vector. As Eq. (5) = 0, we can conclude λ3 by Eq. (6) and Eq. (7)

λ3 = λ3 =

u3t z − ft x frx − u3rz v3t z − ft y fry − v3rz

(6) (7)

The solutions for Eq. (6) or Eq. (7) are obtained by the λ3 value of the distance between static marker (P0) and dynamic marker (P3) using the image coordinates (u3, v3), the t vector and the r vector. From section 2.2, the t vector and the r vector are calculated.

2. Results

To assess the accuracy of the proposed method, we conducted two tests. First, an accuracy test of the 3D position and orientation of the laparoscopic surgical instrument was performed with a precisely designed test device. Second, the relationship between the opening angle of the graspers and the actual distance of the markers was checked. Finally, the 3D tracking module of the laparoscopic surgical instrument was effectively combined with virtual simulation software for use in laparoscopic surgery. 2.1. 3D Coordinates of Surgical Instrument In section 2.2, the 3D coordinates of the laparoscopic surgical instrument were calculated. An accuracy test was conducted which compared the measured values with the theoretical values, as shown in Figure 5. Columns were installed at known lattice positions with intervals of 50 mm by 50 mm for the accuracy tests. After locating the laparoscopic surgical instrument at the accuracy test device’s lattice positions, the 3D coordinates were calculated by the method described in section 2.2. The 3D coordinates for all positions of the accuracy test device were also calculated. One test set consisted of a total of 10 tests, all of which were performed successively to determine the test position in an effort to measure the repeatability. The standard deviations obtained were approximately 1 mm for each position. The results of two test sets are listed in Table 1. The resulting average error of the accuracy tests was determined to be 2.0855 mm.

586


Table 1. Results of the accuracy test (unit mm) Position 1 X

Position 2

Y

Z

X

Y

Z

Avg.

-197.39

-59.52

-2.57

-247.62

-59.89

-3.06

SD

0.4794

0.6477

0.0483

1.6903

0.3871

0.0516

Figure 5. Accuracy test device

2.2. Opening Angle of the Laparoscopic Surgical Grasper The actual distance was measured between the static marker P0 and the dynamic marker P3 with a caliper. The measured range of the graspers’ opening angle was from 0o to 85o. The range of the distances between P0 and P3 is from 11.42 mm to 13.72 mm. The opening angle of the graspers is assumed to be proportional to the distance between P0 and P3. Eq. (8) is given by the theoretical values 2.3 (8) y= x + 11.42 85

where x is the opening angle of the graspers and y is the distance between P0 and P3. As described in section 2.3, the distance between the static marker (P0) and dynamic marker (P3) can be calculated with the image coordinates (u, v) using Haralick’s algorithm. The results are shown in Figure 6. The average error between the theoretical values and the experimental values is 0.2131 mm.

Figure 6. Comparison between the theoretical value and the experimental value for the opening angle of the grasper

2.3. Integration with Simulation Software We combined the proposed surgical instrument tracking module with virtual simulation software that is used for laparoscopic surgery (Figure 7). When a user manipulates the actual laparoscopic surgical instrument, the 3D pose and the opening state of the actual device is measured. A virtual model of the surgical instrument is then manipulated in a virtual simulation environment in real-time. A boundary element method (BEM) is implemented for the physical simulation of a deformable model of the liver [6]. A raytraced collision detection method is used to check for collisions between the surgical tool and the liver model [7]. A real-time interactive simulation was realized at approximately 25 fps, including the time necessary for 3D tracking of the instrument


587

and the physical simulation of the liver model. The bottleneck for the calculation was determined to be the frame speed of the CCD camera.

Figure 7. Laparoscopic surgery simulation system

3. Conclusion

A novel method for the 3D tracking of a laparoscopic surgical instrument using a single camera and fiducial markers is proposed here. Compared with general computer vision technology in which the resolution is close to 2.0 mm, the accuracy of the proposed method for tracking a laparoscopic surgical instrument was considerable. However, the accuracy required in the medical simulation field is higher than that currently available with general computer vision technology. In addition, the tracking of laparoscopic surgical instrument greatly depends on the lighting conditions. A number of issues were considered in an effort to improve the accuracy of the proposed method. These included (1) using a camera capable of high resolution and a high speed frame, (2) providing optimized lighting conditions with uniform brightness, and (3) increasing the number of fiducial markers.

Acknowledgments

This research was supported by the Ministry of Culture, Sports and Tourism (MCST) and by the Korea Creative Content Agency (KOCCA) of the Culture Technology (CT) Research & Development Program of 2010.

References [1] [2]

[3] [4] [5] [6] [7]

C. Doignon: An Introduction to Model-Based Pose Estimation and 3D Tracking Techniques, Scene Reconstruction, Pose Estimation and Tracking (2007), 530 C. Doignon, F. Nageotte, B. Maurin, and A. Krupa: Pose estimation and feature tracking for robot assisted surgery with medical imaging, Unifying Perspectives in Computational and Robot Vision (2008) Vol. 8, 79-101 R. M. Haralick and L. G. Shapiro: Computer and Robot Vision, Addison Wesley Publishing(1992) G. Bradski and A. Kaehler: Learning OpenCV computer vision with the OpenCV library, O’REILLY(2008) Z. Zhang: A Flexible New Technique for Camera Calibration , IEEE Transactions on Pattern Analysis and Machine Intelligence (2000), 1330-1334 D. L. James and D. K. Pai: A unified treatment of elastostatic contact simulation for real-time haptics, Haptics-e: The Electronic Journal of Haptics Research (2001), Vol 2 Y. Kim: Mesh-to-Mesh Collision Detection by Ray Tracing for Medical Simulation with Deformable Bodies, 2010 International Conference on CYBERWORLDS (2010)

588


Perceptual Metrics: Towards Better Methods for Assessing Realism in Laparoscopic Simulators Ravikiran B. SINGAPOGU*, Christopher C. PAGANO*, Timothy C. BURG* and Karen JKL BURG* * Haptic Interaction Lab, Clemson University, Clemson. SC. USA [email protected]

Abstract. This work proposes a novel class of metrics for assessing haptic realism in laparoscopic surgical simulators. Results from a proposed perceptual metric are presented and discussed. Keywords. haptics, perception, laparoscopic simulators, laparoscopic training

Introduction and Background The number of laparoscopic procedures performed in the United States has seen a continual increase in the last decade. Consequently, there is a need to devise training systems that enable faster and more efficient skills training for novices in laparoscopy [1]. Though several Virtual Reality (VR) trainers are currently available, they have not been widely adopted in surgical skills labs [2]. One of the main reasons for this is the lack of realism in VR trainers [3]. Though computer-based trainers feature realistic graphics, most trainers do not simulate the haptic “feeling” arising from tool-tissue interactions [4]. The few simulators that have sought to incorporate simulated haptics have produced only a slight benefit in task performance [5],[6]. For example, Salkini and coworkers demonstrated that the addition of haptic feedback in a specific laparoscopy simulator produced no significant performance benefits [7]. One suggested reason for this is inaccurate or unrealistic haptics. Methods for the assessment of “face validity”, the degree of realism of the simulator, are not well established in the current literature. Most studies reporting face validity for simulators have used a questionnaire-based approach. Subjects were asked to use a Likert-type scale to rate aspects of the simulators’ realism and “feel” [8]. This approach to measuring realism suffers from lack of objectivity and other biases. However, to design better simulators, better metrics for realism need to be designed and evaluated [9]. This work proposes a method to measure the haptic realism of VR simulators using “perceptual metrics.”

R.B. Singapogu et al. / Perceptual Metrics: Towards Better Methods

589

1. Materials and Methods Several studies have shown that humans are capable of accurately estimating length of unseen sticks by holding and wielding them [10],[11]. In this study, sticks of various lengths were rendered using a haptic device and subjects were asked to estimate their lengths based on feeling alone. Eight wooden rods which varied in the lengths and inertial properties were selected for this experiment (Table 1). The haptic interface device used in this experiment was the 5 degree-of-freedom Haptic Wand (Quanser Inc., Canada). Euclidean position and orientation of user’s motion is sensed and is used by the dynamic model of the stick. Force and torque are then calculated based on Newton-Euler laws for 6D motion. The software platform controlling the device consisted of MATLAB (v 7.1) with Real Time Workshop (v 2.1) and Wincon (v. 5.0). The experiment had two sessions: real sticks and virtual sticks. In the real sticks session subjects were given physical sticks that were occluded by a black curtain that eliminated visual feedback. Subjects were asked to wield the stick and estimate its length on a reporting scale. The reporting scale consisted of a sliding pointer, movable by the user to a position from 0-120 cm from the origin of the scale. No markings were visible on the user’s side; the other side had a centimeter scale and when the user estimated the stick length, the reading was noted. In the virtual sticks session, the same set of sticks were rendered by the haptic device and users were asked to wield the virtual stick to estimate length using the same reporting scale. The haptic device was occluded with a black curtain and was not visible to the user. Eight subjects participated in this experiment after providing informed consent. The participants were students between 18-25 years of age. Each user was randomly assigned to receive either the real or virtual session first. Within each session the eight sticks were given twice in a random order.

2. Results and Discussion After data was collected, correlation analysis was performed separately for each of the sessions. In both sessions, actual length was correlated with estimated length. Results of the eight subjects are shown in Table 1, all values are correlation coefficients. The mean value of correlation coefficient for the real sticks was 0.921, while for the virtual sticks it was 0.845. All correlation coefficients had a p-value of < 0.01. It was expected that the correlation coefficient for real sticks would be high (approximately .90) in keeping with previous results. The correlation coefficient of virtual sticks was expected to be lower than for real sticks. However, the closer the virtual correlation value is to the real value, the greater the haptic realism of the simulator. The high virtual value (0.845) in this experiment validates the realism of the haptic device and rendering algorithm.

3. Conclusions and Future Work Can a haptic device accurately render the feel of real surgical instruments and tooltissue interaction? How can the degree of realism of the simulator be accurately measured? This work points to a paradigm for measuring haptic realism using

590

R.B. Singapogu et al. / Perceptual Metrics: Towards Better Methods

“perceptual metrics.” In this study, the degree of realism of the virtual stick was measured by comparing it with real sticks using the perceptual metrics of perceived length. Face validity of haptic simulators can thus be measured using this paradigm, with other haptic perceptual metrics such as stiffness and texture estimation being used to measure other aspects of simulator realism. Table 1. left, correlation coefficients of 8 participants, right, rendered virtual “stick” properties Subject

1 2 3 4 5 6 7 8

Correlation Correlation Coefficient Coefficient Real Sticks Virtual Sticks 0.851* 0.934* 0.762* 0.884* 0.874* 0.903* 0.892* 0.964* 0.769* 0.949* 0.866* 0.837* 0.841* 0.970* 0.921* 0.936* * = p-value < 0.01

Stick

Length

Mass

Inertia

Density

Moment

1

0.50

0.0312

0.0026

0.0624

0.0078

2

0.57

0.0384

0.0042

0.0674

0.0109

3

0.69

0.0508

0.0081

0.0736

0.0175

4

0.80

0.0665

0.0142

0.0831

0.0266

5

0.85

0.0474

0.0114

0.0558

0.0201

6

0.90

0.0689

0.0186

0.0766

0.0310

7

0.95

0.0613

0.0185

0.0645

0.0291

8

1.00

0.0726

0.0242

0.0726

0.0363

References [1] [2]

[3]

[4] [5]

[6]

[7]

[8]

[9]

[10] [11]

K. Roberts, R. Bell, and A. Duffy, “Evolution of surgical skills training,” World Journal of Gastroenterology: WJG, vol. 12, May. 2006, pp. 3219-3224. A.S. Thijssen and M.P. Schijven, “Contemporary virtual reality laparoscopy simulators: quicksand or solid grounds for assessing surgical trainees?,” American Journal of Surgery, vol. 199, Apr. 2010, pp. 529-541. K. Gurusamy, R. Aggarwal, L. Palanivelu, and B.R. Davidson, “Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery,” The British Journal of Surgery, vol. 95, Sep. 2008, pp. 1088-1097. S.M.B.I. Botden and J.J. Jakimowicz, “What is going on in augmented reality simulation in laparoscopic surgery?,” Surgical Endoscopy, vol. 23, Aug. 2009, pp. 1693-1700. L. Panait, E. Akkary, R.L. Bell, K.E. Roberts, S.J. Dudrick, and A.J. Duffy, “The Role of Haptic Feedback in Laparoscopic Simulation Training,” Journal of Surgical Research, vol. 156, Oct. 2009, pp. 312-316. P. Kanumuri, S. Ganai, E.M. Wohaibi, R.W. Bush, D.R. Grow, and N.E. Seymour, “Virtual reality and computer-enhanced training devices equally improve laparoscopic surgical skill in novices,” JSLS, Journal of the Society of Laparoendoscopic Surgeons, vol. 12, 2008, pp. 219–226. M.W. Salkini, C.R. Doarn, N. Kiehl, T.J. Broderick, J.F. Donovan, and K. Gaitonde, “The role of haptic feedback in laparoscopic training using the LapMentor II,” Journal of Endourology / Endourological Society, vol. 24, Jan. 2010, pp. 99-102. I.D. Ayodeji, M. Schijven, J. Jakimowicz, and J.W. Greve, “Face validation of the Simbionix LAP Mentor virtual reality training module and its applicability in the surgical curriculum,” Surgical Endoscopy, vol. 21, Sep. 2007, pp. 1641-1649. B.M.A. Schout, A.J.M. Hendrikx, F. Scheele, B.L.H. Bemelmans, and A.J.J.A. Scherpbier, “Validation and implementation of surgical simulators: a critical review of present, past, and future,” Surgical Endoscopy, vol. 24, Mar. 2010, pp. 536-546. M.T. Turvey, “Dynamic touch,” American Psychologist, vol. 51, Nov 1996, pp. 1134–1152. C.C Pagano, and P.A. Cabe, "Constancy in dynamic touch: Length perceived by dynamic touch is invariant over changes in media," Ecological Psychology, vol. 15, No. 1, 2003, pp. 1-18.


591

Role of Haptic Feedback in a Basic Laparoscopic Task Requiring Hand-eye Coordination Ravikiran B. SINGAPOGU*, Christopher C. PAGANO*, Timothy C. BURG*, Karen JKL BURG* and Varun V. PRABHU* * Haptic Interaction Lab, Clemson University, Clemson. SC. USA [email protected]

Abstract. This work discusses the role and importance of haptic feedback and simulator training for simple laparoscopic tasks akin to the FLS peg-transfer task. Results from a study designed to examine haptics for this purpose are discussed. Keywords. Haptics, skills training, laparoscopic simulators, laparoscopic training

Introduction and Background The role and utility of haptic feedback in laparoscopic surgery is a topic of much debate in the current literature [1]. Recently, quantitative haptic information recorded during in vivo laparoscopy has been documented and demonstrates the presence of haptic (kinesthetic) feedback [2]. Further, these force values lie within a range that are perceivable by human operators [3]. The presence of haptics during surgery raises important questions for laparoscopic training. For example, what type of training will lead resident trainees to efficiently perceive and process haptic information during surgery? Also, what specific tissue properties are more readily perceived by haptic feedback? The Fundamentals of Laparoscopic Skills curriculum is used as the standard for laparoscopic skills training in U.S. medical schools [4]. The technical component of this program consists of five tasks ranging for basic hand-eye coordination to advanced force application and suturing. Previous studies have shown that haptic feedback is useful during force application tasks as well as in determining properties like tissue stiffness [5],[6]. However, the role of haptic feedback for learning hand-eye coordination laparoscopic skills is not well understood. This study investigated the role of haptic feedback in a FLS-based peg transfer-like task.

1. Materials and Methods For this study, virtual “blocks” of three colors were created with identical physical properties. The virtual environment was created using the Chai 3D library (www.chai3D.org). The physics of the environment was handled by Open Dynamics Engine (ODE) which contains collision detection and collision response algorithms.

592

R.B. Singapogu et al. / Role of Haptic Feedback in a Basic Laparoscopic Task

The virtual blocks were manipulated via a standard haptic interface, the Novint Falcon®. The low-level device control was done using the Chai 3D haptic library (Figure 1). The users’ goal was to stack the virtual blocks into sets of three according to their color. Users performed this stacking task with haptic feedback from the device and without haptic feedback. The task of stacking was chosen because it was used in previous studies for basic laparoscopic skill learning [7]. After users completed the virtual tasks, they performed a similar stacking task in the real world. A custom laparoscopic box trainer was built for this purpose using published specifications [8]. One standard laparoscope, inserted through the incision, was used to stack metal nuts of 1.7 cm diameter (Figure 1). Akin to the virtual task, the real task comprised of stacking nine nuts into groups of three according to their color. Participants of the experiment were first briefed about experiment’s objectives and randomly assigned to receive either the haptics or non-haptics virtual task first. The metric for assessing performance was time to completion measured in seconds. After completing both virtual tasks, subjects performed the real task of stacking metal nuts in the physical trainer. Time to complete the task was also used for performance assessment of the real task Ten subjects participated in this experiment after providing informed consent. The participants were students between 18-25 years of age. Recorded time data from all three sessions is shown in Table 1 Table 1. Time to complete stacking task in all three sessions Subject

No Haptics (seconds)

Haptics (seconds)

Real (seconds)

1

165

95

195

2

141

65

150

3

194

117

145

4

119

116

170

5

148

54

99

6

166

143

111

7

99

51

94

8

272

140

300

9

246

104

218

10

182

122

102

Table 1. Time to complete stacking task in all three sessions Figure 1. Physical laparoscopic trainer setup used for task, top virtual interface for haptic task

R.B. Singapogu et al. / Role of Haptic Feedback in a Basic Laparoscopic Task

593

2. Results and Discussion The hypotheses of the experiment are: (1) time to completion with haptics will be significantly shorter than without haptics and, (2) time scores from the haptic session will be more correlated to real task time scores than the non-haptic session scores. Statistical analysis was performed using Minitab (v 15.1). To investigate the first hypothesis a Mann-Whitney U-test was performed to compare the haptic and non-haptic scores. Results showed that scores were significantly different at a p-value of < 0.01. The median completion times were 110 and 165 seconds for the haptics and non haptics sessions, respectively. To investigate the second hypothesis, a correlation analysis was performed between the real scores and the haptics scores as well as real scores and the non-haptics scores. Results showed that non-haptic session scores were significantly correlated with real task scores (r=.747, p-value < 0.05) whereas haptic scores were not significantly correlated with real task scores (r=.432, p-value=.21). This result, contrary to the hypothesis, shows no correlation between haptic scores and real task scores.

3. Conclusions and Future Work The results of this study suggest that haptic feedback does not significantly affect task performance for basic hand-eye coordination tasks in laparoscopic training. This observation confirms earlier results from Chmarra and coworkers who suggested that haptic feedback was not necessary for basic laparoscopic tasks primarily involving hand-eye coordination skills. Consequently, when teaching these skills to residents, visual feedback is the primary sensory mode of learning and should be focused on accordingly.

References [1]

[2] [3] [4]

[5] [6]

[7] [8]

O. van der Meijden and M. Schijven, “The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review,” Surgical Endoscopy, vol. 23, Jun. 2009, pp. 1180-1190. G. Picod, A.C. Jambon, D. Vinatier, and P. Dubois, “What can the operator actually feel when performing a laparoscopy?,” Surgical Endoscopy, vol. 19, Jan. 2005, pp. 95-100. C. Cao, M. Zhou, D. Jones, and S. Schwaitzberg, “Can Surgeons Think and Operate with Haptics at the Same Time?,” Journal of Gastrointestinal Surgery, vol. 11, Nov. 2007, pp. 1564-1569. G.M. Fried, L.S. Feldman, M.C. Vassiliou, S.A. Fraser, D. Stanbridge, G. Ghitulescu, and C.G. Andrew, “Proving the value of simulation in laparoscopic surgery,” Annals of Surgery, vol. 240, Sep. 2004, pp. 518-525; discussion 525-528. M. Chmarra, J. Dankelman, J. van den Dobbelsteen, and F. Jansen, “Force feedback and basic laparoscopic skills,” Surgical Endoscopy, vol. 22, Oct. 2008, pp. 2140-2148. P. Lamata, E.J. Gomez, F.L. Hernández, A. Oltra Pastor, F.M. Sanchez-Margallo, and F. Del Pozo Guerrero, “Understanding perceptual boundaries in laparoscopic surgery,” IEEE Transactions on BioMedical Engineering, vol. 55, Mar. 2008, pp. 866-873. S. Badurdeen, O. Abdul-Samad, G. Story, C. Wilson, S. Down, and A. Harris, “Nintendo Wii videogaming ability predicts laparoscopic skill,” Surgical Endoscopy, vol. 24, 2010, pp. 1824-1828. J.D. Beatty, “How to build an inexpensive laparoscopic webcam-based trainer.” BJU international, vol. 96, 2005, p. 679.

594


A Model for Flexible Tools Used in Minimally Invasive Medical Virtual Environments Francisco SOLERa, M. Victoria LUZONa, Serban R. POPb, Chris J. Hughesb, Nigel W. JOHNb,1 and Juan Carlos TORRESa a University of Granada, Spain b Bangor University, United Kingdom

Abstract. Within the limits of current technology, many applications of a virtual environment will trade-off accuracy for speed. This is not an acceptable compromise in a medical training application where both are essential. Efficient algorithms must therefore be developed. The purpose of this project is the development and validation of a novel physics-based real time tool manipulation model, which is easy to integrate into any medical virtual environment that requires support for the insertion of long flexible tools into complex geometries. This encompasses medical specialities such as vascular interventional radiology, endoscopy, and laparoscopy, where training, prototyping of new instruments/tools and mission rehearsal can all be facilitated by using an immersive medical virtual environment. Our model recognises and uses accurately patient specific data and adapts to the geometrical complexity of the vessel in real time. Keywords. Catheter, guide wire, endoscope, virtual environment, real time

Introduction With the increasing demand for computer based medical simulation, the requirement for realistic visualization has become even more important. Often it is not possible for the physics of every component within the simulation to be calculated correctly in realtime due to computational restraints [1, 3, 4]. Each component within the simulator can be categorized as either essential or supporting to the simulator, where essential components will require the use of the majority of the computational resources. Methods with low computational requirements are typically needed to realistically estimate any supporting components [3]. Varied “real patient” specific data can be acquired from medical scanners in a hospital as a series of two dimensional images. A stack of such images forms a volume of data representing the internal anatomy of the patient where each volume element (or voxel) represents an intensity value on a grid in three dimensional space. Typically, a time consuming segmentation step is then required to extract the 3D geometry of the anatomy of interest [1]. This step cannot be achieved fully automatically and is prone to error. Our approach will employ the medical scanner data in its raw format, however, without any other additional transformations and applies a novel computer generated 1

Corresponding Author: Nigel W. John, Bangor University, UK. E-mail; [email protected]

F. Soler et al. / A Model for Flexible Tools

595

insertion model for the tools used in minimally invasive procedures. This computer model is focused on speed, accuracy physical relevance and integrates successfully into a medical virtual environment. Currently there are very few voxel-based models for guide wire insertion into complex geometries of the human vascular system due to the high complexity and inherent computational demands. However, our model takes advantage of the physical aspects of the problem and limits the computations and the number of nodes describing the virtual tool to a minimum. Moreover, our model will be able to take into account motion effects due to pulsation, respiration or bio-fluid flow (blood, urine, plasma, etc.). Our simulator includes a specific model of bio-fluids flow that is integrated directly and in real-time [2]. The influence of blood flow is an essential component for many procedures in interventional radiology simulators, such as the injection of contrast medium whilst using fluoroscopy. However, in current simulations, blood flow generally plays a supporting role and can contribute to poor face validity. Although the flow does not necessarily need to be accurately computed, it does have to behave realistically and in real-time. The complexity of blood flow represents a challenge for conventional methods of simulation, even at a macroscopic scale (such as flow in arteries, veins). We have used a new computer simulated model to visualize blood flow in arteries using a flocking algorithm approach [7]. In this model each layer of fluid behaves as a flock of blood particles (or boids), interconnected by the parameters that govern the flow dynamics. The method can be successfully used to represent blood flow in the vascular structure and combined with complex haptic interactions of guidewires and catheters in interventional radiology. The blood particle boids react and adjust their position in real-time, hence at each guidewire movement they will recognize the obstacle and the change in the domain and will remodel and reposition in real-time. Accuracy is preserved and the result will be a clear image of the blood velocity field or the tracer concentration field.

1. Methods The research groups at Bangor and Granada are collaborating to develop a voxel-based tool insertion model suitable for catheters and guidewires. This virtual tool is represented by “stretches” (i.e. the amount of the tool inserted in a specific time frame or the length of tool inserted in a “push sequence”); each stretch involves three nodes, two nodes placed at the extremes and the third placed between them and called generically the “control node” (Figure 1). The position of the control node is essential for the tool bending (amount and direction) at the next stretch (if necessary, otherwise no control node is considered). The bending of the tool in a specific node is modelled by associating a “bending-torsion” parameter to the node that employs the tool’s physical characteristics and its interaction with the walls whilst being pushed by the operator. As a general rule, when the position of the control nodes in the wire is computed, the simulation algorithm selects an optimal position considering the value of the “bending-torsion” parameter, the geometry of the obstacle and the force feedback of the push. After any push operation (stretch creation) the previous control nodes, which are considered non-relevant for the motion, are discarded, a procedure which significantly limits the number of nodes in the model, increasing the speed of the simulation. However, when the node head hits a boundary voxel the simulation algorithm computes a “friction” value for each valid voxel in the head node’s

596


neighbourhood, and the voxel with the minimum “friction” value is selected as new location for the head node. The term “friction” in this context is generic and represents the resistance of the head node to changes in the tool’s direction and orientation.

Figure 1. Left: Guidewire representation layout; Right: stretch collision

Each wire node is related with a unique voxel in the voxelized model. We start from a voxelized model built from the same volumetrically model used for generating the polygonal model. A threshold model is applied to characterize voxels within the domain. During the simulation, some collision tests of the last stretch with non-valid voxels are performed. This straightforward test consists on checking if some voxel wrapping each semi-stretch is a non-valid voxel. This set of voxels is computed using a displacement of the vector connecting the two nodes of each semi-stretch. While the head node is moving, the lines connecting the nodes of the last stretch may collide with a non-valid voxel or more. In this situation, the simulation algorithm subdivides the last stretch adding a new stretch in order to adapt the wire representation to the shape of the vessel (Figure 1). However, if the threshold for “bending-torsion” parameter of any of these nodes is exceeded, the movement of the head node is aborted and no subdivision is performed. Any “push” or “pull” operation of the guidewire/catheter is performed using the described procedures and adapting at each step according with the shape of the domain. Blood flow simulation is built using artificial intelligent particles called boids. We consider that the boids flocking group behaviour matches the characteristics of laminar flow (collision avoidance, velocity matching, and flock centring) and is therefore suitable for modelling channel flows. Although a boids model can only be used for visualization purposes, we produce good results by performing a qualitative comparison of our method with existing fluid particle based simulation. Our model uses the idea that each layer of fluid behaves as a flock, interconnected by the parameters that govern the flow dynamics and the fluid physical properties. At the macroscopic level (arterial flow) blood can be considered as a Newtonian fluid and represented using an underlying system of particles. Many structural similarities with existing particle dynamics systems for fluids are considered (e.g., the kernel function in Smooth Particle Hydrodynamics (SPH) is replaced by the flock neighbourhood rule; however the search for nearby particles is still performed in the usual way) (Figure 2, right). The number of particles inside the domain, during the entire simulation, is kept constant in order to satisfy the system’s mass conservation rule. As in SPH method, each particle carries its own physical quantities such as mass, speed and position, which ensure the control over the main physical parameters of the fluid.


597

Figure 2. Left: Stucked particle are dropped from the layer; Right: Comparison between the SPH model and our approach

The simulation consists of two main stages. In the first stage, particles, grouped in layers, are introduced into the domain at equally constant speed. The layers do not mix and if any particle touches the domain’s boundaries it is rendered motionless (velocity reduced to zero, the no-slip condition). The blood particle boids will adapt, recognizing the environment and match their speed and location to the given domain (Example: particles introduced in a tube with homogeneous walls will match their velocity according to Poiseuille’s Law for channel flows). Any obstacle is avoided gradually and particles that during the transition lose speed, and eventually stop, are dropped from their corresponding layers. In this stage, the fluid layers don’t have one single particle leader; the entire group of particles which form the layer act like leaders. This behaviour is implemented when one or more leading particles become “stuck” (with zero velocity) and the layer is forced to move further without them. The number of particles inside the domain during the simulation is kept statistically constant, in order to respect the conservation of mass principle. In the second stage, the simulated flow becomes stable (does not sustain any change over time or the flow is not timedependent anymore). The particles move according to the layer trajectories and any changes in the domain’s geometry automatically triggers the first stage again and the particles start to adapt from the position where that change was made. The results of the blood flow visualization using boids has been compared with existing benchmarks, in particular non-uniform channel flows, with or without obstacles.

2. Results The geometrical and physical aspects of the virtual patient play a very important role in our algorithm's behaviour, its high adaptability being able to minimise the computations required and the information needed to describe the virtual tool. When the inserted virtual guide wire hits a wall of a vessel or an obstacle such as calcification (hardened blockage within an artery) then the tool will respond appropriately. The algorithm dynamically adds bending or friction properties to the tool at the collision point making its further advancement behave accurately (Figure 3). We also use a haptics interface so that the operator can "feel" the force cues when the movement of the tool does become constricted. After the obstacle is passed the extra physical information that was applied will be deleted, preventing the overall computational demands from growing too large. Our model has a very high adaptability being capable to simulate in real time the behaviour of different types of catheters with various

598


properties (The right image in Figure 3 depicts guidewires that have extreme unrealistic values in order to cover the entire example domain).

Figure 3. Guidewire in artery. Control nodes visible (left). Rigid, normal and soft catheters are simulated (right)

3. Conclusions In summary, unless push or pull operations were performed, the tool representation keeps its length constant at every time point, which together with the physically-based parameters involved makes this model extremely fast and accurate. One example application for our model, from interventional radiology, is the accurate simulation of the insertion of a catheter and guidewire into an artery as a part of the Seldinger Technique, and the real-time interaction of these tools with each other and the vessel walls [5, 6]. When combined with our ultrasound and fluoroscopy interfaces, and realtime blood flow simulation, our model can provide a complete environment for medical training, for procedures such as angioplasty and stent placement within a vessel.

References [1] S. Bhata, T. Kesavadasa, K.R. Hoffmann “A physically-based model for guidewire simulation on patient-specific data“, International Congress Series, 1281, 479–484, 2005. [2] C.J. Hughes, S.R. Pop, N.W. John, “Macroscopic blood flow visualization using boids”, Proceedings of the 23rd International Congress of Computer Assisted Radiology and Surgery, Berlin, Germany. 4 (supplement 1), S68-S69, 2009. [3] N.W. John, C.J. Hughes, S.R. Pop, F.P. Vidal, O. Buckley, "Computational Requirements of the Virtual Patient", First International Conference on Computational and Mathematical Biomedical Engineering (CMBE09), Swansea, United Kingdom, June 2009, 140-143. [4] S.D. Laycock, A.M. Day, “Incorporating haptic feedback for the simulation of a deformable tool in a rigid scene”, Computers & Graphics, 29, 341-351, 2005. [5] W. Lawton, R. Raghavan, S.R. Ranjan, R.R. Viswanathan, “Tubes in tubes: catheter navigation in blood vessels and its Applications”, International Journal of Solids and Structures, 37, 3031-3054, 2000. [6] V. Luboz, C.J. Hughes, D.A. Gould, N.W. John, F. Bello, "Real-time Seldinger Technique Simulation in Complex Vascular Models", International Journal of Computer Assisted Radiology and Surgery. 4(6), 589-596, 2009. [7] C.W. Reynolds, “Flocks, herds and schools: A distributed behavioural model”, ACM SIGGRAPH Computer Graphics. 21(4), 25-34, 1987.


599

Segmentation of 3D Vasculatures for Interventional Radiology Simulation Yi SONG a,1 , Vincent LUBOZ b, Nizar DIN b, Daniel KING b, Derek GOULD c, Fernando BELLO b and Andy BULPITT a a School of Computing, University of Leeds b Department of Biosurgery and Surgical Technology, Imperial College London c Department of Radiology, Royal Liverpool University Hospitals

Abstract. Training in interventional radiology is slowly shifting towards simulation which allows the repetition of many interventions without putting the patient at risk. Accurate segmentation of anatomical structures is a prerequisite of realistic surgical simulation. Therefore, our aim is to develop a generic approach to provide fast and precise segmentation of various virtual anatomies covering a wide range of pathology, directly from patient CT/MRA images. This paper presents a segmentation framework including two segmentation methods: region model based level set segmentation and hierarchical segmentation. We compare them to an open source application ITK-SNAP which provides similar approaches. The subjective human influence such as inconsistent inter-observer errors and aliasing artifacts etc. are analysed. The proposed segmentation techniques have been successfully applied to create a database of various anatomies with different pathologies, which is used in computer-based simulation for interventional radiology training. Keywords. Level set, hierarchical segmentation, simulation.

Introduction Interventional radiology (IR) uses angiographic imaging to guide specialized instruments inside vascular systems through tiny incisions. Computer-based simulation proposes an alternative to traditional training since it offers a safe framework to practice specific skills as often as needed. CRaIVE2 is developing such an augmented reality training simulator. One of the key tasks is patient specific segmentation of highly variable vascular networks in terms of shape and texture that represent vascular disease. Many vascular segmentation techniques have been proposed and implemented in recent years, see [1,2,3] for recent reviews. Model-driven, knowledge based image analysis is one of the techniques often used. It aims to describe and capture a priori information regarding the shape, size and position of each structure. Deformable models [4], statistical shape models [5] and probabilistic atlases [6] can also be employed. The limitation of these approaches is the generation of a suitable template 1

Corresponding Author: School of Computing, University of Leeds, LS2 9JT, UK; E-mail: [email protected]. 2 Collaborators in Radiological Interventional Virtual Environments, http://www.craive.org.uk

600

Y. Song et al. / Segmentation of 3D Vasculatures for Interventional Radiology Simulation

model which is able to capture both natural variations and deformations caused by pathology. As our system needs to handle various abnormalities, sometimes with extreme shape deformation, current model-driven approaches are not well suited to our task. Approaches based on the level set method [7,8] have also been used successfully for medical image segmentation. However, the problem of “leaks” on the boundary is still open to research. Other drawbacks include the non-trivial work of tuning parameters and placing seeds. Skeletonization techniques [9] generate very accurate vasculatures but they often require a lot of user interaction to improve the skeleton, especially around artifacts. In this paper we propose a segmentation framework including two general solutions to overcome such limitations: a region model based level set method and an interactive hierarchical method. In section 2, the two methods are compared with ITKSNAP3 [10] which provides both a classic level set approach (semi- automatic) and interactive segmentation tools.

1. Segmentation Methods 1.1. Region Model Based Level Set Method (RMLS) This method [11] segments anatomic structures from CT/MRA images by evolving an implicitly described active surface through the zero level set of a level set function. Traditional level set based approaches require the placement of seeds to initialize the algorithm. Since the segmentation result is sensitive to both the seed positions and seed numbers, an inaccurate initialization can result in slow convergence to the targeted boundary and increase the risk of leaks. To overcome the problem, we replace the seed placement by a region model, which is a close approximation to the actual structure shape. This can significantly reduce the time of convergence and thus reduce the risk of “leaking” during the level set evolution process.

Figure 1. Edge enhancement. (a) Original slice in sagittal view (image I1). The area inside the rectangular box indicates the location of the abdominal aorta. (b) The gradient magnitude of image I1 (only partial image I1, i.e. the area inside the rectangular box of (a) is displayed). (c) Spurious edge suppression. (d) Image I2 after subtracting the enhanced edge from image I1 (e) Feature map of image I0.

3

SNAP is an application developed on the National Library of Medicine Insight Segmentation and Registration Toolkit (ITK). http://www.itksnap.org


601

First, we apply edge enhancement (Figure 1). The gradient magnitude of image I1 (Figure 1a) at each pixel is computed (Figure 1b) and a lower hysteresis threshold T is applied to the result to suppress spurious edges, as shown in Figure 1c. T is estimated using the histogram of the gradient image. The remaining edges are enhanced to the maximum intensity value C of image I1, Eq. (1). I 1' ( x)

C if 0

I 1 ( x) else

T

(1)

Next, we employ mathematical morphology operations on the enhanced image I2 = I1 - I1 (Figure 1d, 1e) and capture the major object structures to form the region model which is then embedded into the level set function for further refinement. As the level set starts from a close approximation to the actual structure shape, this approach can improve the accuracy of results and dramatically decrease processing time. Consequently, the risk of leaks which is common to previous level set based approaches can be minimized. In our approach, a user selects intensity values of the target tissues and no seed placement is required. 1.2. Hierarchical Segmentation (HS) An alternative segmentation algorithm, not dependent on the gradient of the dataset, is provided by hierarchical segmentation. This algorithm starts by creating a hierarchy in a bottom up approach. Initially, there are as many regions as there are voxels in the dataset. These regions are then merged together iteratively based on edge strength between adjacent regions to converge into homogenous regions of similar intensities. Combining the homogeneity and edge stability constraints leads to a single edge measure [12]: I . arctan

I 2

(2)

I

I

Where I is a region mean intensity, 2

is the absolute difference between the two

adjacent regions’ mean intensities, and I is the average of the mean Laplacians of the two regions. At the end of this process, the initial hierarchy is computed. The method subsequently allows the user to define interactively the inside and outside of the required anatomical structure by marking seed points in the image (Figure 2, left). Starting from these points, the algorithm iteratively merges adjacent regions of the hierarchy based on their intensity and therefore separates the inside and outside of the anatomical structure. Griffin et al. [12] proposed to apply this basic hierarchical segmentation to segment a whole dataset in two different ways: perform it on each 2D slice of a whole dataset, or use it on several slices and use the same seeds to propagate the segmentation in 3D to all other slices automatically. The first method can be extremely slow (depending on the number of slices) while the second one is often not very accurate as the intensity of the wanted anatomical structure might vary along the dataset.

602


Figure 2. Left: Hierarchical segmentation of the aorta arch. Right: The intersection of the 2D segmentations produces seeds in perpendicular slices, in black for the exterior and in white for the interior seeds.

We propose a third way to adapt it: “2.5D propagation”. It is based on the segmentation of few 2D slices and a new propagation method computing the segmentation results more quickly and accurately. In the automatic propagation phase, a perpendicular slice intersects the user’s 2D segmentations to form lines of seeds (Figure 2). These seeds are used to perform new 2D segmentations on the next slices as they provide enough information for a hierarchical segmentation in those perpendicular slices. Because this algorithm only performs a 2D perpendicular segmentation at a time and each resulting segmentation is stored, it requires significantly less memory than the 3D propagation algorithm, and it is therefore faster. The user can place more seeds, which leads to more accurate results. No parameter or threshold has to be set for this technique.

2. Evaluation 2.1. Evaluation Datasets and Methods The RMLS and HS methods have been evaluated on 20 patient datasets. 19 cases are contrast enhanced images with average resolution of 0.7×0.7×1mm3 and typically 600 slices with 512x512 in-plane pixels (provided by St. Mary’s London Hospital). The remaining dataset is a magnetic resonance angiography image provided by Royal Liverpool Hospital with resolution of 1.5×1.5x2mm3 and 384 slices with 256x256 pixels. The criteria of segmentation includes ascending aorta, aortic arch, brachiocephalic arteries, common carotids, subclavian arteries, descending aorta, celiac artery, superior mesenteric artery, renal artery, common iliac arteries, external lilac arteries, internal iliac arteries and common femoral arteries, if visible in each dataset. ITK-SNAP provides an intuitive and easy to use interactive segmentation tool to assist manual segmentations which are taken as evaluation references. As ITK-SNAP also provides a classic semi-automatic level set implementation, we study the differences between our RMLS approach and the SNAP level set approach. The results of manual segmentation, ITK-SNAP (level set) and HS were provided by Imperial College London (and by the same person). The results of the RMLS method were completed at the University of Leeds. Both research centres used the same PC configurations and recorded the segmentation time. The evaluation of accuracy was conducted by using the MESH4 tool. The metric is the symmetric mean error which is the approximation of Hausdorff distance between discrete 3D surfaces represented by triangular meshes. We created surface meshes from 4

Measuring Error between Surfaces using Hausdorff distance: http://mesh.berlios.de/


603

segmentation results using the marching cubes algorithm from the VTK5 library. The same parameters are applied on all datasets. 2.2. Performance 2.2.1. Accuracy For each of the three methods, the symmetric mean error (SME) of each patient is plotted in Figure 3. Summary results for Figure 3 are given in Table 1, including the minimum, maximum, average and RMS of the symmetric mean errors of the 20 datasets. 2.2.2. Efficiency The evaluation was conducted on an Intel Core2 2.66GHz processor. As expected, even with the assistance of the interactive segmentation tool, the manual segmentation was still the most time consuming, taking on average 527.8mins with a standard deviation (SD) of 182mins. Although HS is an interactive method, it is much more efficient with a mean of 104mins and a SD of 37mins. Since the ITK-SNAP level set implementation involves many time-consuming seeds placement operations, it takes a mean of 120mins and a SD of 60mins. On the contrary, RMLS does not require seeds placement and takes less time to converge, it takes the least time with an average of 10 minutes. Figure 4 gives the running time for each patient with each of the three methods.

Figure 3. Comparison of the symmetric mean errors for each dataset and for ITK-SNAP, RMLS and the HS.

Figure 4. Graph of time efficiency for each segmentation method. 5

Visualization Toolkit (VTK): http://www.vtk.org/

604


2.3. Human Factors As all the methods require human interventions, the subjective influence from observers is unavoidable. For example, although there is a basic segmentation guidance on which part of the vasculature should be segmented, it is impossible to define case by case on how far those tiny branches should be represented in the segmentation result (as shown in Figure 5). In this study, the manual segmentation, HS and ITK-SNAP (level set) segmentation were completed by one person with a medical background, while the RMLS was done by a different person from a non-medical background. Therefore, the RMLS results include larger artifacts than other approaches. These artifacts mainly involve inconsistent inter-observer errors. Figure 5 visualizes a case with the largest symmetrical mean errors. As marked by the circles, the blue depicts those voxels which are missed by the RMLS but included by the manual segmentation; the yellow illustrates voxels which are missed by the manual segmentation but correctly identified by the RMLS. Here, red indicates voxels segmented by both methods. The segmentation resolution can also cause differences between two methods. This is also illustrated in Figure 5, where the descending aorta has been amplified and displayed at the left corner of (a) and (b), respectively. As shown in Figure 5b, being completed slice by slice on DICOM data, the manual segmentation normally has many aliasing artifacts in the result. The HS method is based on 2.5D propagation which requires fewer 2D slice segmentations. Therefore its results contain less aliasing artifacts compared to manual segmentation. RMLS and ITK-SNAP (level set) are both 3D segmentation approaches, resulting in smoother blood vessel walls, as shown in Figure 5a. HS and ITK-SNAP (level set) demand users to place relatively large amounts of seed points at appropriate positions. Consequently human operation time and the computational time are different from case to case. On the other hand, requiring only minimal user interventions, the RMLS method can be used by researchers with little anatomy knowledge. Another advantage is that the segmentation time varies little between datasets.

Figure 5. Visualization of the differences between the results of the RMLS and the manual segmentation. (a) RMLS result. (b) Manual segmentation result. (c) Visualization of differences.


605

Table 1. Summary of the symmetric mean errors. ITK-SNAP RMLS HS

Min. SME (mm) 1.22 1.46 0.904

Max. SME (mm) 5.24 10.74 6.44

Avg. SME (mm) 2.36 4.93 2.37

RMS SME (mm) 7.45 14.96 5.99

3. Conclusion This study presents and evaluates two segmentation methods which aim to overcome the shortcomings of traditional level set and interactive approaches. It also analyses the subjective influence of human observers on the segmentation results. The proposed segmentation techniques have been successfully applied to create a database of various anatomies with different pathologies, which is used in computer-based simulation for interventional radiology training.

4. Acknowledgements This work is partly funded by the UK Engineering and Physical Sciences Research Council (EP/E002749).

References [1]

J.S. Suri, K. Liu, L. Reden, and S. Laxminarayan, “A Review on MR Vascular Image Processing: Skeleton Versus Nonskeleton Approaches II”. IEEE Trans. on Information Technology in Biomedicine, vol. 6, No. 4, December, 2002. [2] C. Kirbas and F. Quek, “A Review of Vessel Extraction Techniques and Algorithms”. ACM Computing Surveys, vol. 36, no. 2, pp. 81-121, June, 2004. [3] P. Campadelli and E. Casiraghi, “Liver Segmentation from CT Scans: A Survey”. Lecture Notes in Computer Science, Springer Berlin, vol. 4578, pp. 520-528, August, 2007. [4] L. Gao,, D.G. Heath, E.K. Fishman, “Abdominal Image Segmentation Using Three-dimensional Deformable Models”. Investigative Radiology, 33(6), pp. 348-355, 1998. [5] T. Heimann, H.P. Meinzer and I. Wolf, “A Statistical Deformable Model for the Segmentation of Liver CT Volumes”. MICCIA 2007 workshop proceedings, pp. 161-166, 2007. [6] H. Park, P.H. Bland and C.R. Meyer, “Construction of an Abdominal Probabilistic Atlas and Its Application in Segmentation”. IEEE Transactions on Medical Imaging, vol. 22, no. 4, pp. 483-492, April, 2003. [7] J.A. Sethian, “Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational”. Geometry, Fluid Mechanics, Computer Vision and Materials Science, Cambridge Univ. Press, 1999. [8] V. Caselles, R. Kimmel and G. Sapiro, “Geodesic Active Contours”. International Journal of Computer Vision, 22(1) 6179, 1997. [9] V. Luboz, X. Wu, K. Krissian, C.F. Westin, R. Kikinis, S. Cotin, and S. Dawson, “A segmentation and reconstruction technique for 3D vascular structures”. Proceedings of the MICCAI Conference, pp 43-50, Palm Spring, CA, October 2005. [10] P.A. Yushkevich, J. Piven, H.C. Hazlett, R.G. Smith, S. Ho, J.C. Gee and G. Gerig, “User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability”. NeuroImage, 31 (3), 1116-1128, 2006. [11] Y. Song, A.J. Bulpitt and K.W. Brodlie, “Efficient Semi-automatic Segmentation for Creating Patient Specific Models for Virtual Environments”. MICCAI 2008 workshop on Computer Vision for Intravascular Imaging (CVII), pp.22-34, 2008. [12] L.D. Griffin, A.C.F. Colchester, S.A. Roell, “Hierarchical Segmentation Satisfying Constraints”. BMVC94, 135-144, 1994.

606


EEG-based “Serious” Games and Monitoring Tools for Pain Management Olga SOURINA1, Qiang WANG, and Minh Khoa NGUYEN Nanyang Technological University, Singapore

Abstract. EEG-based “serious games” for medical applications attracted recently more attention from the research community and industry as wireless EEG reading devices became easily available on the market. EEG-based technology has been applied in anesthesiology, psychology, etc. In this paper, we proposed and developed EEG-based “serious” games and doctor’s monitoring tools that could be used for pain management. As EEG signal is considered to have a fractal nature, we proposed and develop a novel spatio-temporal fractal based algorithm for brain state quantification. The algorithm is implemented with blobby visualization tools for patient monitoring and in EEG-based “serious” games. Such games could be used by patient even at home convenience for pain management as an alternative to traditional drug treatment. Keywords. EEG, serious games, pain management, neurofeedback, fractal dimension

Introduction Electroencephalogram (EEG) is a non-invasive technique recording the electrical potential over the scalp with multiple electrodes. Neurofeedback (NF) is a technique that allows the user voluntary change his/her brain state based on the visual or audio feedback corresponding to the recognized brain state of the user from his/her EEG signals. Some research reveals that the EEG and Event Related Potential (ERP) distortion can reflect psychological disorders such as Attention Deficit Hyperactivity Disorder (ADHD) [1-2], Autistic Spectrum Disorders (ASD) [3-4], Substance Use Disorders (SUD) including alcoholic and drug abuse [5-6], etc. Similar to other parts of our body, the brain function can be trained as well. Neurofeedback is an alternative choice as a treatment to these disorders besides a medical treatment. Many neurofeedback games were assessed and it was proved that they have a healing effect on psychological disorders, e.g. ADHD [7]. Current treatments for pain syndrome employ multidisciplinary approach such as chemical (drugs), physical (therapeutic exercise, acupuncture), psychological approach (relaxation with music, biofeedback, and hypnosis) or a combination of the abovementioned approaches. Recently, virtual games with the distraction effect were proposed for pain management [8-10]. 3D games and Virtual Reality (VR) games were used during the burn dressing of children in pain [8], during treatments of wounds [910], etc. It was also reported successful applications of EEG-based games for Central 1

Corresponding Author: Olga Sourina, Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798; E-mail: [email protected].

O. Sourina et al. / EEG-Based “Serious” Games and Monitoring Tools for Pain Management

607

Pain Syndrome (CPS) pain management [11], and migraine management [12]. Although a great improvement in pain relief is reported by the therapy with EEG-based games, fundamental research and clinical experiments are needed to validate the results. Novel algorithms of pain level quantification and adaptive algorithms for pain therapy should be proposed. From our preliminary study, there are four main directions in game therapy for the pain management: 1) through distraction, 2) alteration of mood (emotion induction), 3) the use of relaxation, and 4) improved sense of control. In this paper, we describe EEG based “serious” games we developed that could be used for pain management through distraction and improved sense of control by relaxation versus concentration.

1. Methods & Materials The EEG-based game design includes two main parts: signal processing algorithms and a 2D/3D or VR game part. Raw EEG signals recorded from the user’s brain are filtered and analyzed by signal processing methods, and the resulting values are interpreted in the game as an additional game control just by the user’s “brain power”. A therapeutic effect of such games consists of combination of the distraction effect of the game and effect from the learning by the user/patient how to control the game by changing voluntary his/her brain state. The mainly used signal processing algorithm in the neurofeedback implementation is a power spectrum analysis in different frequency bands, i.e. band (30 Hz) of the EEG signals. Each frequency band is related to different brain functions [13]. ERP analysis, e.g. SCP and P300 component analysis, is another useful tool in neurofeedback that widely used in ADHD treatment [7], and drug abuse rehabilitation [6]. The linear features such as a power spectral density or amplitude extracted from EEG cannot represent the brain activities perfectly well due to the nonlinearity of the EEG signal. Thus, nonlinear methods, e.g. entropy analysis and fractal dimension analysis, are used for EEG processing in many medical applications and could be applied to neurofeedback systems to model brain activities. Fractal dimension (FD) is a measurement of complexity and irregularity of a signal. In our work [14-15], we studied a generalized Renyi approach based on Renyi entropy and applied it to quantification of EEG signals. Entropy is a measure of disorder in physical systems. Thus, with calculation of the fractal dimension we could estimate the brain signal complexity. There are different fractal dimension algorithms. In our work [1617], for calculation of fractal dimension values, we implemented and analyzed two well-known Box-counting [18] and Higuchi [19] algorithms. Both of them we evaluated using Brownian and Weierstrass functions where theoretical true values are known. The Higuchi algorithm gave a better accuracy as its FD values were closer to the theoretical ones [17]. Thus, we used Higuchi algorithm to quantify level of concentration as opposite to relaxation. In this paper, we proposed to apply the fractal-based algorithm to quantify levels of concentration for “serious” game implementation that could be used for pain management through the user distraction and the improved sense of the user control. The concentration/relaxation level values could be interpreted in the games with any visual/audio effects or a change of characters behavior. We also proposed and implemented real-time doctors monitoring tools named “VisBrain” system that could be used for spatio-temporal visualization of the EEG signals. Amplitude and/or

608


calculated FD values are mapped to a 3D head using the time-dependent blobby model [20]. “VisBrain” could be applied for a study of spatio-temporal patterns of patient cases with pain syndromes of different origins. With the proposed system, EEG data could be collected during the doctor patient sessions, and a questioner on the pain assessment should be given to each patient. Based on the EEG monitoring and the questioner, the pain level could be quantified, and spatio-temporal maps of the pain location in the brain for different cases including CPS cases could be created.

2. Results “Brain Chi” and “Dancing Robot” are two simple single-player games that we developed and applied for the pain management. In the “Brain Chi” game, the quantified level of the user concentration is associated with the radius of the “growing/shrinking” ball that allows a “little boy” character to fight enemies by “growing” the ball. In the “Dancing Robot” game, the relaxation/concentration level is associated with a “robot” character behavior. When the concentration level increases, the “robot” character starts to move faster. Examples of the game settings are shown in Figure 1 (a) and (b). In order to play the developed EEG-based games, the user needs an EEG reading device and computer. In our study, EEG data is collected by the Emotiv device or Pet 2. Currently, only O1 electrode in the occipital lobe following the American Electroencephalographic Society Standard is active in our games. EEG signal is transmitted to the computer with Bluetooth. Our final target is to implement series of “Brain Chi” games that allow the user just by playing the games to “reprogram” the corresponding parts of the brain that could be monitored as changing fractal dimension values. Such brain exercises could lead to the pain relief as it was validated in our preliminary study. Different games could be proposed for groups of patients. To monitor the patient state, we use real-time system “VisBrain”. The raw data is read by Emotive device and then filtered by 2-42 Hz bandpass filter. All electrodes are active in the system. A doctor could choose the appropriate mode of 3D signal mapping. It could be a 3D blobby, “pins”, or color visualization. Currently, we visualize the changing signal amplitude and FD values. The 3D blobby mapping allows the doctor to assess the spatio-temporal pattern of the patient brain states. In Figure 2 (a), (b), a spatio-temporal visualization of EEG signals is shown with the blobby and color mapping correspondingly.

Figure 1. Examples of neurofeedback concentration/relaxation games: a) “Brain Chi”; b) “Dancing Robot”


609

Figure 2. Online EEG monitoring tools: a) with blobby visualization; b) with color mapping.

3. Discussion The EEG-based “serious” games for pain management could be played by a patient not only in the doctor’s office but also at home convenience as well. Thus, the doctor during the session using the “VisBrain” monitoring tools could advice to the patient the schedule of the game therapy. More research should be done in future on the use of audio and visual stimuli in the games to improve the therapeutic effect of the games. In work [21], music stimuli for emotion induction were studied, and in [22], we proposed and implemented a real-time fractal-based emotion recognition algorithm where we mapped fractal dimension values to a 2D Arousal-Valence emotion model. To evoke emotions in the game, different stimuli could be used: visual, auditory, and/or the combined ones. Currently, we can recognize online the following emotions: satisfied, pleasant, happy, frustrated, sad, fear, and neutral with 89.13% and 90% for arousal and valence levels respectively. Only 3 channels are necessary. In future, we are going to use the results of the study in pain management game development by adding emotional dimension into the game. The following innovations were proposed and are expected in the future. Technological: We proposed a novel EEG-based technology that includes spatiotemporal fractal-based algorithms of brain state recognition, and the doctor’s monitoring tools as software implementation. We have to further improve EEG-based pain management therapy tools. Scientific: We proposed a new spatio-temporal fractalbased model of pain level quantification and EEG-based pain management approach leading to the implementation of doctor’s and patient tools for the EEG-based monitoring and pain treatment. Economic: We proposed a cost-effective EEG-based technology for the pain treatment. The cost of the developed software is included in the current project cost and can be minimized just to the cost of CD copy. The final cost of the treatment could include a doctor session fee and one EEG headset with an access to PC computer that currently has a tendency to reduce. Then, the patient could have any amount of sessions prescribed by the doctor free of charge. Social: By research among institutionalized elderly, 71% to 83% of them report at least one pain problem, and 60% of adult men and women experience some pain [23]. Central Pain Syndrome not only causes physical discomfort, but also interferes with social relationships, family life and self-esteem, and there is a high correlation between the chronic pain and depression. Considering all above the proposed pain treatment tools could improve quality of life of 60% of adult and 70-80% of elderly people giving the patients a good alternative to more expensive traditional drug treatment.

610


Acknowledgment This project is supported by grant NRF2008IDM-IDM004-020 “Emotion-based personalized digital media experience in Co-Spaces” of National Research Fund of Singapore.

References [1] J. F. Lubar, et al., Evaluation of the effectiveness of EEG neurofeedback training for ADHD in a clinical setting as measured by changes in T.O.V.A. scores, behavioral ratings, and WISC-R performance, Biofeedback and Self-Regulation 20 (1995), 83-99. [2] T. Fuchs, et al., Neurofeedback treatment for attention-deficit/hyperactivity disorder in children: A comparison with methylphenidate, Applied Psychophysiology Biofeedback 28 (2003), 1-12. [3] R. Coben, et al., Neurofeedback for autistic spectrum disorder: A review of the literature, Applied Psychophysiology Biofeedback 35 (2010), 83-105. [4] M. E. J. Kouijzer, et al., Neurofeedback treatment in autism. Preliminary findings in behavioral, cognitive, and neurophysiological functioning, Research in Autism Spectrum Disorders 4 (2010), 386399. [5] E. Saxby and E. G. Peniston, Alpha-theta brainwave neurofeedback training: An effective treatment for male and female alcoholics with depressive symptoms, Journal of Clinical Psychology 51 (1995), 685693. [6] T. M. Sokhadze, et al., EEG biofeedback as a treatment for substance use disorders: Review, rating of efficacy, and recommendations for further research, Applied Psychophysiology Biofeedback 33 (2008), 1-28. [7] H. Gevensleben, et al., Distinct EEG effects related to neurofeedback training in children with ADHD: A randomized controlled trial, International Journal of Psychophysiology 74 (2009), 149-157. [8] http://videos.howstuffworks.com/sciencentral/2888-virtual-pain-relief-video.htm. [9] http://www.myfoxaustin.com/dpp/news/local/111909-Video-Game-Therapy-Helping-Soldiers. [10] http://www.impactlab.com/2006/03/19/introducing-video-game-therapy. [11] http://www.youtube.com/watch?v=6qocxopS5fc&feature=player_embedded. [12] http://www.youtube.com/watch?v=SKY-TlAt4co. [13] J. N. Demos, Getting Started with Neurofeedback, WW Norton & Company, New York, 2005. [14] V. Kulish, A. Sourin, O. Sourina, Human electroencephalograms seen as fractal time series: mathematical analysis and visualization, Computers in Biology and Medicine, Elsevier-Pergamon 36 (2005), 291-302. [15] V. Kulish, A. Sourin, O. Sourina, Analysis and visualization of human electroencephalograms seen as fractal time series, Journal of Mechanics in Medicine & Biology 26 (2006), 175-188. [16] Q. Wang, O. Sourina, M. K. Nguyen, A Fractal Dimension Based Algorithm for Neurofeedback Games, In Proc. of CGI 2010, SP(26), Singapore, 8-11 Jun 2010. [17] Q. Wang, O. Sourina, M. K. Nguyen, EEG-based “Serious” Games Design for Medical Applications, In Proc of 2010 Int. Conf. on Cyberworlds, 270-276, Singapore, 20-22 Oct 2010. [18] A., W. Block, V. Bloh, and H. J. Schellnhuber, Efficient box-counting determination of generalized fractal dimensions, Physical Review A 42(1990), 1869-1874. [19] T. Higuchi, Approach to an irregular time series on the basis of the fractal theory, Physica D: Nonlinear Phenomena 31(1988), 277-283. [20] O. Sourina, A Sourin, V. Kulish, EEG Data Driven Animation and its Application, In Proc. of International Conference Mirage 2009, 380–388, 4-5 May 2009. [21] O. Sourina, V. Kulish, A Sourin, Novel Tools for Quantification of Brain Responses to Music Stimuli, In Proc. of 13thInternational Conference on Biomedical Engineering ICBME, 411-414, Singapore, 3–6 Dec 2008. [22] Y. Liu, O. Sourina, M. K. Nguyen, EEG-based Human Emotion Recognition and Visualization, In Proc of 2010 Int. Conf. on Cyberworlds, 262-269, Singapore, 20-22 Oct 2010. [23] L. Galieze, Chronic Pain in Elderly People, Pain 70 (1997), 3-14.


611

A New Part Task Trainer for Teaching and Learning Confirmation of Endotracheal Intubation Dr Cyle SPRICK a,1 Prof Harry OWEN a Dr Cindy HEIN b Dr Brigid BROWN c a Flinders University b SA Ambulance c Southern Adelaide Health Service Abstract. Endotracheal intubation is a skill employed by a diverse range of healthcare professionals in a wide variety of circumstances. Failure to put this tube in the right place (in the trachea) can result in serious injury and death. There are a wide variety of methods for verification of proper placement. Some are more widely accepted than others. A universal guideline should be adopted to allow consistent and safe practice in all situations by all who do this procedure. Training for endotracheal intubation must also include training in the verification methods. We have developed a new airway part-task trainer that allows the use of all of the methods of tube placement verification.. Keywords. Simulation, endotracheal intubation, ETI, SimTools, confirmation

Introduction Healthcare practitioners from acute care disciplines need to be proficient in a range of airway management techniques. At our hospital, intensive care and emergency medicine trainees, paramedics, Royal Flying Doctor Service nursing staff, rural GPs and medical students have clinical placements in the operating theatre suite with the objective of acquiring or refreshing airway intervention skills. Most want to be able to intubate but the educational attachments are typically short and individuals have had varying amounts of preparation. The consequences of unrecognised oesophageal intubation are catastrophic but this event can be avoided through verifying correct placement of an endotracheal tube. Staff and patients move between clinical areas (e.g. an ICU nurse may transfer to anaesthesia or the RFDS) and patients may be handed over (e.g. from paramedics to the ED) so a single, universal, evidence-based process of checking the position of an ET tube would be valuable for patient safety. This would also facilitate cross-checking of the process by other team members which is an important consideration.

1


612

C. Sprick et al. / A New Part Task Trainer for Teaching and Learning Confirmation

1. Methods A search was undertaken for methods of confirming (verifying) endotracheal intubation for anaesthesia and in other areas of patient care. We determined there were two distinct populations that required endotracheal intubation: patients with an effective circulation and those without. The first group includes patients being intubated for ventilation during anaesthesia or needing respiratory support as a part of intensive care. These patients had an effective circulation, some preparation was possible (e.g. preoxygenation) and the intubation usually took place in a controlled environment. The second group was made up of patients that had severe shock or have had a cardiac arrest so that little or no CO2 is being delivered to the lungs. These patients needed oxygenated cerebral blood flow now and anything that delayed that would worsen outcome. In addition they could be in a noisy or dirty environment with inappropriate lighting. Methods for confirming tracheal intubation needed to suit both of these groups. Part-task trainers (desktop simulators) are used for initial training in endotracheal intubation. Whole-body manikins (patient simulators) of varying sophistication are used mostly for practising difficult intubation scenarios. The investigators searched for and then evaluated as many of these models as they could and reviewed their features. A new part task trainer was constructed that incorporates ALL the recommended methods of confirmation of proper tube placement. This allows trainees to practice the full placement confirmation procedure each time they undertake this task.

2. Results Many ways of confirming endotracheal intubation have been described and these are listed in table 1. Currently, auscultation is the only clinical method that identifies the tip of the ET tube is in the trachea or a bronchus. No single method of confirming endotracheal intubation can be relied on, not even the reputed ‘gold standard’ of waveform capnography, so several methods must be used in combination. A universal process needs to be applicable to all intubation settings, be quick to perform, not require major additional investment and have high face validity. This eliminated the majority of methods listed in table 1 and left only those recommended in resuscitation guidelines from ILCOR. Standardised teaching and practise of confirmation of intubation through a consistent sequence of observations is a desirable objective. It is stressed that the tube must be seen to pass between the vocal cords and these methods are then used for confirming endotracheal intubation. Whilst a single process of checking would be ideal, alternatives for patients that have or don’t have a pulse (circulation) reduced the number of steps for most patients (See table 2). Healthcare professionals are already familiar with resuscitation guidelines on the management of patients with and without a pulse and this approach to confirmation of intubation is an extension of that. A summary of findings when we tried to use these methods on the various airway trainers is given in Table 3. Unfortunately, none of the airway trainers incorporated all of the recommended methods for confirming tube placement. This results in learners not having the opportunity to practice the whole procedure including confirmation of placement. When learners rehearse only part of a procedure, they are much more likely to perform only that part of the procedure in actual practice.


613

Table 1: Methods of confirmation of tracheal intubation

Method

Comments

Bilateral chest rise

Not always possible to assess

Auscultation of axillae and epigastrium

Sounds can be transmitted. Can differentiate tracheal from bronchial intubation

Breath by breath (continuous) CO2

Very useful if patient has an effective circulation.

Oesophageal detector device

Very useful and quick but rarely taught. Less reliable after breaths have been delivered

Condensation in tube (misting, fogging)

Not reliable (ref) but widely taught

“Clicking” of bougie on tracheal rings

Useful, requires a bougie with a Coude tip

Visualisation of tracheal rings or carina

Very specific but uses expensive equipment

Transcricothyroid membrane ultrasound

Appears useful but requires additional equipment and training

Transillumination

Inexpensive but needs shade

Computerised breath sound analysis

Requires electronic stethoscope and proprietary software

Transthoracic impedence

Little evidence of value yet

Reflected sound waves

Under development, could be useful

Magnet and sensor or RFID device

No commercial product available

Repeat direct laryngoscopy

Error prone and likely delay CPR during resuscitation

Pulse oximetry

Should not be used for verification of intubation

Chest x-ray

Not to be used for verification but may show need to reposition Table 2: Suggested guidelines for verifying endotracheal intubation

ET tube seen entering the trachea through the vocal cords, then… Patient has pulse (effective circulation)

No pulse (Cardiac arrest/severe shock)

Oesophageal detector device *

Oesophageal detector device

Bilateral chest rise**

Bilateral chest rise**

Auscultation of R and L axillae and epigastrium

Auscultation of R and L axillae and epigastrium

CO2 detected with every breath

CO2 detector attached***

Pulse oximeter checked

Pulse oximeter checked

Note tube length at teeth and fix

Note tube length at teeth and fix

Record above observations and tube length at teeth. If patient is moved, head or neck position is changed, expired CO2 changes or SpO2 falls unexpectedly, tube marking at teeth is different, a cuff leak develops or lung compliance changes the position of the ETT must be re-evaluated urgently.

We modified existing airway trainers to incorporate all of the recommended methods of tube placement including: Oesophageal Detector Device, unilateral and bilateral chest rise, auscultation of the epigastrum and axillae, and CO2 detection. This was done using a combination of mechanical modifications to commercial airway trainers and addition of the SimTools CO2 capnometer emulator (which is modeled after the EasyCapII from Nellcor Puritan Bennett) and the SimTools stethoscope. (Figure 1)

614

C. Sprick et al. / A New Part Task Trainer for Teaching and Learning Confirmation Table 3: Functionality of airway trainers for verifying endotracheal intubation

Model TRM LAT LFC LDA LTA SAK KOK CLA AST LAT AAT STT

ODD – () – – – – – – –

Bilat CR () – – – () – – – – –

Ausc. axillae – – – – – – – –

Ausc. epigastrium – – – () – () – – – –

CO2 – – – – – – – – – – – –

The key to airway models is available from the authors. = feature is present, () feature present but not well emulated and – feature not present.

The SimTools stethoscope was described previously at MMVR as part of a suite of tools to provide simulated patient information for manikins or actors.[1] In this case, the stethoscope provides presence or absence of breath sounds as appropriate in the chest, axillae and epigastrum. The SimTools capnometer emulator (shown in Figure 2) uses a pressure sender placed in the airway of the model (Figure 3) to detect ventilations.

Figure 1: Customised Airway Trainer

Figure 2: Prototype display and breath detector

The modular Resus Anne system was chosen because it provides a full torso and is fairly easily upgraded. In particular, the oesophagus is sufficiently long and soft enough to allow the ODD to function as designed. A low resistance one-way valve was added to the end of the oesophagus to allow exhaust of ventilations yet give a positive indication with the ODD. (Figure 3) The exhaust of the one-way valve could be plumbed to the abdominal bladder to simulate stomach distention. An alternate method of deflation is necessary. Figure 4 shows the one-way valve in position next to the breath detector attached to the lung. A future modification would be to split the lung into left and right. Placement of the breath detector in the left lung helps to distinguish right mainstem bronchus intubation. Oesophageal intubation will not trigger the breath detector. When the lungs are ventilated, a wireless signal is sent to the display to trigger a colour change display to the clinician. Another wireless message signals the stethoscope to play the appropriate


615

sound as selected by the facilitator based on chestpiece placement and clinical situation. (Figures 3 & 4)

Figure 3: Chest view of airway and oesophagus

Figure 5: No CO2

Figure 4: Breath detector attached to trainer

Figure 6: CO2 detected

The colour of the central portion of the display smoothly changes from purple to tan with each breath. (Figures 5 & 6) The prototype uses a PDA as a display. Future models are planned which use a dedicated in-line display which sits between the ET tube and the ventilation device and looks much more like the real device. 3. Conclusion Resuscitation and anaesthetic guidelines have recommended several approaches to verification of endotracheal tube placement. We have surveyed the available methods and devices and summarised these guidelines in Table 2. Current part-task trainers do not allow the user to practice the full guideline of tube placement verification. Skipping steps during training can lead to omission of these steps during clinical practice. We have created a new part-task airway trainer that allows the entire procedure to be practiced. We are currently investigating various sequences of tube placement verification to determine one that will minimise the time to detection of an incorrect placement. 4. References [1]

Sprick, C.; Reynolds, K. J. & Owen, H. (2008), SimTools: a new paradigm in high fidelity simulation., Stud Helath Technol Inform 132(2008), 481-483 Published by IOS Press.

616


Mobile Three Dimensional Gaze Tracking Josef STOLL a,1 , Stefan KOHLBECHER b , Svenja MARX a , Erich SCHNEIDER b and Wolfgang EINHÄUSER a a Neurophysics, Philipps-University Marburg, Germany b Institute for Clinical Neurosciences, University Hospital, Munich, Germany Abstract. Mobile eyetracking is a recent method enabling research on attention during real-life behavior. With the EyeSeeCam, we have recently presented a mobile eye-tracking device, whose camera-motion device (gazecam) records movies orientated in user’s direction of gaze. Here we show that the EyeSeeCam can extract a reliable vergence signal, to measure the fixation distance. We extend the system to determine not only the direction of gaze for short distances more precisely, but also the fixation point in 3 dimensions (3D). Such information is vital, if gaze-tracking shall be combined with tasks requiring 3D information in the peri-personal space, such as grasping. Hence our method substantially extends the application range for mobile gaze-tracking devices and makes a decisive step towards their routine application in standardized clinical settings. Keywords. Mobile eyetracking, 3D gaze calibration, vergence eye movements

Introduction Gaze allocation in natural scenes has been a subject of research for nearly a century [1,2]. Possible applications reach from advertisement [1,3,4], over basic research to clinical applications [5,6,7]. Most experimental studies, however, measure eye movements in constrained laboratory settings. While such data have some predictive quality for gaze allocation in real-world environments, plenty of qualitative features remain unexplorable for principled reasons [8]. Recently, we have introduced a wearable eye-tracking device (EyeSeeCam) that allows recording gaze-centered movies while an observer pursues natural tasks in a natural environment [9]. Unlike stimuli presented on a screen, however, the real world is inherently 3D. Despite of research in virtual reality (VR), where eye trackers have been coupled with VR goggles [10,11] and in remote eye-tracking applications [12], most of today’s commercial eye tracking systems ignore this fact and restrict their calibration to one plane or use a recording setup that avoids parallax errors2 . Here we propose a solution that in addition yields distance information. To achieve robust 3D gaze-tracking, each eye needs to be represented in its own coordinate system under the constraint that the gaze directions of both eyes converge onto the fixation point. Fulfilling this condition allows the measurement of disjunctive eye movements, yielding a vergence signal for depth measurement. Here we present an extension 1 Corresponding

Author: Josef Stoll, AG Neurophysik, Philipps-Universität Marburg, Karl-von-Frisch-Str. 8a, 35032 Marburg, Germany; E-mail: [email protected]. 2 e.g., ISCAN, Woburn MA, USA, http://www.iscaninc.com

J. Stoll et al. / Mobile Three Dimensional Gaze Tracking

617

of the EyeSeeCam software that allows calibration in depth. Besides the identification of fixated targets in space, this allows the system to compensate for inevitable errors of parallax arising from the distance between the gaze-controlled camera-motion device (gazecam) and eyes. We quantify advantages in calibration accuracy and provide a proof of principle that eye-tracking can be used to tag objects in 3D space.

1. Methods EyeSeeCam Hardware The basic design of the EyeSeeCam has been described previously [13]. In brief: The EyeSeeCam consists of a binocular video-oculography (VOG) device and a head-mounted camera-motion device (gazecam), that is continuously oriented to the user’s point of regard by the eye movement signals . The gazecam captures images nearly identical to the user’s retina-centered visual stimulus, thus operating as an artificial eye. The whole apparatus is lightweight, mobile, battery-driven and controlled and powered by one laptop computer (Apple, MacBook). Altogether four digital cameras (Point Grey, Firefly MV) are integrated in the EyeSeeCam (Figure 1,B). The gazecam reaches the dynamic properties of the human ocular motor system - velocities above 500 deg/s and accelerations of up to 5000 deg/s2 - with a latency of 10 ms. The workspace lies in the range of ± 30 deg for horizontal and vertical movements [9]. For minimal user restriction and high orientation accuracy, a compact, light-weight, noiseless, and precise system is realized by a parallel kinematics setup with small piezo linear motors (Physik Instrumente, Germany). The camera platform, which is connected by a gimbal joint to the supporting head mount, gets rotated via two universal joints and push rods through two parallel displaced sleds, driven by the piezo actuators (modelled in figure 1C). Model-Free Gazecam Calibration In routine usage, the direction of the gazecam is aligned to the observer’s direction of view by the following procedure. The gazecam is moved towards 25 pre-defined locations on a 5x5 grid, whose central point is approximately aligned with the user’s gaze straight ahead. The user is asked to fixate the location a laser aligned with the optical axis of the gazecam pointer indicates. The mapping between known camera commands and VOG signal is interpolated and extrapolated to

Figure 1. Setup A) EyeSeeCam device; VOG cameras (600Hz, low-res) are visible to behind the hot mirrors; gazecam(752x480 pixels, 60Hz) on top; wide-angle head fixed camera (below gaze cam) is not used in the present experiment. B) Optical axes under two distance conditions; note the difference in eye vergence and parallax between gazecam and eyes. C) Simplified mechanical model simulated by CAD software; dependence between gazecam orientation and piezo sled position. By symmetry, the perpendicular position of the platform joint plane generates its zero position; actual shift by 0.0009 is the consequence from the push rod’s inclination.

618


arrive at a mapping valid for most of the visual field. Since the whole setup (including the laser pointer) moves with the head, this calibration method is insensitive to head movements and thus allows an accurate alignment of gaze and gazecam. Limitations of Model-Free Calibration In the model-free calibration, the mapping between VOG and camera-motion commands is computed directly, but no information is obtained on the angular orientation of the eye in its orbit. The validity of the calibration is restricted to the distance the laser pointer is projected to. This is tolerable if the distance is always large (i.e., virtually infinite) or all operations happen near this plane. Real-world tasks, however, often require switching between peri-personal and far space, such that a depth-corrected calibration is required. Similarly, parallax errors due to the inevitable misalignment of eye and gazecam need to be corrected in near space. Since such depth corrections require vergence information, thus orientation of the eye, the modelfree calibration approach is insufficient for parallax correction and 3D calibration. Eye-in-Head Calibration In routine usage, the gazecam calibration is complemented by an independent calibration for eye-in-head orientation [14]. This has to be done for each individual and session, as the device’s position relative to the observer’s head may vary when the device is taken off and back on. For this calibration, an additional headfixed laser is mounted between the eyes. The laser is equipped with a diffraction grating that generates a face-centered grid of dots on, e.g., a nearby wall (Figure 1B). The 4 first-order diffraction maxima have an angular distance of 8.5◦ to the central zerothorder maximum. This central laser projection defines the primary position, i.e., the origin of the gaze angle coordinates. By use of an eyeball model, these 5 gaze directions are integrated to map the VOG signal to the coordinates over the full calibration plane. So far, eye-in-head calibration and model-free calibration were performed independently. Although this allowed mutual verification of the two methods, a depth-correct allocation of the gazecam was impossible. Thus we here present a novel method that combines both strategies to arrive at a calibration and distance estimation in 3D space. Offset Correction To align the gazecam with the calibrated eye position, a second laser pointer is mounted in parallel to the optical axis of the gazecam. The offset of the gazecam is then adjusted by the experimenter through the graphical user interface such that this pointer matches the central (0th order) maximum of the projected calibration grid. This represents the uniquely adjusted origin of the gazecam coordinates plus a parallax correction matching the calibration distance. This procedure is independent from eye-in-head calibration. Only after offset correction is performed, the gazecam is set to follow gaze (i.e., is in tracking mode). During this usage of the EyeSeeCam the pointer can be used to verify the calibration against drift and to tag items of interest. 3D Calibration The 3D-information about the user’s fixation point is observable, if both eyes are calibrated in the same reference (coordinate) system. At fixation in infinite distance, the zero direction of each eye is parallel to the calibration laser grid’s central ray. For fixation at finite distances, a vergence eye movement adds to the zero direction. Since calibration must be performed at finite distances, we correct for the vergence angle occurring at each calibration point. By adding this angle, we adjust the coordinate systems such that both eyes point in parallel directions if their measured gaze coordinates are equal. For this correction, the relative positions between the eyes and the calibration points are needed and thus the projection distance and the relative position between the eyes and the source of the laser grid are required. While the distance between pupils has to be adjusted individually, inter-individual differences in the distance to the source of


619

the diffraction pattern, i.e., the laser, are negligible. Given the distance of the eyes to the laser and the application of the offset correction, the eye-in-head calibration yields a vergence angle as the difference of the spherical angles of both eyes according to the following calculation. Equal polar angles imply fixation at infinite distance. Any vergence angle greater than zero implies that rays originating from both eyes cross. The vergence angle is the inclination in the fixation point that is equal to the difference in both eyes polar angles. Due to the symmetry of the problem, the azimuth angle does not influence this computation. Using the polar angles and the interpupillary distance, we construct two linear equations whose solution is the fixation point in 3D space. Gazecam Vergence Correction Now we change the axis of symmetry and ignore the polar angle for the parallax correction. The gaze-direction in azimuth, the fixation distance b and the relative position of the gaze-camera’s gimbal joint allow spanning a triangle, whose unknown angle γ is the difference in azimuth between eyes and gazecam rays and equals the parallax correction - the gimbal joint lies on the optical axis of the gazecam. The distance between the center of the eyeballs and the gimbal joint is used in a. γ is the difference of the averaged gaze azimuth and the angle included by gaze zero direction and the direction from eyeballs center to the gimbal joint (Figure 1B). This problem is solved in plain trigonometry by formulas valid for oblique triangles, where γ a−b two sides a, b and their included angle γ are available. Thus, tan( α−β 2 ) = a+b cot 2 and α + β + γ = π yield the parallax correction: γ a−b α = π−γ 2 arctan( a+b cot 2 ). Parameterized Gazecam Positioning Distance variations of a user’s fixation imply an adaptation of the angle, which corrects for the parallax error of gazecam orientation. This requires the positioning of the gazecam to be implemented as a function of spherical angles that stays constant given the present mechanical conditions. It needs only the geometry of the systems to be known and thus is a one time measurement. The gimbal joint as the center of rotation coincides with the optical axis of the gazecam. This facilitates the mapping from a gazecam direction to the linear positions of the two piezo actuator sleds. The function is built on the holonomic constraints of the three-point coupling behind the camera platform, whose two universal joints are displaced by the piezo actuators via push rods. The transformation from angle to linear sled position replaces the previously employed point-by-point matching, but not the origin-offset adjustment, which depends on the individual fit of the EyeSeeCam on the user’s head. This means, the novel gazecam control still would need the angle relative to its zero position, in which the gazecam is oriented parallel to the primary eye position. The issue of offset-independent positioning is solved by a separation of the orientation into a vertical rotation followed by a horizontal rotation. First, the tilt given by the azimuth is virtually executed by a symmetric displacement of both piezo actuator bars, which is related by a simple sine mapping. Then the new positions of the pivots mounted on the camera platform are computed by projecting them on the plane of polar rotation, the pan. The resulting triangle gets rotated by the polar angle and the final positions of the universal joints are reached. Their projection on a linear parallel to the linear motor direction provides the asymmetric sled displacement. Additionally taking into account their perpendicular projection enables correction pushrod inclination. This approach simplifies the previous solution [15] and is easily adaptable to future systems with different geometries due to its parametric nature. The resulting mapping from direction angles to piezo actuator bar positions ensures an accurate

620


rotation of the gazecam. We verified the analysis with a 3D mechanical CAD program (SolidWorks, Dassault Systèmes, France).

2. Results To compare the novel calibration method to the previous one, we performed an experiment with predefined fixation targets. The projection distance during calibration was held constant at 2 m, alike in normal usage.The measurement process included fixations at 1, 2 and 4 m with a pattern of fixation targets, whose dot size and extension was scaled proportional to the distance. The dots were distributed on the corners and edge midpoints of squares, whose edge midpoints are oriented in the cardinal directions and have an angular distance of 2.4◦ - like the corners of the board game mill. To compare the accuracy of both calibration methods, the images from gazecam recordings were analyzed for deviations of the fixated target from image center. To quantify the parallax error, the statistics from the absolute vertical pixel deviation were exploited and plotted in angular degrees. We opposed the parallax error resulting from the former method to the vergence corrected error remaining when 3D calibrated. The old method shows a clear decrease of the vertical error with increasing distances (Figure 2A bottom, red diamonds). This parallax error could be diminished substantially, as the level of the vertical error is around 1◦ for all measured fixation distances (black circles). These results provide the proof of concept for a systematical vergence correction and mark a considerable increase for the accuracy with respect to the parallax error. To evaluate the overall performance of both calibration methods, the absolute value (direction independent) of the target-image-center deviation was analyzed. The mean over each eccentricity is plotted separately in Figure 2A, top. 3D calibration increases accuracy for the peri-personal range and also at large distances.

inner middle outer calibration square 1m

0.5

center


1.5 vertical error [deg]

vertical error [deg]

3.0

0

center

inner middle outer calibration square

0.5

B

total error [deg]

new old center

4m

3.0

0

center


1.5

center


0.5

C 4 2 d (m)

0

2m

3.0

vertical error [deg]

total error [deg]

1m

total error [deg]

A 3.0

1

0.5 0.25 center


0

5

10

t (s)

15

20

25

Figure 2. A) Accuracy Absolute error measured at validation points for different fixation distances (top of each panel) for four eccentricities (x–axes). Top: Absolute error; bottom: vertical component of error. Red diamonds: old calibration method; black circles: novel approach. Mean and sem over users (n=2), trials (n=3) and fixated points (n=8 for inner, middle and outer calibration square, 23 for center). Note: different scales for total and vertical error. B) User approaches a cube in space while fixating it, online display of distance, note the vergence correction; http://www.staff.uni-marburg.de/~einhaeus/ESC_3D.html for video C) estimated log fixation distance vs. walking time.


621

To investigate the 3D calibration in a truly 3D setting, we hang a small cube in space and ask one user to walk towards it. We observe an accurate measurement of fixated locations in 3D space (Figure 2B). The distances represent a usable estimator with an uncertainty of around 10% in the peri-personal range. During the approach the time course of estimated distance gets rather smooth for distances below 1m (Figure 2C). Above 1.5m, when eyes are almost parallel, the distance estimate gets increasingly instable and eventually looses in accuracy. Nonetheless, the robust and accurate estimation of 3D fixation distance in near space further validates our setup for use in tasks requiring operation in 3D space.

3. Discussion We present a novel calibration that uses angle calibrated binocular eye movements for 3D gaze tracking and extracts a reliable vergence signal that defines a 3D point for the gazecam’s target direction. In addition to fixation determination in 3D, the new procedure increases accuracy for the near field by analytic correction of parallax induced errors. The procedure simplifies calibration and makes it more efficient despite improvements in reliability and accuracy, which is of particular importance in clinical settings where large cohorts of patients have to be measured under strict time limitations. The uncertainty of fixation estimation is inversely proportional to the distance: distance is reliably estimated in the peri-personal range and gets imprecise only above 2m. Although far distances are worse to measure based on the vergence signal, this error does not affect gaze accuracy, as the vergence correction is then converging to zero. The nearer the fixation point, in turn, the lower is the uncertainty for the distance. Over all distances, performance is sufficient to capture human behavior as device accuracy (in terms of standard error) is below usual human variations over fixations, which are about 1◦ [16]. In addition, the human vergence system itself seems to be rather variable - the vergence angle varies between fixations, although the target position is the same. This variation seems to be increased after large shifts in depth of fixation, where vergence movements during slow disjunctive saccades arise. Humans are expected to have their fixation point sharply tuned only after about 800ms [17]. Thus, eye-movement dynamics influence the evaluable fixation distance considerably. By measuring vergence movements, we can apply our system to address vergence dynamics in a variety of realistic scenarios. Interacting with others in real-world space requires prediction of their intentions, which is often achieved by an estimate of the other’s gaze [18]; impairment of this function is fundamental to several clinical conditions in which social interactions or the perception of others’ intentions are impaired [19,20]. Using gaze direction as clinical tool [7] will thus be greatly fostered if combined with standardized clinical paradigms. As those often require action in peri-personal space or switching from peri-personal to far space, the 3D calibration presented here is inevitable for an eventual combination of gaze tracking with other tasks in standardized clinical settings. First tests in clinical populations with neuro pathological disorders (e.g., schizophrenia, Parkinson’s disease) have demonstrated the applicability of the EyeSeeCam in clinical settings. In addition, the EyeSeeCam has successfully been used as a training tool for surgeons by recording gaze-centered videos from a first person perspective [21]. Such videos intuitively visualize the experienced surgeon’s action planning. By applying the new calibration method the camera may be

622


actively focused at the correct distance thereby further improve the first-person feeling. Eventually, such application as teaching tool can extend well beyond surgery - for example towards dentists and anaesthesists [22] and even to other professions, like mechanics and engineers.

Acknowledgements We gratefully acknowledge support by the DFG under excellence cluster CoTeSys, BMBF (No. 01EO0901) (SK), grant EI 852/1 (WE), and research training unit 885 (JS).

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]

[15] [16] [17] [18] [19] [20] [21] [22]

G.T. Buswell, How People Look at Pictures: A Study of The Psychology of Perception in Art, The University of Chicago Press, 1935. A.L. Yarbus, Eye movements and vision, Plenum Press, New York, 1967. R. Carmi, L. Itti, The Role of Memory in Guiding Attention during Natural Vision, J Vis 9 (2006), 898-914. N. Höning et al., GoodGaze: Eine Technologie aus der Hirnforschung analysiert Webseiten auf ihre Aufmerksamkeitswirkung, up08 Lübeck, 2008. J. Vockeroth, K. Bartl, S. Pfanzelt, E. Schneider, Medical documentation using a gaze-driven camera, Stud Health Tech Informat 142 (2009), 413-416. Published by IOS Press. R.J. Leigh, C. Kennard, Using saccades as a research tool in the clinical neurosciences, Brain 127 (2004), 460-477. E.H. Pinkhardt et al., Differential diagnostic value of eye movement recording in PSP-parkinsonism, Richardson’s syndrome, and idiopathic Parkinson’s disease, J Neurol 255 (2008), 1916-1925. B.M. ’t Hart et al., Gaze allocation in natural stimuli: comparing free exploration to head-fixed viewing conditions, Vis Cog 17(6) (2009), 1132-1158. E. Schneider et al., EyeSeeCam: An eye movement-driven head camera for the examination of natural visual exploration, Ann N Y Acad Sci 1164 (2009), 461-467. A.T. Duchowski et al., Binocular eye tracking in VR for visual inspection training, VRST 8 (2001). G.P. Mylonas et al., Gaze-contingent soft tissue deformation tracking for minimally invasive robotic surgery, MICCAI (2005), 843-850. C. Hennessey, P. Lawrence, 3d point-of-gaze estimation on a volumetric display, Proceedings of the 2008 symposium on Eye tracking research & applications 59. J. Vockeroth et al., The combination of a mobile gaze-driven and a head-mounted camera in a hybrid perspective setup, IEEE SMC (2007), 2576-2581. T. Dera, G. Boning, S. Bardins, E. Schneider, Low-latency video tracking of horizontal, vertical, and torsional eye movements as a basis for 3dof realtime motion control of a head-mounted camera. IEEE SMC (2006). T. Villgrattner, H. Ulbrich, Piezo-driven two-degree-of-freedom camera orientation system. IEEE ICIT (2008), 1-6. T. Eggert, Eye movement recordings: methods, Dev Ophthalmol 40 (2007), 15-34. C. Rashbass, G. Westheimer, Disjunctive eye movements, J Physiol 159 (1961), 339-360. R. Stiefelhagen, J. Yang, A. Waibel, Estimating Focus of Attention Based on Gaze and Sound, ACM Int Conf Proc Series 15 (2001). A. Frischen, A.P. Bayliss, S.P. Tipper, Gaze Cueing of Attention, Psychol Bull 133(4) (2007), 694-724. M.J. Green, J.H. Waldron, M. Coltheart, Eye Movements Reflect Aberrant Processing of Social Context in Schizophrenia, Schizophr Bull 31(2) (2005) 470. E. Schneider et al., Documentation and teaching of surgery with an eye movement driven head-mounted camera: see what the surgeon sees and does, Stud Health Tech Informat 119 (2006), 486-90. C. Schulz et al., Eye tracking for assessment of workload: a pilot study in an anaesthesia simulator environment, Br. J. Anaesth. first published online October 30, (2010) doi:10.1093/bja/aeq307


623

High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions Hao SUa,1, Alex CAMILOa, Gregory A. COLEa, Nobuhiko HATAb Clare M. TEMPANYb and Gregory S. FISCHERa a Worcester Polytechnic Institute, Worcester, MA b Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

Abstract. This paper presents the design of a magnetic resonance imaging (MRI) compatible needle placement system actuated by piezoelectric actuators for prostate brachytherapy and biopsy. An MRI-compatible modular 3 degree-offreedom (DOF) needle driver module coupled with a 3-DOF x-y-z stage is proposed as a slave robot to precisely deliver radioactive brachytherapy seeds under interactive MRI guidance. The needle driver module provides for needle cannula rotation, needle insertion and cannula retraction to enable the brachytherapy procedure with the preloaded needles. The device mimics the manual physician gesture by two point grasping (hub and base) and provides direct force measurement of needle insertion force by fiber optic force sensors. The fabricated prototype is presented and an experiment with phantom trials in 3T MRI is analyzed to demonstrate the system compatibility. Keywords. MRI compatible robot, prostate brachytherapy, biopsy.

1. Introduction Prostate cancer continues to be the most common male cancer and the second most common type of cancer in human. The estimated new prostate cancer cases (192,280) in 2009 account for 25% incident cases in men [1]. The current "gold standard" transrectal ultrasound (TRUS) for guiding both biopsy and brachytherapy is accredited for its real-time nature, low cost, and ease of use. However, TRUS-guided biopsy has a detection rate as low as 20%-30% and the radiation seeds cannot be effectively observed on the images [2]. On the other hand, the MRI-based medical diagnosis paradigm capitalizes on the novel benefits and capabilities of the scanner. These are created by the combination of capability for detecting seeds, high-fidelity soft tissue contrast and spatial resolution. The challenges, however, arise from the manifestation of the bidirectional MRI compatibility requirement - both the device should not disturb the scanner function and should not create image artifacts and the scanner should not disturb the device functionality. Moreover, the confined physical space in closed-bore high-field MRI presents formidable challenges for material selection and mechanical design. Early MRI-compatible robots focus on manual driven or ultrasonic motor driven and the latter cannot run during imaging due to significant signal loss. Chinzei, 1

Corresponding Author. Hao Su, Automation and Interventional Medicine (AIM) Robotics Laboratory, Worcester Polytechnic Institute, Higgins Lab 130, 100 Institute Road, Worcester, MA 01609, USA. Tel.: +1508-831-5191; Fax: +1-508-831-5680; E-mail: [email protected].

624

H. Su et al. / High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions

et al. developed a general-purpose robotic assistant for open MRI [3] that was subsequently adapted for transperineal intraprostatic needle placement. Krieger et al. [4] presented a 2-DOF passive, un-encoded, and manually manipulated mechanical linkage to aim a needle guide for transrectal prostate biopsy with MRI guidance. Stoianovici et al. [5] described a MRI-compatible pneumatic stepper motor PneuStep, which has a very low level of image interference. Song et al. [6] presented a pneumatic robot for MRI-guided transperineal prostate biopsy and brachytherapy. However the scalability, simplicity, size and inherent robustness of electromechanical systems present a clear advantage over pneumatically actuated systems [7-9]. The difficulty arises from the actuator driving controller that usually induces significant image artifact using off-the-shelf control circuits [10-11]. Needle steering is becoming a practical technique to address needle placement accuracy issues in recent years. Mahvash et al. [12] have experimentally demonstrated that increased needle velocity is able to minimize tissue deformation and damage and reduce position error. To bridge the gap between MRI compatible mechatronics and needle steering techniques, the contributions of this paper are: (1) design of a modular needle driver that can be coupled to a base Cartesian motion platform to improve feasibility and accuracy of MRI-guided prostate interventions and their outcome and (2) experimental demonstration of real-time in-situ MRI compatibility and potential for multiple imager compatible surgery. To the authors’ knowledge, this is the first needle steering robot capable of operating in real-time MRI.

2. Methods and Materials In this paper, we presented a 3-DOF needle driver as slave robot to provide haptic feedback as shown in Fig. 1. The overall goal is to provide force feedback using fiber optic force sensor during interventional MRI-guided prostate interventions [13-14]. The primary design requirements and the features of the needle driver include: 1) 3-DOF motion needle driver. It provides cannula rotation and insertion (2DOF) and stylet translation (1-DOF). The independent rotation and translation motion of the cannula can increase the targeting accuracy while minimize the tissue deformation and damage. 2) Safety. Interventional robots require a redundant safety mechanism. Three approaches are implemented to minimize the consequences of system malfunction. a) Mechanical travel limitations mounted on the needle insertion axis that prevents linear motor rod running out of traveling range; b) Software calculates robot kinematics and watchdog routine that monitors robot motion and needle tip position; and c) Emergency power button that can be triggered by the operator. 3) MRI Compatibility. The robot components are primarily constructed of acrylonitrile butadiene styrene (ABS) and acrylic. Ferromagnetic materials are avoided. Limiting the amount of conductive hardware ensures imaging compatibility in the mechanical level. The piezoelectric driver has proven minimal image interference in the electrical level. 4) Operation in confined space. To fit into the scanner bore, the width of the driver is limited to 6cm and the operational space when connected to a base platform is able to cover the perineal area using traditional brachytherapy 60 mm × 60mm templates.


625

5) Sterilization. Only the needle clamp and guide (made of low cost plastic) have contact with the needle and are disposable. 6) Compliance with transperineal needle placement, as typically performed during a TRUS guided implant procedure. This design aims to place the patient in the supine position with the legs spread and raised with similar configuration to that of TRUS-guided brachytherapy.

Figure 1: (Left) System architecture for the master - slave haptic interface. The fiber optic force sensor and robot are placed near the iso-center of the MRI scanner, the master manipulator is connected to the navigation software interface, and the two are couple through the robot controller in the scanner room using a fiber optic network connection. (right) The robot prototype in the bore of a 3T MRI scanner with a phantom.

2.1. Needle Placement Robot Design The needle placement robot consists of a needle driver module (3-DOF) and Cartesian positioning module (3-DOF). The material is rapid prototyped with ABS and laser cut acrylic. Considering the supine configuration and the robot workspace, the width of the robot is limited to 6cm. As shown in Fig. 2 (left), the lower layer of the needle driver module is driven with linear piezoelectric motor and the upper layer provides cannula rotation motion and stylet prismatic motion. To design a needle driver that allows a large variety of standard needles, a new clamping device shown in Fig. 2 (right) rigidly connects the needle shaft to the driving motor mechanism. This structure is a collet mechanism and a hollow screw made of stereolithography ABS is twisted to fasten the collet thus rigidly locks the needle shaft on the clamping device. The clamping device is connected to the rotary motor through a timing belt that can be fastened by an eccentric belt tensioner. The clamping device is generic in the sense that we have designed 3 sets of collets and each collet can accommodate a width range of needle diameters. The needle driver is designed to operate with standard MR-compatible needles of various sizes. The overall needle diameter range is from 25 Gauge to 7 Gauge. By this token, it can not only fasten brachytherapy needle, but also biopsy needles and most other standard needles instead of designing some specific structure to hold the needle handle.

626


Figure 2: (Left) Physical prototype of 6-DOF piezoelectric needle placement robot consisting of needle driver module and Cartesian gross positioning module, (right) a exploded view of the needle clamping mechanism, optical tracking frame and rotary motor fixture with timing belt tensioner.

Once a preloaded needle or biopsy gun is inserted, the collet can rigidly clamp the cannula shaft. Since the linear motor is collinear with the collet and shaft, we need to offset the shaft to manually load the needle. We designed a brass spring preloaded mechanism that provides lateral passive motion freedom. The operator can squeeze the mechanism and offset the top motor fixture then insert the loaded needle through plain bearing housing and finally lock with the needle clamping. This structure allows for easy, reliable and rapid loading and unloading of standard needles. 2.2. Needle Placement Robot Navigation Dynamic global registration between the robot and scanner is achieved by passive tracking the fiducial frame in front of the robot as shown in Fig. 2 (right). The rigid structure of the fiducial frame is made of ABS and seven MR Spot fiducials (Beekley, Bristol, CT) are embedded in the frame to form a Z shape passive fiducial. Any arbitrary MR image slicing through all of the rods provides the full 6-DOF pose of the frame, and thus the robot, with respect to the scanner [7]. Thus, by locating the fiducial attached to the robot, the transformation between patient coordinates (where planning is performed) and the needle placement robot is known. To enhance the system reliability and robust, multiple slices of fiducial images are used to register robot position using principal component analysis method. The end effector location is then calculated from the kinematics based on the encoder positions. 2.3. Piezoelectric Actuator Driver The piezoelectric actuators (PiezoMotor, Uppsala, Sweden) chosen are non-harmonic piezoelectric motor which have two advantages over a harmonic drive: the noise caused by the driving wave is much easier to suppress, and the motion produced by the motors is generally at a more desirable speed and torque. Even though piezoelectric motor does not generate magnetic field, commercial motor driver boards usually induce significant image artifact due to electrical noise according to the most recent result [15]. A new low noise driver was developed and its architecture is shown in Fig. 3 (left) and Fig. 3 (right) shows the board prototype. Waveform tables are stored in RAM and utilized by a synthesizer running on the FPGA to generate four independent control waveforms of


627

arbitrary phase and frequency. These control waveforms are then streamed out to the analog amplification stage at 25 mega samples per second.

Figure 3: (Left) piezoelectric actuator driver architecture using FPGA generated waveform, (right) the piezoelectric driver board prototype, a key aspect of generating the low noise high precision motion.

3. Results Four imaging protocols as shown in Table 1, were selected for evaluation of compatibility of the system: 1) diagnostic imaging T1-weighted fast gradient echo (T1 FGE/FFE), 2) diagnostic imaging T2-weighted fast spin echo (T2 FSE/TSE), 3) highspeed real-time imaging fast gradient echo (FGRE), and 4) functional imaging spin echo-planar imaging (SE EPI). Details of the scan protocols are shown in Table 1. All sequences were acquired with a slice thickness of 5mm and a number of excitations (NEX) of one. Three configurations were evaluated and used in the comparison: 1) baseline of the phantom only, 2) motor powered with controllers DC power supply turned on and 3) system servoing inside MRI board. Three slices were acquired per imaging protocol for each configuration. Table 1: SCAN PARAMETERS FOR COMPATIBILITY EVALUATION Protocol T1W FFE T2W TSE FGRE SE EPI

FOV 240 mm 240 mm 240 mm 240 mm

TE 2.3 ms 90 ms 2.1 ms 45 ms

TR 225 ms 3000 ms 6.4 ms 188 ms

FA 75 90 50 90

Bandwidth 751 Hz/pixel 158 Hz/pixel 217 Hz/pixel 745 Hz/pixel

As can be seen in Fig. 4 (left), the motors and encoders provide very small visually identifiable interference with the operation of the scanner. Fig. 4 (right) depicts one slice of the tracking fiducial frame which provides the full position information of the robot. We utilize signal to noise ratio (SNR) as the metric for evaluating MR compatibility with baseline phantom image comparison. For comparison, the SNR of each configuration was normalized by the average SNR of the 3 baseline images for each imaging protocol. SNR was calculated as the mean signal in the center of the phantom divided by the noise intensity outside the phantom [10]. Statistical analysis with a Tukey Multiple Comparison confirms that no pair shows significant signal degradation with a 95% confidence interval.

628


Figure 4: (Left) Representative results showing the images obtained of baseline and system servoing inside scanner bore conditions, (right) one slice of tracking fiducial frame besides a phantom. This result presents significant improvement over recent research [15].

4. Discussion This paper presents the design of a MRI-compatible needle placement system actuated by piezoelectric actuators for transperineal prostate brachytherapy. It consists of a modular 3DOF needle driver module coupled with a 3-DOF x-y-z stage. Initial comparability testing verified the system architecture and electrical comparability. This test has confirmed that no pair showed significant signal degradation with a 95% confidence interval. Piezoelectric driven robot position control accuracy is being investigated. Future works include integrating the fiber optic sensors and phantom brachytherapy evaluation.

Acknowledgements We gratefully acknowledge the support from the Congressionally Directed Medical Research Programs Prostate Cancer Research Program New Investigator Award W81XWH-09-1-0191 and Worcester Polytechnic Institute internal funds.

References [1] A. Jemal, R. Siegel, E. Ward, Y. Hao, J. Xu, and M. J. Thun, “Cancer statistics, 2009,” CA Cancer J Clin, vol. 59, pp. caac.20006–249, May 2009. [2] J. C. Presti, “Prostate cancer: assessment of risk using digital rectal examination, tumor grade, prostatespecific antigen, and systematic biopsy.” Radiol Clin North Am, vol. 38, pp. 49–58, Jan 2000. [3] K. Chinzei and et al, "MR Compatible Surgical Assist Robot: System Integration and Preliminary Feasibility Study," in MICCAI 2000, 2000, pp. 921–930. [4] A. Krieger, C. Csoma, I. I. Iordachital, P. Guion, A. K. Singh,G. Fichtinger, and L. L. Whitcomb, “Design and preliminary accuracy studies of an MRI-guided transrectal prostate intervention system.,” MICCAI2007, vol. 10, pp. 59–67, 2007. [5] D. Stoianovici, D. Song, D. Petrisor, D. Ursu, D. Mazilu, M. Muntener, M. Mutener, M. Schar, and A. Patriciu, “MRI stealth robot for prostate interventions.,” Minim Invasive Ther Allied Technol, vol. 16, no. 4, pp. 241–248, 2007.


629

[6] S.E. Song, N. B. Cho, G. Fischer, N. Hata, C. Tempany, G. Fichtinger, and I. Iordachita, “Development of a pneumatic robot for MRI-guided transperineal prostate biopsy and brachytherapy: New approaches,” in Proc. IEEE International Conference on Robotics and Automation ICRA, 2010. [7] Fischer GS, Iordachita I, Csoma C, Tokuda J, DiMaio SP, Tempany CM, Hata N, Fichtinger G, MRICompatible Pneumatic Robot for Transperineal Prostate Needle Placement, IEEE / ASME Transactions on Mechatronics - Focused section on MRI Compatible Mechatronic Systems, Vol 13, No 3, pp 295305, June 2008. [8] Y. Wang, H. Su, K. Harrington and G. Fischer, “Sliding Mode Control of Piezoelectric Valve Regulated Pneumatic Actuator for MRI-Compatible Robotic Intervention”, ASME Dynamic Systems and Control Conference, Boston, USA, 2010 [9] H. Su and G. S. Fischer, “High-field MRI-Compatible Needle Placement Robots for Prostate Interventions: Pneumatic and Piezoelectric Approaches”, eds. T. Gulrez and A. Hassanien, Advances in Robotics and Virtual Reality, Springer-Verlag, to appear in 2011 [10] Y. Wang, G. Cole, H. Su, J. Pilitsis, and G. Fischer, “MRI compatibility evaluation of a piezoelectric actuator system for a neural interventional robot,” in Annual Conference of IEEE Engineering in Medicine and Biology Society, (Minneapolis, MN), pp. 6072–6075, 2009. [11] G. Cole, K. Harrington, H. Su, A. Camilo, J. Pilitsis, G. S. Fischer, “Closed-Loop Actuated Surgical System Utilizing In-Situ Real-Time MRI Guidance”, 12th International Symposium on Experimental Robotics (ISER2010), New Delhi & Agra, India, 2010 [12] M. Mahvash and P. Dupont, “Fast needle insertion to minimize tissue deformation and damage,” in Proc. IEEE International Conference on Robotics and Automation ICRA 2009, pp. 3097 – 3102, 2009. [13] H. Su and G. Fischer, “A 3-axis optical force/torque sensor for prostate needle placement in magnetic resonance imaging environments,” 2nd Annual IEEE International Conference on Technologies for Practical Robot Applications, (Boston, MA, USA), pp. 5–9, IEEE, 2009. [14] H. Su, W. Shang, G. Cole, K. Harrington, and F. S. Gregory, “Haptic system design for MRI-guided needle based prostate brachytherapy,” IEEE Haptics Symposium 2010, (Boston, MA, USA). [15] A. Krieger, I. Iordachita, S. Song, N. Cho, G. Fichtinger, and L. Whitcomb, "Development and Preliminary Evaluation of an Actuated MRI-Compatible Robotic Device for MRI-Guided Prostate Intervention," in Proc. of IEEE International Conference on Robotics and Automation (ICRA 2010)

630


Electromyographic Correlates of Learning during Robotic Surgical Training in Virtual Reality Irene H. SUH a,b, Mukul MUKHERJEE d, Ryan SCHRACK b, Shi-Hyun PARK d, Jung-hung CHIEN a,b,d, Dmitry OLEYNIKOV b,c, Ka-Chun SIU a,b,d,1 a

College of Public Health, b Center for Advanced Surgical Technology c Dept of Surgery, University of Nebraska Medical Center; d Nebraska Biomechanics Core Facility, University of Nebraska at Omaha, Omaha, Nebraska, USA.

Abstract. The purpose of this study was to investigate the muscle activation and the muscle frequency response of the dominant arm muscles (flexor carpi radialis and extensor digitorum) and hand muscles (abductor pollicis and first dorsal interosseous) during robotic surgical skills training in a virtual environment. The virtual surgical training tasks consisted of bimanual carrying, needle passing and mesh alignment. The experimental group (n=5) was trained by performing four blocks of the virtual surgical tasks using the da VinciTM surgical robot. During the pre- and post-training tests, all subjects were tested by performing a suturing task on a “life-like” suture pad. The control group (n=5) performed only the suturing task without any virtual task training. Differences between pre- and post-training tests were significantly greater in the virtual reality group, as compared to the control group in the muscle activation of the hand muscle (abductor pollicis) for both the suture tying and the suture running (p < 0.05). In conclusion, changes in electrographic activity shows that training in virtual reality leads to specific changes in neuromotor control of robotic surgical tasks. Keywords. Electromyography, Training, da VinciTM Surgical System, Simulation

1.

Introduction

Some of the problems commonly encountered in traditional laparoscopy are visual constraints and reduced dexterity [1]. Robotic surgical systems commonly used in minimally invasive surgery can overcome some of these drawbacks of traditional laparoscopy. Virtual Reality (VR) has been used to improve training for traditional laparoscopy and to give surgeons superior performance in the operating room [2]. VR simulations can also provide user-friendly, attractive and inexpensive environments to learn robotic surgical skills. In our previous research studies, we have shown that robotic surgical skills learning through VR simulations are comparable with real world surgical skill improvement tasks [3, 4, 5]. We implemented the VR simulations as part of a training 1

Corresponding Author: Email: [email protected] web: http://www.unmc.edu/cast/

I.H. Suh et al. / Electromyographic Correlates of Learning During Robotic Surgical Training in VR 631

program and determined the effect of learning on a common real-world surgical task, such as suturing. Despite significant increases in the popularity of robotic surgeries, current literature has seldom addressed the physiologic effects of training with a robotic surgical system, particularly muscle activity [6]. The purpose of this study was to investigate the muscle activation and the muscle frequency response of signals of dominant arm and hand muscles (flexor carpi radialis (FCR), extensor digitorum (ED), abductor pollicis (AP), and first dorsal interosseous (DI)) after VR training using the da VinciTM Surgical System (Intuitive Surgical, Sunnyvale, CA).

2.

Methods

Subjects Eleven young healthy student volunteers from the University of Nebraska Medical Center and the University of Nebraska at Omaha participated in this study. Subjects were randomly assigned to either the VR group or the control group. Experimental Protocol Subjects in the VR group performed the three tasks in four blocks. In each block, each of the three tasks was performed five times. The order of tasks was randomized within each block. The Webots software (Cyberbotics, Lausanne, Switzerland) was used to build the VR environment which was driven by kinematic data streaming in real-time from the operating console of the da VinciTM surgical system. Subjects in the control group performed only the pre- and post-test before and after a gap of 2.5 hours (the average time to complete the VR training). Training Tasks Subjects who were in the VR training group performed three tasks in a virtual environment (Figure 1): bimanual carrying (BC), needle passing (NP) and Mesh Alignment (MA). (A)

(C)

(B)

Figure 1. The surgical tasks in the virtual environment: (A) Bimanual Carrying, (B) Needle passing, (C) Mesh Alignment.

632 I.H. Suh et al. / Electromyographic Correlates of Learning During Robotic Surgical Training in VR

For the BC task, the volunteers simultaneously transferred two plastic pieces in opposite directions five times consecutively. The NP task required the passing of a surgical needle through six pairs of holes. In the MA task, a virtual rolled-up mesh was opened up by the simulated arms of the robot and placed on a pre-marked virtual task platform. These tasks were designed to mimic real-life surgical skills training in terms of their cyclic nature (BC task), decision-making skills (determining location of touch sensors to unroll the mesh in the MA task) and grasping, transferring, and release skills in both BC and NP tasks. Testing Task For the testing task, all subjects performed three trials of the following procedure repairing an enterotomy on a life-like suture pad (Figure 2). This procedure consisted of using the da VinciTM Surgical System for making three single knots (suture tying), five running passes (suture running) followed by three single knots again between predefined locations on the suture pad. The two surgical skill components (suture tying and suture running) were used for data analysis.

Figure 2. Repairing an enterotomy on the life-like suture pad: One subject’s performance at the Pre-testing (left) and Post-testing (right)

Data Collection and Analysis The electromyography (EMG) data was collected using the Delsys Myomonitor® Wireless EMG system (Delsys, Inc., Boston, MA) and was sampled at 1,000 Hz using the EMGworks Acquisition software (Delsys, Inc., Boston, MA). Surface electrodes were placed over the bellies of the following four muscles of the dominant forearm (FCR and ED) and hand (AP and DI). The EMG signals were then analyzed according to Narazaki et al. [7] and Judkins et al. [8]. To quantify the extent of muscle activation, the normalized EMG signals (i.e., percentage of raw EMG outputs relative to maximal EMG output) for each muscle in each trial were integrated for the entire task completion time, and the total volume of muscle activation (EMGv) was obtained. Moreover, the activation rate (EMGr) was calculated by dividing EMGv by time. Frequency-domain analysis of EMG provides a window into muscle fatigue and motor unit recruitment [9]. Raw EMG was first filtered using a 2nd order Butterworth band-pass filter from 20-300 Hz. Raw EMG was then converted to the frequency domain using a Fast Fourier Transform to determine the power spectrum. Median frequency and frequency bandwidth were computed from the power spectrum. Median frequency was computed as the frequency at half of the integrated power spectrum as given by the following equation [9]:

I.H. Suh et al. / Electromyographic Correlates of Learning During Robotic Surgical Training in VR 633

Eq.(1)

where P(f) is the power at frequency f, fmed is the median frequency (Fmed), and fmax is the maximum frequency of the power spectrum. Frequency bandwidth (Fband) is the difference between the highest and lowest frequency where the power exceeds half the maximum power of the power spectrum. The differences between the pre- and post-training tests were calculated for statistical analysis. The relative differences were also calculated between the pre- and post-training tests for the descriptive statistics of each muscle. Statistical Analysis Separate independent t-tests were applied for both suture tying and suture running of the testing task. Dependent variables: EMGv, EMGr, Fmed and Fband

3.

Results

Our results showed that the differences between pre- and post-training tests were significantly greater in the VR group, as compared to the control group for the EMGv of the AP muscle for both suture tying and the suture running (p < 0.05) (Figure 3). For the FCR muscle, the EMGv showed a significantly greater difference but only for suture running (Figure 4).

0.25

* 0.2

FCR Right

CTRL VR

*

0.15

0.1

0.05

Differences between Pre - Post training test EMGv (mV sec)

Differences between Pre - Post training test EMGv (mV sec)

AP Right

CTRL

VR

0.06

*

0.05 0.04 0.03 0.02 0.01 0

0

Suture Tying

Suture Running

Figure 3. The hand muscle (abductor pollicis (AP)) activations between pre- and post-training tests in suture tying and running

Suture Tying

Suture Running

Figure 4. The forearm muscle (flexor carpi radialis (FCR)) activations between pre- and post-training tests in suture tying and running

634 I.H. Suh et al. / Electromyographic Correlates of Learning During Robotic Surgical Training in VR

The relative differences in EMGr for the AP muscle were 26% for control group and 55% for VR group in suture tying. For suture running, the relative differences were 28% and 57% for control and VR groups respectively. For the FCR muscle, the relative differences in EMGr were 10% for control group and 3% for VR group in suture tying. For suture running, the relative differences were 20% and 32% for control and VR groups respectively. No differences were found in both EMGv and EMGr for DI and ED muscles as well as in both Fmed and Fband between two groups for all muscles.

4.

Conclusions

Our results are highly encouraging in indicating that training with simulated surgical tasks may result in improvement of actual surgical skills. These results were also reflected in our previous study [10], in terms of the differences in the speed of the instrument tips between the pre and post-training tests on the suture pad. In that study, the VR group had shown a higher increase in speed in both aspects of the surgical task. We had shown these improvements previously in kinematic parameters [6] and through this study. We demonstrated that these improvements also have electrophysiological correlates, especially in those muscles (AP and FCR) which seem to be the primary contributor for controlling the telemanipulators of the surgical system. Further improvement of the VR environment can enhance the learning effect even more.

5.

Acknowledgement

This work was supported by the Nebraska Research Initiative and the Center for Advanced Surgical Technology at the University of Nebraska Medical Center.

References [1]

J.D. Hernandez, S.D. Bann, Y. Munz, K. Moorthy, V. Datta, S. Martin, A. Dosis, F. Bello, A. Darzi and T. Rockall, Qualitative and quantitative analysis of the learning curve of a simulated surgical task on the da Vinci system. Surgical Endoscopy 18 (2004), 372-378. [2] A.G. Gallagher, E.M. Ritter, H. Champion, G. Higgins, M.P. Fried, G. Moses, C.D. Smith, R.M. Satava, Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 241 (2005), 364-72. [3] D. Katsavelis, K.C. Siu, B. Brown-Clerk, I. Lee, Y.K. Lee, D. Oleynikov, N. Stergiou, Validated robotic laparoscopic surgical training in a virtual-reality environment. Surg Endosc. 23 (2009), 66-73. [4] B. Brown-Clerk, K.C. Siu, D. Katsavelis, I. Lee, D. Oleynikov, N. Stergiou, Validating advanced robotassisted laparoscopic training task in virtual reality. Stud Health Technol Inform. 132 (2008), 45-9. [5] M.J. Fiedler, S.J. Chen, T.N. Judkins, D. Oleynikov, N. Stergiou, Virtual reality for robotic laparoscopic surgical training. Stud Health Technol Inform.125 (2007), 127-9. [6] T.N. Judkins, D. Oleynikov, N. Stergiou, Electromyographic response is altered during robotic surgical training with augmented feedback. J Biomech. 5 (2009), 71-6. [7] K. Narazaki, D. Oleynikov, N. Stergiou, Robotic surgery training and performance: identifying objective variables for quantifying the extent of proficiency. Surg Endosc, 20 (2006), 96-103. [8] T.N. Judkins, D. Oleynikov, K. Narazaki, N. Stergiou, Robotic surgery and training: electromyographic correlates of robotic laparoscopic training. Surg Endosc. 20 (2006), 824-9. [9] Basmajian & De Luca., Muscles alive, their functions revealed by electromyography, Williams & Wilkins, Baltimore, MD, 1985 [10] M. Mukherjee, K.C. Siu, IH Suh, A. Klutman, D. Oleynikov, N. Stergiou, A virtual reality training program for improvement of robotic surgical skills. Stud Health Technol Inform. 142 (2009), 210-4.


635

Web-Based Interactive Volume Rendering Stefan SUWELACK, Sebastian MAIER, Roland UNTERHINNINGHOFEN and Rüdiger DILLMANN Institute for Anthropomatics, Karlsruhe Institute of Technology, Germany Abstract. In this paper we present a web-based remote visualization system. The system makes use of video stream based techniques to reduce the network bandwidth requirements and is capable of performing interactive volume rendering on computed tomography data in real-time. The technique allows embedding interactive volume rendering into a website. The concrete contribution of this paper is twofold. First, we outline a Microsoft Silverlight based implementation of the prototype and describe the applied video encoding techniques. Furthermore we present experimental results that allow evaluating the system in terms of latency and image quality. In particular, we show that the additional delay of stream based remote visualization is very small if compared to picture based techniques. Keywords. interactive remote visualization, volume rendering, web-based visualization

1. Introduction Current medical visualization and simulation techniques often have demanding hardware requirements. A typical example is the volume rendering of computed tomography (CT) data. In order to perform interactive volume rendering on portable devices such as netbooks or tablet PCs the actual rendering has to be done on a remote server [1]. Most systems use compressed images to deliver the rendered data to the client [3]. The amount of data that has to be streamed over the network can be significantly reduced by using video encoding techniques [2]. The disadvantages of this method are the additional encoding time for the video stream and the additional computational effort for the decoding on the client side. In this paper we present a stream-based remote visualization system whose client is based on the Microsoft Silverlight application framework. Thus no additional software has to be installed on the client as even complex medical visualizations can be delivered as web-based content. Additionally the GPU based video decoding capabilities of MS Silverlight reduce the computational load of the client. We analyze the system with respect to encoding time and image quality for the remote visualization of volume rendered CT images. 2. Methods Web-Based Video Streaming The system can encode the data either as a H.264 or a WMV1 video stream. The encoding parameters for both formats have been optimized for real-time encoding. In particular no

636

S. Suwelack et al. / Web-Based Interactive Volume Rendering

Figure 1. Close-ups of a volume rendered CT image with a resolution of 512x512 pixels. The image is encoded using the H.264 algorithm (middle column) and the WMV1 algorithm (right column) at bitrates of 1000kbit/s (upper row) and 250 kbit/s (lower row).

bidirectional frames are allowed. On the client side the video images are extracted and the video stream is decoded natively using the Silverlight GPU accelerated algorithms, thus keeping the client hardware requirements low. Remote Visualization Pipeline The complete system can be integrated into the visualization toolkit (VTK) [4]. The mouse and keyboard events which trigger the update of the visualization pipeline are transferred from the client to the server using a custom protocol. The realtime transport protocol (RTP) is used for video transmission. On the client side the encoded video stream is delivered to a MS Silverlight MediaElement through the MediaStreamSource API. The server component that handles the socket connections to the clients allows transmitting several video streams per client. GPU Raycasting The volume rendering is performed using GPU based ray casting. In order to increase the computational efficiency the algorithm adapts the sampling step size based on a spectral analysis of the dataset and the transfer function [5].

3. Results The system was evaluated in terms of image quality and latency. It has to be noted that the encoded video frames differ in quality and size. In particular there is a huge difference between keyframes and predicted frames (P-frames). We address this problem by averaging the performance measurements over 1000 frames. In order to assess the quality of the images we compare typical non-keyframe sequences. The results of the performance measurements are displayed in Table 1. We point out that the encoding time is very low for the WMV1 algorithm (9ms for a resolution of 768x768 pixels). In comparison the encoding using H.264 compression takes more than

S. Suwelack et al. / Web-Based Interactive Volume Rendering

637

Table 1. Encoding time for the H.264 and WMV1 algorithms and image resolutions of 512x512 and 768x768 pixels. Delay of the system for different networks and image resolutions (far right column). Video resolution

Encoding time (ms)

Network delay

@ bit rate

WMV1

H264

(ms)

5122 @1000 kbit/s

3.8

24.0

1.1

7682 @2500 kbit/s

9.0

57.0

2.0

LAN

5122 @1000 kbit/s 7682 @2500 kbit/s

3.8 9.0

24.0 57.0

38.7 37.3

W-LAN

5122 @1000 kbit/s

3.8

24.0

122.3

7682 @2500 kbit/s

9.0

57.0

165.3

5122 @1000 kbit/s 7682 @2500 kbit/s

3.8 9.0

24.0 57.0

209.4 282.5

Localhost

Internet

six times as long. Table 1 also reveals that the delay introduced by the WMV1 encoding is nearly negligible if compared to the network delay. The difference in image quality between the two encoding algorithms is clearly visible when low bandwith settings are used (see Fig. 1). Although the H.264 encoding gives better results in all scenarios, the difference in image quality is low for high bandwidth streams. 4. Conclusions We presented a novel approach for web-based remote visualization of volume rendered CT images. The system allows embedding interactive volume rendering into a website as a MS Silverlight application. A detailed analysis shows that the encoding time is much smaller than the network delay. We thus conclude that the additional delay of stream based remote visualization is very small if compared to picture based techniques. Furthermore, stream based techniques significantly reduce the bandwith requirements and can deliver high quality content over end-user internet connections. Future work includes the reduction of the H.264 encoding time by using GPU accelerated algorithms or special purpose hardware. Also, we are currently integrating the presented system into an online anatomy learning tool which features volume rendered images to improve the learning experience of the students. References [1] [2] [3]

[4] [5]

K. Engel, O. Sommer, and T. Ertl. A framework for interactive hardware accelerated remote 3dvisualization. In Proceedings of EG/IEEE TCVG Symposium on Visualization. Citeseer, 2000. F. Lamberti and A. Sanna. A streaming-based solution for remote visualization of 3D graphics on mobile devices. IEEE Transactions on Visualization and Computer Graphics, 2007. B. Paul, S. Ahern, E.W. Bethel, E. Brugger, R. Cook, J. Daniel, K. Lewis, J. Owen, and D. Southard. Chromium renderserver: Scalable and open remote rendering infrastructure. IEEE Transactions on Visualization and Computer Graphics, 2008. Will Schroeder, Kenneth M. Martin, and William E. Lorensen. The visualization toolkit (2nd ed.): an object-oriented approach to 3D graphics. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1998. Stefan Suwelack, Eric Heitz, Roland Unterhinninghofen, and Rüdiger Dillmann. Adaptive gpu ray casting based on spectral analysis. In Medical Imaging and Augmented Reality, Lecture Notes in Computer Science, pages 169–178. Springer Berlin / Heidelberg, 2010.

638


A Method of Synchronization for Haptic Collaborative Virtual Environments in Multipoint and Multi-level Computer Performance Systems Kazuyoshi Tagawa a,1, Tatsuro Bito a, and Hiromi T. Tanaka a a Ritsumeikan University, Japan

Abstract. We have developed a novel volume-based haptic communication system. It allows participants at remote sites on a network to simultaneously interact with the same target object in virtual environments presented by multilevel computer performance systems. It does this by only exchanging a small set of manipulation parameters for the target object and an additional packet to synchronize the status of the binary tree and the deformation of the local region of the shared volume model. We first developed online-remesh volume models, which we call dynamic adaptive grids, to simulate deformable objects such as soft tissues at remote sites. Then, haptic sensation during interaction with the target object was achieved by rendering the reflection force from the object, which was simulated with the online-remesh volume model from the manipulation parameters and additional packets exchanged among all remote sites. We investigated the efficiency of our system via experiments at remote locations on a WAN between Tokyo, Osaka, and Kyoto. Keywords. Elastic object, haptic collaborative virtual environment, multi-level computer performance systems

Introduction Virtual reality technology has improved as computers and networks have become faster and more powerful. Some telecommunication systems that allow users to work at remote locations on the network have been developed, and medical and educational applications are expected. The VizGrid project [1] has enabled the volume modeling of output data to be developed from simulation or experimental results and the volume modeling of images in the real world by using a multiple-view camera. Hikichi et al. [2] proposed a system of haptic collaboration without loss of quality of service (QoS). They conducted an experiment to evaluate their system using a rigid and a surface object, and the delay time, packet loss, and information loss were measured. Mortensen et al. [3] presented a study on remote collaboration between people in a shared virtual environment. Two people, one at University College London (UCL) and the other at the University of North Carolina, Chapel Hill (UNCCH), met together in the shared virtual environment 1

[email protected]

K. Tagawa et al. / Synchronization for Haptic Collaborative Virtual Environments

639

and lifted a rigid body object together and moved it to another place. Gunn et al. [4] proposed techniques allowing long-distance sharing of haptic-enabled, dynamic scenes. At the CSIRO Virtual Environments Laboratory, they have successfully used their system to connect a prototype of a surgical-simulation application between participants on opposite sides of the world in Sweden and Australia, over a standard Internet connection. However, previous work did not achieve realistic sensations for the representational model or real-time performance, and the sense of touch was not well defined. We propose a novel system of volume-based haptic communication in this paper, which allows participants at remote sites on the network to simultaneously interact with the same target object in virtual environments presented by multi-level computer performance systems by only exchanging a small set of manipulation parameters for the target object and an additional packet to synchronize the status of the binary tree and the deformation of the shared volume model. We first developed an online-remesh volume model, which we called a dynamic adaptive grid, to simulate deformable objects such as soft tissues at remote sites. Then, haptic sensation during interaction with the target object was achieved by rendering the reflection force from the object, which was simulated with the online-remesh volume model from the manipulation parameters exchanged among all remote sites. Finally, we investigated the efficiency of our system via experiments at remote locations on a WAN between Tokyo, Osaka, and Kyoto.

1. Online-Remesh Deformation Model We developed a mesh generator in our previous work [5], where an input mesh model such as an organ was represented using a binary tree of a set of tetrahedrons without any cracks being formed. The model was based on a tetrahedral adaptive mesh for the parallel hierarchical tetrahedralization of volume data. We used a dynamic tetrahedral adaptive grid of volume data [6] in the onlineremesh deformation model to rapidly simulate deformation of a visco-elastic object. This algorithm could refine this tetrahedral adaptive mesh interactively. Figures 1 and 2 outline examples of binary refinement and simplification.

Figure 1. Binary refinement and simplification

Figure 2. Examples of online-remesh

640


2. Synchronization Method 2.1. Overview We proposed a volume-based haptic communication system [7] to present the same deformation to all users, which allowed participants at remote sites on the network to simultaneously interact with the same target object in virtual environments by only exchanging a small set of manipulation parameters for the target object. Each PC had the same deformation model, and then the same manipulation parameters were input to all deformation models. As a result, the same simulation results were obtained and presented to the users. This method enabled the target object’s deformation to be shared by enabling small packets (manipulation parameters) to be exchanged via the network. However, if multi-level systems of computer performance are used in the communication system, interaction is apt to be unstable because of the long time-delays caused by late simulation on low-spec PC(s). To solve this problem, we extended the communication system. In the approach overviewed in Figure 3, additional packets to synchronize the status of the binary tree and the deformation of the local region (Figure 4) of the shared volume model are exchanged.

Figure 3. Overview of our synchronization method

Figure 4. Definition of local region

2.2. Send/Receive Packet The manipulation parameters (position, orientation, time stamp, and simulation time) are exchanged at a rate of 1 kHz. Additional packets are sent and received according to need in three steps (where location ID subject to synchronization is 0): Step 1 Step 2

Step 3

Exchange manipulation parameters. Search location_max, where location_max is a location ID of a PC that has maximum simulation time SimTimelocation _ max . If SimTime ࠉ0 SimTime location _ max ! D then Send an additional packet to location_max else if SimTime location _ max SimTime 0 ! D then Receive an additional packet from location_max

641


In Step 3, D is the threshold of difference in the simulation time. The additional packet contains the status of the binary tree and the deformation of the local region. Details on this packet and synchronization method are described in the following subsections. 2.3. Synchronization of Binary Trees The binary trees are synchronized in four steps (where location ID subject to synchronization is 0). Step 1

The status of the binary trees are expressed by bit sequences to reduce the amount of communications traffic. As shown in Figure 5, we define tetrahedrons that belong to the binary tree as 1, otherwise 0.

Step 2

These bit sequences are exchanged via the network.

Step 3

By using logical operation, the difference T0c

between a bit

sequence of binary tree T0 and bit sequences of other binary trees

T1 ,, TN 1 sent from other locations is obtained as:

T0c (T0 T1 TN 1 ) T0 .

(1)

There is an example of this logical operation in Figure 6. Step 4

Add tetrahedrons indicated by T0c to T0 .

We can obtain a synchronized and crack-free (without any cracks forming) binary tree at low computation cost and with low communications traffic by using this algorithm.

Figure 5. Coding of binary tree

Figure 6. Synchronization of binary trees

2.4. Synchronization of Deformation Tetrahedrons that belong to a large deformation region are subdivided locally in the online-remesh deformation model. These tetrahedrons appear at deep leaf nodes of the binary tree as the subdivision levels of these subdivided-tetrahedrons increase. In our approach to synchronization, deformations of these tetrahedrons are shared with other locations as follows:

642


Step 1

Begin at user’s manipulating tetrahedrons.

Step 2

Search adjacent tetrahedrons recursively to find which subdivision levels are E and over where E is the threshold of the subdivision level.

Step 3

Send IDs, positions, and velocities of nodes that consist of retrieved tetrahedrons to location_max.

3. Experiment We conducted an experiment to evaluate how effective our method of synchronization was by using a liver model of initial level 3 (the number of initial nodes was 96) and this level changed to level 6 (the number of nodes was 3757). Displacements based on a sine function were given as the input for three nodes, and then coordinates of one arbitrary point at each location were measured as shown in Figure 7.

Figure 7. Experimental model

3.1. Experimental Conditions Three multi-level computer performance PCs were used as shown in Table 1. Highspec. PC was allocated in Tokyo, middle-spec. PCs in Osaka, and low-spec. PCs in Kyoto. These PCs were connected with a peer-to-peer wide area network (WAN) in which we used a TCP/IP connection and network delay emulation software to emulate the JGN2plus network between Tokyo, Osaka, and Kyoto. The average round trip times were about 10 [msec] between Tokyo and Osaka and between Tokyo and Kyoto, and about 1 [msec] between Kyoto and Osaka. Table 1. Specifications of PCs Low-spec. PC Middle-spec. PC High-spec. PC

CPU Intel Xeon X5355 2.6 GHz Intel Core i7 920 2.6 GHz Intel Core i7 975 3.3 GHz

OS CentOS 5.5 CentOS 5.5 RedHat Enterprise Linux 6.0 beta

Memory 8 GB 10 GB 12 GB

3.2. Experimental Results Figures 8 and 9 plot the trajectories of nodes measured at locations with asynchronous and synchronization methods. Figure 10 is a close-up of Figure 9. The solid lines, short-dashed lines and long-dashed lines correspond to trajectories measured at low-


643

spec., middle-spec. and high-spec. PCs. The trajectories of each pair were more similar in the synchronous method (Figures 9 and 10) than those in the asynchronous method (Figure 8), because the differences in coordinates were periodically modified. Table 2 lists the average errors in coordinates where the threshold of difference in simulation time ( D ) was changed to 0.01, 0.05, 0.1, 0.3, and 0.5. The average errors in coordinates were reduced by using the synchronous method. Figure 11 shows examples of interaction force. Discontinuous force was observed in Figure 11(c); however, each author’s subjective opinion is that unpleasant force was not felt.

Figure 8. Deformation (asynchronous)

Figure 9. Deformation (synchronous,

Figure 10. Deformation (synchronous,

D

=0.1,

E =3, closeup)

Table 2. Average errors in coordinates [mm] Threshold of difference in simulation time ( D )

0.01

0.05

0.1

0.3

0.5

Low-spec. PC Middle-spec. PC

0.002 0.0007

0.01 0.003

0.05 0.004

0.37 0.02

0.63 0.04

D =0.1, E =3)

644


(a) High-spec. PC

(b) Middle-spec. PC

(c) Low-spec. PC Figure 11. Interaction force (synchronous,

D

= 0.1,

E = 3)

4. Conclusion We described a volume-based system of haptic communication that shares an adaptive volume model between remote locations and provides haptic communication to users. The model of shared volume in virtual environments was presented by multi-level computer performance systems and was shared by only exchanging a small set of manipulation parameters and additional packets to synchronize the status of the binary tree and the deformation of the local region of the model. The efficiency of the proposed algorithm was confirmed through experiments at three remote locations on a WAN between Tokyo, Osaka, and Kyoto.

References [1] VizGrid Project, Final Report of Vizgrid Project. Technical report, Japan, 2007. [2] K. Hikichi, H. Morino, I. Fukuda, S. Matsumoto, K. Sezaki, and H. Yasuda. Proposal and evaluation of system for haptics collaboration. Journal of The Institute of Electronics, Information and Communication Engineers, J86-D(2):268. 278, 2003. [3] J. Mortensen, V. Vinayagamoorthy, M. Slater, A.Steed, B. Lok, and M. Whitton. Collaboration in TeleImmersive Environments. In 8th EurographicsWorkshop on Virtual Environments, pages 93.101, 2002. [4] C. Gunn, M. Hutchins, and M. Adcock. Combating latency in haptic collaborative virtual environments. Presence: Teleoperators and Virtual Environments, 14(3):313.328, 2005. [5] Y. Takama, A. Kimura, and H. T. Tanaka. Tetrahedral adaptive mesh for parallel hierarchical tetrahedralization of volume data. Journal of The Information Processing Society of Japan, 48(SIG 9(CVIM 18)):67.78, 2007. [6] Y. Takama, H. Yamashita, and H. T. Tanaka. Dynamic tetrahedral adaptive mesh generation of volume data. In Symposium on VC/GCAD2007. [7] S.Yamaguchi, H. T. Tanaka. Toward Real-time Volume-based Haptic Communication with Realistic Sensation. Proc. of IEEE/RSJ 2007 Intr. Conf. on Intelligent Robots and Systems(IROS2007) Workshop on Modeling, Identification, and Control of Deformable Soft Objects, pp.97-104, 2007.


645

A Hybrid Dynamic Deformation Model for Surgery Simulation Kazuyoshi TAGAWAa,1 and Hiromi T. TANAKA a a Ritsumeikan University, Japan

Abstract. A hybrid dynamic deformation model that is capable of haptic interaction with dynamically deformable objects is presented. This approach is also capable of taking into account large deformation and topological changes of objects. The amount of computation required in this approach is proportional to the sum of the square of the number of shared nodes and the number of whole nodes of an online-remesh model. We implemented the approach in our prototype system and confirmed the feasibility and performance through experiments. Keywords. Elastic object, deformation model, impulse response deformation model, haptic rendering

Introduction Haptic interaction with elastic objects is an important topic in the field of haptics. The greatest difficulty in such interaction is the large computation cost required to calculate the object’s deformation and the user’s reaction force. Recently, the use of multi-core processors (e.g. multi-core CPU or GPU) and novel deformation models (e.g., a record reproduction model) [2, 3, 5] have enabled haptic interaction with relatively complex objects. However, operations that involve topological changes (e.g., cutting or rupturing operations) of such objects are still difficult. We present here a hybrid deformation model as an approach to solve the above problems. This approach is capable of simulating large deformation and topological changes of dynamically deformable objects at a haptic rate.

1. Related Work 1.1. Deformation Model Researchers in the field of virtual reality often prefer to use the mass-spring model (MSM) and an explicit method to solve the model [7] because it is easy to change boundary conditions. However, the relationship between the parameters of the model and the physical properties of the target object is not clear.

1

E-mail: [email protected]

646

K. Tagawa and H.T. Tanaka / A Hybrid Dynamic Deformation Model for Surgery Simulation

In contrast, the finite element method (FEM) [9] is based on continuum dynamics. However, we have to solve large simultaneous equations; thus, the amount of computation of FEM is very large. 1.2. Acceleration Method for Simulation of Deformable Object Of course, some fast computation methods have been proposed. Kikuuwe et al. proposed a computationally efficient formulation and an algorithm for tetrahedral finite-element simulation of elastic objects subject to the Saint Venant-Kirchhoff (StVK) material law [4]. However, in cases where the model is solved by an implicit method, it is difficult to take topological changes into consideration. In previous papers, the authors proposed an online-remesh model that uses tetrahedral adaptive mesh [10]. Moreover, the authors proposed a deformation model based on a record reproduction approach which we call the impulse response deformation model (IRDM) [5]. In IRDM, the computation amount of calculation of the interaction force is not dependent on the model complexity. This is advantageous for haptic interaction; however, there are two problems. One is that the model is based on the idea of linear assumption. The other is that it is difficult to represent topological changes. This is because we must retain a large number of impulse responses (a combination of patterns of the user’s interaction and topological changes). 1.3. Rupture Model Kume et al. proposed a rupture model for a cholecystectomy simulator [6]. This is a FEM-based soft tissue destruction model that behaves according to variable tearing manipulation. However, haptic feedback is difficult because of the computation cost of recalculating a stiffness matrix. Moreover, this model is a static model; therefore, the dynamic feature of deformation is unavailable. As an approach to solve the problem above, Hirota et al. [1] and Cotin et al. [8] proposed a tensor mass model (TMM) which uses the finite element model and an explicit method. In the approach, it is easy to change boundary conditions; however, numerical computation of the model is apt to be unstable. In general, internal organs subject to surgery are connected to other organs by blood vessels or adhesion. In addition, deformation and topological change operations are applied to a specific portion of the organ. Cotin et al. proposed a hybrid deformation model [8]. In their approach, the organ is simulated by combining the TMM (deformation and topological change operations are applied) and linear FEM models. However, use of the online-remesh model was not discussed, and the dynamic feature of deformation was unavailable.

2. Hybrid Deformation Model In this paper, a hybrid deformation model is presented. In this model, IRDM and the online-remesh model are combined via shared nodes in order to perform coupled analysis. To do this analysis, we define shared nodes in each of the two models, as shown in Fig. 1. The hybrid deformation model is calculated according to the following procedure:


647

0.

Obtain the position of the stylus of the force-feedback device.

1.

Calculate internal forces of all nodes of the online-remesh model.

2.

Send the present internal forces of the shared nodes of the online-remesh model as the present external forces of shared nodes in IRDM.

3.

Calculate displacements of shared nodes of IRDM using a convolution integral [5].

4.

Receive displacements of shared nodes of the online-remesh model from displacements of shared nodes of IRDM.

The numbers of shared nodes in IRDM and the online-remesh model are not equivalent because the tetrahedral adaptive mesh in the online-remesh model is locally subdivided based on the maximum shear strain of tetrahedrons. Therefore, we used interpolated values based on the initial positions of the shared nodes. This calculation is easy because tetrahedrons are recursively subdivided equally.

Figure 1. Hybrid deformation model.

3. Experiments We implemented our approach with our prototype system and confirmed the feasibility and performance through experiments. 3.1. Experimental Setup As shown in Fig. 2, we employed three rectangular objects that each had a width and breadth of 128 mm and 32 mm. The height of each model was 128 mm, 192 mm, and 256 mm, respectively. The bottom nodes of each object were attached to the floor. We assigned the online-remesh model to the upper side of each object. The number of nodes of the online-remesh model of each object at the initial mesh was 135, and the number of shared nodes of the models was 27. In IRDM, there were 135, 243, and 351 nodes for the respective objects. The impulse responses were obtained by FEM. The sampling frequency of the impulse response was set to 500 Hz, and the response duration was set to 1 s. We employed Young’s modulus E=2000N/m2, Poisson’s ratio =0.49, and density =110kg/m3. In the online-remesh model, we employed MSM. Young’s modulus E=10000N/m2 and density =100000kg/m3. In some part whose height was from quarter to middle of

648


the online-remesh model, Young’s modulus was multiplied by 0.1 in order to rupture easily. We used a PC (CPU: Core i7 3.3GHz, Memory: 12GB, OS: Linux) and a PHANTOM Omni device.

Figure 2. Experimental models.

3.2. Experimental Results Fig. 3 shows an example of dynamic deformation and topological change of an object done by user manipulation. The user pulled up a node on the upper surface of the object because the superior half of the model was the online-remesh model; tetrahedral elements with large deformation were locally subdivided. The computation time of the online-remesh model was 0.35 ms. Compared to this, the computation time of IRDM of each model was 9.6 ms. Computation times of IRDM of these objects were about the same because the computation cost of IRDM is not dependent on the complexity of the model (i.e., the number of whole nodes of an object) but is dependent on the number of shared nodes of the object.

Figure 3. Example of dynamic deformation and topological change.


649

3.3. Acceleration by GPU Parallelization of computation of IRDM is possible because the algorithm of IRDM consists of sets of convolution integrals. Therefore, we tested the acceleration of the computation using a graphics processing unit (GPU). The computation time of IRDM was 0.11 ms; for graphics cards, we used nVIDIA GeForce GTX 295 and CUDA 3.1. This speed was sufficiently fast for haptic interaction.

4. Conclusion This paper introduced a novel deformation model that can consider both dynamic behavior and topological changes of elastic objects. A simple shape of a hybrid model was implemented, and through experiments, we confirmed that the approach was feasible and had good performance.

References [1] K. Hirota and T. Kaneko. A study on the model of an elastic virtual object. Trans. of the Society of Instrument and Control Engineers, 34(3):232–238, 1998. [2] Doug L. James and Kayvon Fatahalian. Precomputing interactive dynamic deformable scenes. Proc. ACM SIGGRAPH 2003, pages 879–887, 2003. [3] K. Hirota and T. Kaneko. Haptic representation of elastic object. Presence, 10(5):525–536, 2001. [4] R. Kikuuwe, H. Tabuchi, and M. Yamamoto. An edge-based computationally efficient formulation of saint venant-kirchhoff tetrahedral finite elements. ACM Trans. on Graphics, 28(1):8:1–8:13, 2009. [5] K. Tagawa, K. Hirota, and M. Hirose. Impulse response deformation model: an approach to haptic interaction with dynamically deformable object. Proc. IEEE Haptic 2006, pages 209–215, 2006. [6] N. Kume, M. Nakao, T.Kuroda, H. Yoshihara, and M. Komori. Simulation of soft tissue ablation for a vr simulator. Trans. of Japanese Society for Medical and Biological Engineering, 43(1):76–84, 2005. [7] A. Norton, G. Turk, B. Bacon, J. Gerth, and P. Sweeney. Animation of fracture by physical modeling. Visual Computer, 7:210–219, 1991. [8] Cotin Stëphane, Delingette Hervë, and Ayache Nicholas. A hybrid elastic model for real-time cutting, deformations, and force feedback for surgery training and simulation. The Visual Computer, 16:437– 452, 2000. [9] G. Yagawa and S. Yoshimura. Computational Dynamics and CAE Series 1: Finite Element Method. Baifu-kan, Tokyo, 1991. [10] S. Yamaguchi and H. T. Tanaka. Toward real-time volume-based haptic communication with realistic sensation. Proc. of IROS2007, pages 97–104, 2007.

650


Single and Multi-User Virtual Patient Design in the Virtual World D TAYLOR MSc1, V PATEL MRCS, D COHEN MRCS, R AGGARWAL PhD, K KERR PhD, N SEVDALIS PhD, N BATRICK FRCS and A DARZI PC KBE HonFREng FMedSci Division of Surgery, Department of Surgery and Cancer, Imperial College London

Abstract. This research2 addresses the need for the flexible creation of immersive clinical training simulations for multiple interacting participants and virtual patients by using scalable open source virtual world technologies. Initial development of single-user surgical virtual patients has been followed by that of multi-user multiple casualties in a field environment and an acute hospital emergency department. The authors aim to validate and extend their reproducible framework for eventual application of virtual worlds to whole hospital major incident response simulation and to multi-agency, pan-geographic mass casualty exercises. Keywords. Virtual Patient, Virtual Worlds, Medical Training, eTraining, Major Incident Response, Simulation, Mass Casualty, Trauma, Emergency Response

Background Virtual Patients are computer simulations designed to train or assess clinicians in information gathering, diagnostic reasoning and management of individual patients[1]. The majority of online Virtual Patient designs have focused on a single user or group interaction with a single patient. The aim of this research was to develop a series of complex Virtual Patients for both single and multi-user simulations, using a reproducible design methodology and subsequently to validate their use in clinician and emergency responder training and assessment. Its novelty lies in the use of open source based technology and Second Life®/Opensim[2] for the flexible development of scalable multi-user and multi-Virtual Patient scenarios.

1 Corresponding

Author: Division of Surgery, Imperial College London, 10th Floor QEQM, St Mary's Hospital, South Wharf Road, London W2 1NY, UK; E-mail: [email protected]

2

Acknowledgements: Phase 1 was supported by the London Deanery under their Simulation Technology-enhanced Learning Initiative (STeLI). Phase 2 was supported by funding from the Health Protection Agency. Aggarwal and Sevdalis are supported by the UK’ s National Ins titute for Health Research through a Clinician Scientist award and the Imperial Centre for Patient Safety and Service Quality, respectively.

651

D. Taylor et al. / Single and Multi-User Virtual Patient Design in the Virtual World

1. Method The development of the Virtual Patients was undertaken in two phases. Phase 1 involved the development of a series of single Virtual Patients (Figure 1) for postgraduate surgical trainees to individually assess and manage. Phase 2 focused on the development of mass casualty scenarios for multiple simultaneous emergency responders and multi-disciplinary teams to assess and treat. Clinical decision trees to control Virtual Patients' responses were modelled using an editor based on an open source framework and subsequently compiled to a web player that was developed to maintain the state of multiple Virtual Patients in an open source and scalable architecture. A message broker was developed to communicate between the virtual world (flexibly Second Life or the open source equivalent, Opensim) and the web player (Figure 2). In the second phase, 3 existing modalities of training were studied: field training for HART paramedics[3] responding to an incident involving hazardous materials, an Emergo Train tabletop[4] exercise for hospital emergency departments - approved by the Department of Health as an acceptable alternative to a live exercise, and a desktop mass casualty exercise involving multiple agencies. A training needs analysis was undertaken through semi-structured interviews with trainers and users who had experienced 1 or more of the 3 exercise types. Subsequently an integrated exercise scenario was conceptualised by a multi-disciplinary team so as to meet the identified training needs, particularly those that were not adequately or costeffectively met by the 3 existing exercise modalities.

Second Life or Opensim Broker

Virtual World Client

VPs and devices

Avatars

Figure 1 Virtual Patient in Ward

Figure 2 System Architecture

Web Browser

Servlet

Web Server

VP data: Logic and Decision Trees

Open Source Application Framework MySQL Database

Editor

652

D. Taylor et al. / Single and Multi-User Virtual Patient Design in the Virtual World

Figure 3 HART Paramedics in Hazmat Suits

Figure 4 A Bomb Blast Casualty

2. Results For the first phase, 3 general surgical Virtual Patients in 3 different ward settings (Figure 1) at 3 training levels were created. These are undergoing face and construct validity testing with both junior and Consultant grade surgeons [3]. For the second phase, 3 aspects of the scenario (referred to as vignettes) were designed and implemented and their face and content validity assessment is now being undertaken with paramedics and hospital practitioners. The 1st vignette is set in a simulated field environment involving hazardous materials (Figures 3 and 4), where paramedics and other first responders could undergo concurrent training and assessment. The other 2 vignettes are set in a simulated emergency department receiving multiple casualties.

3. Discussion A reproducible framework for both single and multi-user virtual patient development was achieved. Face and content validity testing is being carried out separately for each phase 2 vignette but it is clear that the technology could support multiple teams at different virtual sites exercising at the same time in a fully integrated exercise. This might include multiple agencies and teams at the site of the incident, several hospitals and regional control centres, each managing their resources as in a real mass casualty event.

References [1] [2] [3] [4]

Triola, M.M., et al., An XML standard for virtual patients: exchanging case-based simulations in medical education. AMIA Annu Symp Proc, 2007: p. 741-5. http://opensimulator.org/wiki and http://education.secondlife.com/ (both accessed on 14 July 2010). http://www.ambulancehart.org.uk/about_hart/ (accessed on 14 Oct 2010). http://www.emergotrain.com (accessed on 14 Oct 2010).


653

Terahertz Imaging of Biological Tissues Priyamvada TEWARIa,b, Zachary. D. TAYLORa,b,1, David BENNETTa,b , Rahul. S. SINGHa,c,g, Martin O. CULJATa,b,c,g, Colin P. KEALEYc, Jean Pierre HUBSCHMANd, Shane WHITEe, Alistair COCHRANf, Elliott . R. BROWNg, Warren S. GRUNDFESTa,b,c, a Center for Advanced Surgical and Interventional Technology (CASIT) b Biomedical Engineering IDP, cDepartment of Surgery, dJules Stein Eye Institute, e School of Dentistry, fDepartment of Pathology, University of California, Los Angeles; g Dept. Electrical and Computer Engineering, University of California Santa Barbara

Abstract. A reflective THz imaging system sensitive to small variations in water concentrations has been developed. Biological tissues such as skin, eyes and teeth were imaged to ascertain the systems response to tissue hydration. Difference in water concentrations translated to contrast in the THz images. Contrast was also seen in THz images of skin cancer and burns suggesting the potential diagnostic capability of THz imaging system in clinical settings. All specimens analyzed were freshly excised ex-vivo tissues. These encouraging preliminary results have motivated us to explore the in vivo potential of our imaging system. Keywords. Terahertz imaging, hydration, skin burns, dental, corneas

1. Introduction The terahertz band occupies the part of the electromagnetic spectrum between infrared and microwaves spanning the frequency range 0.3 THz – 3 THz. Properties like strong absorption and reflection by water, non-ionizing photon energy, low loss transmission through common fabrics, and reduced scatter compared to infrared and better resolution than microwave have motivated the exploration of THz radiation for medical imaging [1-2]. Studies have reported the capability of THz in identifying inflamed skin regions, cancers, burns, stratum corneum hydration profiling, and early detection of dental caries [3-4]. These findings coupled with recent advances in instrumentation have motivated the development of medical THz imaging systems [5-6]. From a clinical standpoint, reflective imaging systems are more practical since they allow in vivo assessment of body surface abnormalities and may provide contrast currently unavailable with existing medical imaging modalities. We have developed a reflective THz imaging system that operates at a center frequency of 0.525 THz with a bandwidth of 0.125 THz [7]. Previous results obtained by our group suggest the capability of our system in imaging biological tissues and generating a contrast between areas differing in water concentrations [7-8]. This paper provides a summary of preliminary experiments done on various biological samples to explore the sensitivity of our imaging to hydration gradients in a range of biological tissues. 1

Corresponding Author: Zachary. D. Taylor, [email protected]

654

P. Tewari et al. / Terahertz Imaging of Biological Tissues

2. Results 2.1. Skin Grafts Skin grafts play an important role in the closure of large wounds, and in reconstructive surgery. The thickness and type of skin grafts is dictated by procedural requirements. Hydration state assessment of skin grafts before and after implantation can be useful in monitoring the success of functional and cosmetic assimilation [9]. To gauge the ability of our system to assess hydration in skin grafts, different layers of skin were imaged. Porcine skin was obtained and sliced into grafts of varying thicknesses using a dermatome. Specimens were placed face-down onto a polypropylene sample mount, and imaged using the THz system (Figure 1).

Figure 1. (Left) Representative skin graft data scan, with metal tape and specimen’s labeled, and (Right) average THz reflectivity as function of skin thickness

THz reflectivity increased monotonically with increasing graft thickness. It has been experimentally shown that the deeper dermal layer of skin is more hydrated than the superficial epidermis [10]. Our findings are consistent with these earlier studies and demonstrate the ability of our system to detect small differences in water content between different layers of the skin. These findings suggest the utility of THz imaging as a method for assessing pre- and post-operative graft hydration levels, potentially alerting clinicians to at-risk grafts prior to failure. 2.2. Skin Burns 1.25 million patients are treated for burns annually in the United States with burn injuries being the fifth leading cause of injury-related deaths [11]. Since accurate assessment of burn extent and depth is important for making clinical decisions, the need for a noninvasive imaging tool is immense. For evaluating the capability of our imaging system in visualizing skin burns, fresh porcine skin was obtained and sectioned into pieces 1” x 1”. A brass brand in the shape of cross was heated to 350°C and pressed against the skin for 5-7 sec to induce a full thickness burn. The sample was then mounted on a polypropylene mount and imaged.

Figure 2. (Left) visible picture, and (Right) THz image of full thickness burn on porcine skin


655

The cross-shaped burnt area is clearly visible in the THz image. A contrast is obtained between the burnt and unburnt area where lighter shades of gray correspond to areas of higher reflectivity. Burning involves local evaporation of water making the burnt areas relatively dehydrated as compared to the surrounding normal areas in skin. There was a 98.4% drop in reflectivity between burnt and unburnt area. The SNR of the image was calculated as 17 dB. Scattering due to skin surface roughness contributed to the overall variance seen in the image. This result is in accordance with a previous published experimental result [12]. 2.3. Skin Cancer Tumors are associated with higher water content than normal tissues. Since THz imaging is so sensitive to water and changes in absorption and refractive indices, which are found to vary between normal and malignant tissues, a difference in THz reflection is expected to generate contrast [4,13]. A skin scalp biopsy sample with recurrent melanoma was procured with the permission of UCLA Institutional Review board. The specimen was placed on a polypropylene mount and imaged.

Figure 3. (Left) Skin scalp biopsy specimen, and (Right) THz image

The red outline in the THz image denotes the outer boundary of the sample delineating it from the polypropylene mount. The sample gives an overall lower THz reflectivity as compared to the mount. Further contrast is seen within the specimen (Figure 3). The dark pigmented lesion in the center of the scalp is found to be relatively less reflective to THz as compared to the surrounding scalp. Areas of higher reflectivity (outlined in red) close to the lesion in normal skin boundary are also observed in the THz image. In order to demonstrate the clinical utility of terahertz technology, we are working closely with pathologists to develop procedures that map histological findings to terahertz images. 2.4. Tooth Enamel and Dentin A horizontal cross-section of a molar was studied; this comprised of a ring of surface enamel encircling deeper dentin. The tooth was fully hydrated and embedded in epoxy resin for imaging (Figure 4). Enamel and dentin contrasted markedly. The yellow region in the 2-D image corresponds to the thin outer ring of enamel surrounding the dentin. Darker regions within the red area can be mapped to thinner areas of circumpulpal dentin. Even though the shape of the underlying subsurface pulp horn shape in the dentin was not exactly replicated in the image, different dentin regions in the sample can be differentiated. The THz imaging contrast between enamel and dentine is due to a difference in refractive indices. Higher THz reflectivity is obtained from enamel (3.1), which has a higher refractive index than dentine (2.6) [14]. The THz image obtained delineates enamel from dentin suggesting that our system can be used to image the dentino-enamel

656


junction. Previously most of THz images have been obtained from dry samples. The tooth sample was fully hydrated at the time of embedding so as to simulate inherent the moistness of teeth in vivo.

Figure 4. (Left) Tooth sample encased in epoxy, and (Right) corresponding THz image

2.5. Cornea Hydration profiling of cornea can help in early detection of corneal edema, trauma, inflammation, corneal endothelial cell pathology and monitoring laser ablation rate during corrective surgeries [15]. The following experiment was done to see whether THz imaging could play a role in hydration detection and monitoring in cornea. Pig eyes were obtained and corneal flaps ~ 130 µm were sliced from the eyes using a microkeratome. Flaps were placed on the polypropylene mount with an Al strip on one side to calibrate the system. The height of stage was adjusted to point of maximum reflectance on the top surface. Setting the maximum as the origin, line scans were taken across the corneal flaps with a step size of 0.5 mm at a bias of 30 V. Each line scan crossed over polypropylene, porcine cornea, and aluminum (Figure 5).

Figure 5. (Left) 2-D image of cornea, (Middle) horizontal cut, and (Right) vertical cut through middle of cornea

A radially varying hydration profile is evident from line scans of the corneal flap. The point of maximum reflectance was found to be coincident with the center of cornea. Reflectance decreased from the origin to the edges, suggesting that center of cornea was more hydrated than the edges. The edges were found to have similar reflectivites, indicating symmetry in the corneal structure [16]. Though the results are obtained by forcing a curved corneal flap onto a flat scanning surface they are promising, and in concert with previous experimental observations. We expect improved accuracy with the development of systems for imaging curved surfaces.

3. Conclusions/Future direction Preliminary results indicate that a reflective THz imaging system is able to generate a contrast in biological specimens between regions differing in water concentrations. This can be used not only in distinguishing between tissues but also in identifying diseased states. The successful imaging of ex-vivo tissues like skin and eyes along with the non-invasive nature of THz radiation opens up many potential clinical uses of THz imaging technology. The next step is repeating some of the above listed experiments on


657

live animals. The Animal Research Committee at UCLA has approved our project. In vivo THz imaging is currently underway.

4. Acknowledgements The authors would like to thank CASIT collaborators in UCSB for their support of this project, along with CASIT collaborators at UCLA: Dr. Jean-Louis Bourgeis, Dr. Benjamin Burt, Dr Matthew DeNicola and student researchers Jon Suen, Shijun Sung and Ashkan Maccabi. The authors most gratefully appreciate the funding provided by the Telemedicine and Advanced Technology Research Center (TATRC)/ Department of Defense under award W81XWH-09-Z-0017 and the National Science Foundation under grant number IHCS-801897.


[4]

[5] [6] [7] [8] [9] [10] [11] [12] [13] [14]

[15] [16]

J.E Bjarnason, T.L.J Chan, A.W.M Lee, A Celis & E.R Brown, “Millimeter-wave, terahertz, and midinfrared transmission through common clothing,” Applied Physics Letters, 85(4) (2004), 519-521 P.H Siegel, “Terahertz technology in biology and medicine”, IEEE transactions on microwave theory and techniques, 52(10) (2004), 2438-2446, A.J Fitzgerald, E Berry, N.N Zinv’ev, S Homer-Vanniasinkam, R.E Miles, M Chamberlain & M.A Smith, “Catalogue of Human Tissue Optical Properties at Terahertz Frequencies ,” J. Bio. Phys. 129 (2003), 123-128 R.M Woodward, B.E Cole, V.P Wallace, R.J Pye, D.D Arnone, E.H Linfield & M Pepper, “Terahertz pulse imaging in reflection geometry of human skin cancer and skin tissue,” Phys. Med. Biol. 47, 38533863 (2002). D.R. Grischkowsky & DM Mittleman, “Introduction. In: Mittleman DM, editor. Sensing with terahertz radiation. Berlin, Heidelberg, New York: Springer ( 2003), 1-38 Z.D. Taylor, R.S. Singh, M.O. Culjat, J.Y. Suen, W.S. Grundfest, E.R Brown, “THz imaging based on water-concentration contrast,” Proc . of SPIE 6949 (2008) R.S. Singh, Z.D. Taylor, M.O. Culjat, W.S. Grundfest & E.R. Brown, “Towards THz Medical Imaging; Reflective Imaging of Animal Tissues,” MMVR 16 (2008) Z.D. Taylor, R.S. Singh, M.O. Culjat, J.Y. Suen, W.S. Grundfest, H Lee & E.R. Brown, “Reflective terahertz imaging of porcine skin burns,” Optice letters 33(11) ( 2008) D.Ratner, “Skin grafting,” Seminars in cutaneous mediucine and surgery 22(4) (2003), 295-305 K. Martin, "In vivo measurements of water in skin by Near-Infrared Reflectance," Applied spectroscopy 52(7) (1998), 1001-1007 Burn incidence and treatment in the US: 2007 fact sheet. D.M .Mittleman, M Gupta, R Neelamani, R.G. Baraniuk, J.V. Rudd & M Koch, “Recent advances in THz imaging,” Applied Physics B: Lasers and Optics 68 (1999), 1085-1094 R.M. Woodward, B Cole, Wallace V.P, D.D. Arnone, Pye R, Linfield E.H, Pepper M & Davies A.G. “Terahertz pulse imaging of in-vitro basal cell carcinoma samples.” CLEO (2001); 329-330. D.A. Crawley, C. Longbottom, B.E. Cole, C.M. Ciesla, D. Arnone, V.P. Wallace & M. Pepper, "Terahertz pulse imaging: A pilot study of potential applications in dentistry," Caries Research 37 (2003), 352-359 S Mishima, “Corneal thickness,” Surv Ophthalmol 13 (2) (1968),57-96 R. S. Singh, P. Tewari, J.L. Bourges, J.P. Hubschman, D.B. Bennett, Z.D. Taylor, H. Lee, E.R. Brown, W.S. Grundfest, M.O. Culjat, "Terahertz Sensing of Corneal Hydration," Proc. of IEEE EMBS 2010

658


Quantifying Surgeons’ Vigilance during Laparoscopic Operations Using Eyegaze Tracking Geoffrey TIEN a,1 , Bin ZHENG b and M. Stella ATKINS a a School of Computing Science, Simon Fraser University, Canada b Department of Surgery, University of British Columbia, Canada Abstract. The vigilance of surgeons while operating is an important consideration for patient safety. Using a lightweight mobile eyegaze tracker, we can objectively observe and quantify a surgeon’s vigilance measured as the frequency and duration of time spent gazing at an anaesthesia monitor displaying various patient vital signs. Expert surgeons and training surgical residents had their eyegaze recorded while performing a mock partial cholecystectomy on a computer simulator. Results show that experts glanced at the patient vital signs more than the residents, indicating a higher level of surgical vigilance. Keywords. Laparoscopic surgery simulator, Head-mounted eyegaze tracker

Introduction Vigilance is the state of being watchful to avoid danger. In an operating room (OR) setting, surgical vigilance can be extended to encompass awareness of potential dangers to a patient. A high level of mental judgment ability inclusive of awareness of patient condition is an important part of ensuring patient safety [1,5,6]. When observing surgical performance in the OR, it is noticeable that the senior surgeon usually keenly detects signs that may concern patient safety. But little is known whether vigilance is associated with a surgeon’s competency in performing the surgical procedure. The first goal of this study is to examine the relationship between vigilance and surgical skills. To achieve this first goal, we asked surgeons with a wide range of surgical experience to perform a laparoscopic procedure in a simulated environment. We chose laparoscopy due to a simple fact that a sufficient level of vigilance can be more difficult to maintain in a laparoscopic setting where only a part of the surgical field is visible by a video display from an endoscope, and additional mental processing is needed to maintain orientation of the patient anatomy, further compounded by the increased difficulty of precisely controlling the laparoscopic instruments compared to open surgery. A problem with observing the vigilance of surgeons is the lack of a method of measuring this skill. To this end we propose to use eyegaze tracking as an approach to objec1 Corresponding Author: Geoffrey Tien, School of Computing Science, Simon Fraser University, 8888 University Drive, Burnaby, B.C. Canada; E-mail: [email protected].

G. Tien et al. / Quantifying Surgeons’ Vigilance During Laparoscopic Operations

659

tively quantify surgical vigilance. Our second goal is to prove the value of using eyetracking in a surgical context, based on our earlier work showing how the eyegaze of novices and experts differ in a virtual laparoscopic training environment [3]. We hypothesize that as surgical experience increases, cognitive effort in performing the primary surgical task will decrease, hence freeing attentional resources to observe the patient condition. In this study we aim to track surgeons’ eye movements during a mock laparoscopic procedure and to use this as a measure of awareness of changes in a patient’s condition displayed on a simulated anaesthesia monitor.

1. Method 1.1. Apparatus The study was conducted in the surgical skills training lab at the Centre of Excellence for Surgical Education and Innovation (CESEI) of the University of British Columbia (UBC). Two high-fidelity simulators were used to create patient scenarios. The first, a SurgicalSim VR manufactured by Medical Education Technologies, Inc. (METI) provided the main visual and tactile interface of the apparatus. This PC-based simulator includes a set of slender tools and a foot pedal to mimic the form and function of laparoscopic instruments, and a 17" LCD monitor as the simulated laparoscopic display. SurgicalSim was used to create a virtual surgical training environment for our participants to perform a partial cholecystectomy. A separate MacOS-based METI Emergency Care Simulator (ECS) includes a lifesized pneumatically controlled mannequin, whose simulated vital signs such as heart rate (with audible beep), blood pressure, and blood oxygen saturation were displayed on a 15" LCD monitor placed to the right side of the main SurgicalSim VR display. It is important to note that the ECS and SurgicalSim VR systems were placed closed to each other creating a sensation for the participant that they were operating on one single patient; however, the ECS and SurgicalSim VR do not communicate with one another. Finally, eyegaze tracking was accomplished by a head-mounted PT-Mini system manufactured by Locarna Systems, Inc. The PT-Mini headgear consists of two linked video cameras—one aimed at the wearer’s eye, and one facing forward to capture the scene relative to the wearer’s head. The two video feeds were saved to a portable notebook computer for post-processing. The components of the experimental apparatus are shown in Figure 1. 1.2. Task The experimental task required a surgeon to hold a grasper and a monocautery hook to dissect the gall bladder from the liver. A foot pedal placed on the ground controlled cautery. For each participant, the partial cholecystectomy exercise was performed on the SurgicalSim VR under two different patient conditions. One patient presented a stable heartbeat controlled by ECS, while the other patient’s heartbeat became slightly erratic at set intervals. Because ECS and SurgicalSim VR are unlinked, changes in patient condition on ECS do not alter the scene on SurgicalSim VR.

660

G. Tien et al. / Quantifying Surgeons’ Vigilance During Laparoscopic Operations

(a)

(b)

Figure 1. (a) METI SurgicalSim VR and ECS, (b) Locarna PT-Mini headgear.

1.3. Participants Participants included surgical residents, laparoscopic fellows, and attending surgeons from the surgery department at UBC. A pre-test questionnaire was administered to gather demographic data and to measure their laparoscopic surgical experience score as detailed by Zheng et al [6]. 1.4. Procedure After signing their consent to participate and completing the pre-test, each participant was allowed to complete the simulated partial cholecystectomy once without the patient vitals to learn the characteristics of operating the SurgicalSim VR. Each participant then put on the Locarna headgear and was guided through a short calibration procedure to ensure that his eye could be reliably tracked across the scene camera’s field of view. Participants then performed the partial cholecystectomy task once for each of the two patient conditions, for a total of 2 trials. The patient histories were presented on a printed sheet of paper before each trial. The order of the patient conditions was counterbalanced across participants. 1.5. Data processing and analysis Eyegaze was recorded over the duration of each trial of the cholecystectomy task and analyzed using Locarna’s Pictus Eyegaze Calculation software. Eyegaze fixation detection was done using a dispersion threshold algorithm with a minimum duration of 100 ms and a maximum dispersion of 40 pixels relative to the captured video scene frame. Results were analyzed in a 2×2 ANOVA using statistical software from SPSS Inc., where P50

Years of performing laparoscopic surgery Number of performing endoscopy procedure/year

Task performance in the forward-view condition was significantly faster (35 ± 15 sec) than performance in the retroflexed-view condition (51 ± 31 sec, P < .001). On average, the experts finished tasks in shorter time (19 ± 6 sec) than the novices (60 ± 20 sec, P < .001). Secondary analysis of the interaction between view condition and surgeon’s group revealed that experts and novices responded differently to each image viewing condition. The experts performed slightly worse in the retroflexed view (20 ± 6 sec) then the forward view (18 ± 5 sec) condition. In contrast, the novices were much more vulnerable to image distortion; their performance deteriorated significantly in the retroflexed condition (72 ± 21 sec) compared to the forward condition (47 ± 6 sec, P = .002).

3. Discussion Results support our research hypothesis – retroflexed image does impede task performance in NOTES. The reason, we believe, can be attributed to the eye-hand coordination difficulty related to the NOTES procedure. Unlike laparoscopy which requires one level of mental calibration by changing the viewing perspective from the eyes to the scope, NOTES requires additional mental work because the viewing perspective of the endoscope is constantly changing during the procedure [8]. The endoscope must constantly be maneuvered to maintain the horizon and keep track of spatial orientation. The changing perspective of the endoscope also internally changes the configuration of surgical instruments. When performing NOTES procedures with the endoscope retroflexed, another level of mental calibration must be included in the mental adjustment of a surgeon, which can make a surgeon easily lose orientation and dexterity. Loss of orientation and dexterity brings up significant safety concerns [3, 8]. We argue that when possible, the surgical approach needs to be chosen carefully to avoid using the retroflexed view during any NOTES procedure. Currently both forward and retroflexed view approaches are commonly available for a given NOTES procedure. For example, removing a patient’s gall bladder (cholecystectomy) has been achieved through both transgastric (retroflexed) and transvaginal/transcolon (forward view) approach. Now that we have demonstrated that retroflexion has negative impact on surgical task performance, we argue that surgeons should consider a transvaginal/transcolon approach for cholecystectomy.

B. Zheng et al. / Maintaining Forward View of the Surgical Site for Best Endoscopic Practice

747

It is interesting to observe that experienced surgeons were able to perform tasks in the retroflexed condition with minimal delay in comparison to the novice group. This is mainly due to the fact that experts are already experienced having performed large volume of endoscopic procedures on a daily basis. Extensive endoscopic experience allows experts to develop sophisticated cognitive strategies to deal with misalignment between perception and movement as presented by NOTES procedures [9]. Evidence presented in this study indicates that extensive training is required for a novice surgeon to overcome the difficult vision-motion coordination before they can perform NOTES effectively and safely. There are a number of limitations related to this study. The first limitation was that successful performance in a true endoscopic surgery requires skills much more complicated than those needed for the aiming and pointing task used in this study. The second limitation was that only a single surgeon was required to perform the pointing task, unlike the more commonly practiced surgical scenario that requires at least two surgeons work in a team for a NOTES procedure. Recently, we have incorporated bimanual coordination tasks into a new NOTES simulation model which was constructed on a double channel endoscopic platform. Two surgeons are allowed to work side-by-side, one to control the scope, the other to manipulate instruments on the surgical site. Replication of the current study with this new model will help to improve the generalization of our findings to a clinical setting. The third limitation was in the measurement used in the study. We used time to completion to describe the observable impact of visual-motion misalignment on the task performance. In any goal-direct movement such as the task we incorporated in this study, before the observable action, there is a period of cognitive process where environmental information is processed and an appropriate movement is planned[10]. This cognitive process is more sensitive to visual-motion alignment condition, rather than execution of the chosen movement plan. A superior measurement for the cognitive process would be the reaction time, defined as the time from the moment where visual information is presented to an operator, to the moment a movement is performed [11]. Future studies on the human factors of NOTES procedures will integrate the reaction time to measures, to give a comprehensive description of the impact of visual-motion misalignment on surgeons’ performance. In conclusion, the retroflexed view condition in NOTES procedure built on an endoscopic platform has a negative impact on surgeon’s performance. Careful planning is required for selecting an appropriate approach to avoid retroflexion and subsequent image distortion. To ensure safe performance of NOTES procedure, extensive endoscopic training is recommended for general surgeon before they can perform NOTES effectively and safely.

4. Acknowledgments This project has been funded by NOSCAR (Natural Orifice Surgery Consortium for Assessment and Research) research grant in 2007. The authors wish to thank the Boston Scientific Corporation for providing experimental devices of this study.

748

B. Zheng et al. / Maintaining Forward View of the Surgical Site for Best Endoscopic Practice

References [1] [2]

Kavic MS. "Natural orifice translumenal endoscopic surgery: "NOTES"",JSLS, 10(2),133-4, 2006. Bardaro SJ, Swanström L. "Development of advanced endoscopes for Natural Orifice Transluminal Endoscopic Surgery (NOTES)", Minim Invasive Ther Allied Technol, 15(6),378-83, 2006. [3] Volckmann ET, Hungness ES, Soper NJ, Swanstrom LL. "Surgeon Perceptions of Natural Orifice Translumenal Endoscopic Surgery (NOTES)", J Gastrointest Surg, 2009. [4] Swanstrom L, Swain P, Denk P. "Development and validation of a new generation of flexible endoscope for NOTES", Surg Innov, 16(2),104-10, 2009. [5] Sclabas GM, Swain P, Swanstrom LL. "Endoluminal methods for gastrotomy closure in natural orifice transenteric surgery (NOTES)", Surg Innov, 13(1),23-30, 2006. [6] Swanstrom LL, Volckmann E, Hungness E, Soper NJ. "Patient attitudes and expectations regarding natural orifice translumenal endoscopic surgery", Surg Endosc, 23(7),1519-25, 2009. [7] Kim W, Tendick F, Stark L. "Visual enhancements in pick-and-place tasks: Human operators controlling a simulated cylindrical manipulator", IEEE Robot Autom Mag, 3(5),418 - 425, 1987. [8] Swanstrom L, Zheng B. "Spatial Orientation and Off-Axis Challenges for NOTES", Gastrointest Endosc Clin N Am, 18(2),315-24, 2008. [9] Thompson CC, Ryou M, Soper NJ, Hungess ES, Rothstein RI, Swanstrom LL. "Evaluation of a manually driven, multitasking platform for complex endoluminal and natural orifice transluminal endoscopic surgery applications (with video)", Gastrointest Endosc, 2009. [10] MacKenzie CL, Iberall T. The Grasping Hand. Amsterdam; New York: North-Holland; 1994. [11] Martenuik RG, MacKenzie CL. "Methods in the study of motor programming: is it just a matter of simple vs. choice reaction time? a comment on klapp et al. (1979)", J Mot Behav, 13(4),313-9., 1981.


749

Phenomenological Model of Laser-Tissue Interaction with Application to Benign Prostatic Hyperplasia (BPH) Simulation Xiangmin ZHOUa,1, Nan Zhang a, Yunhe SHENa, Dan BURKEa, Vamsi KONCHADAa, Robert SWEETa a Center for Research in Education and Simulation Technologies (CREST), University of Minnesota, Minneapolis, MN

Abstract. Laser-tissue interaction is a multi-physics phenomenon not yet mathematically describable and computationally predictable. It is a challenge to model the laser-tissue interaction for real time laser Benign Prostatic Hyperplasia (BPH) simulation which requires the laser-tissue interaction model to be computationally efficient and accurate. Under the consideration and enforcement of the thermodynamic first law and treating the laser-tissue interaction as a graybox, utilizing the sensitivity analysis of some key parameters that will affect the laser intensity on the tissue surface with respect to the tissue vaporization rate, a phenomenological model of laser-tissue interaction is developed. The developed laser-tissue interaction model has been implemented for a laser BPH simulator and achieves real time performance (more than 30 frames per second). The model agrees well with the available experimental data. Keywords. Laser tissue interaction, Phenomenological Model

Introduction Benign prostatic hyperplasia (BPH) or "enlarged prostate" is a non-cancerous increase in the size and number of cells that make up the prostate. As the prostate enlarges, it impinges the flow of urine through the urethra. BPH is a common problem suffered by the majority of aged men. It causes a number of urinary symptoms, such as frequent urinating, urgent urinating, nocturia, and hesitancy. Traditionally, Transurethral Resection of the Prostate (TURP) is the long-standing BPH treatment of choice, where surgeons use an electrical loop to cut tissues piece by piece and seal blood vessels. Recently, laser Photoselective Vaporization of the Prostate (PVP) has emerged as a safe, less invasive and effective alternative to the "gold standard'' of TURP procedure. In laser PVP, surgeons perform the high-energy laser therapy, a form of heat therapy which vaporizes the overgrown prostate tissue, to provide swift symptom relief. Compared to TURP, laser therapy generally causes less bleeding and shorter hospital stay. It also does not cause impotence or prolonged incontinence.

1

Corresponding Author. E-mail: [email protected]

750

X. Zhou et al. / Phenomenological Model of Laser-Tissue Interaction

During the surgical procedure, a specially designed fiber optic delivery system is used, which is non-contact and side-firing. The fiber optic device is inserted into the urethra of the patient through a standard cystoscope, which is a tube-like instrument used to view the interior of the bladder. The laser light pulses Figure 1: Illustration of laser PVP procedure. are then directed toward the enlarged prostate tissue. The surgeon slowly drags the tip of the laser fiber along the prostatic urethra from the bladder neck to the level of the verumontanum to create a furrow of evaporated tissue. This process is shown in Figure 1. The surgeon repeats this process to create multiple furrows along the prostatic urethra, until a patent bladder outlet is resulted in. The average operative time is typically less than sixty minutes. Once the procedure has been completed, most patients experience immediate symptom relief and a dramatic improvement in symptoms such as urinary flow rate and bladder emptying. Inherently, every surgery is invasive, which may cause unwanted damage to a patient. For example, in laser PVP, the sphincter (the valve that turns the urine flow on or off) may be mistakenly vaporized off. The unique interface style of laser PVP requires the surgeon to acquire different skills than conventional open or laparoscopic surgery. Therefore, to overcome the learning curve in this therapy, new surgical teaching methods have to be developed. The development of simulators can facilitate the transfer of surgical skills to novice surgeons. For example, simulators allow the naive surgeon to develop skills and pass the learning curve without the medico–legal implications of surgical training, limitations in trainee working hours, and ethical considerations of learning basic skills on humans. Furthermore, simulators allow a trainee to gain experiences without increasing the risk to patients’ safety, e.g. making errors which are not allowed in real surgery. In building the laser BPH simulator, an appropriate laser-tissue interaction model is crucial. It is required that (a) the laser-tissue interaction can accurately predict the vaporization volume with respect to the system setting such as power and operating parameters such as working distance and treatment speed; (b) the computational complexity of the laser-tissue interaction model need to be suitable for the real-time simulation. The second requirement eliminates the feasibility of modeling the lasertissue interaction via the physical based approach. And the only feasible approach to model the laser-tissue interaction is the phenomenological approach. However, based on the limited experimental data available in the literature, it is insufficient to build a phenomenological model.


751

1. Laser-tissue interaction modeling In order to build the phenomenological model of laser-tissue interaction, we treat the laser-tissue interaction as gray-box. And we choose the inputs as the operating power, sweep speed, and working distance for the laser beam. The output of the model is chosen as the tissue vaporization rate. Form the physical phenomenon of laser-tissue interaction, we have the following assumption: a) at any given time, the tissue vaporization rate is limited by the operating power; b) there is a threshold limit for the linear dependence between the laser intensity and the tissue vaporization rate (volume per time); c) beyond the threshold limit, the vaporization rate is directly correlated to the power, d) below the threshold limit, the vaporization rate is linearly with the insufficient data available from the literature, we are able to construct a laser-tissue interaction model that is suitable for the real time surgical simulation. Given the operating environment of the Laser BPH therapy, the laser is interacting with the soft tissue in a fluid environment. The thermal interaction of the laser-tissue interaction involves coagulation and vaporization. Coagulation of the soft tissue occurs when the temperature of the underlying tissue is reaching 60°C and the thermal damage is induced. For the 532 nm wave length KTP laser that we are modeling, the depth of coagulation zone is consistently 0.8 mm regardless of the power setting of the laser beam (within the maximum power of 120W) and the working distance (the distance between the fiber surface and the tissue surface) [1]. Vaporization of the soft tissue occurs when the temperature of the underlying tissue is reaching 100°C and the water contained in the tissue is vaporized. Since the BPH therapy is operating in a fluid environment, and the coagulation zone of the soft tissue is always 0.8 mm depth, this implies that the peak temperature during the laser-tissue interaction is a minimum of 100°C. As a consequence, there exists a vaporization threshold power, Pv, of the laser beam for the occurrence of the vaporization phenomenon. Below the threshold power, the soft tissue cannot reach the vaporization temperature of 100°C, and no vaporization of the tissue should occur. When the power of the laser beam is higher than the vaporization power, the surface temperature of the tissue reaches 100°C and vaporization occurs. If the laser intensity on the tissue surface is increased beyond the vaporization threshold power, the excessive energy will contribute to the vaporization effect and the tissue surface temperature will keep increasing until the columns or slugs boiling effect occurs. This phenomenon causes the vaporization of the tissue reach a saturation state and the excessive energy is lost to the surrounding fluid due to the increased rate of heat transfer for the nucleate boiling. As the consequence, the relation between the tissue vaporization rate and the working distance is not linear. Utilizing the thermodynamic first law, the heat balance locally describing the laser ablation of the soft-tissue processes can be written as the following for a given domain of an open set and with the boundary B. @ d ` \ +g g g g d r r @? Where H is the enthalpy of which the phase change due to ablation is accounted for, k is the thermal conductivity, T is the temperature, ρb is the blood density, cb is the blood heat capacity, wb is the blood perfusion rate, Tb is the blood temperature, Qm is the metabolic heat generation, and Ql is the volumetric laser heat source. The volumetric laser heat source is obtained by

752


Q_ Where α is the absorption coefficient, I is the laser intensity, and ω’ is the solid angle. The laser intensity is described by the solution of the following differentialintegral equation of the transport equation. r £!¤ ` r r ¡¢ where k is the scattering coefficient, s is the laser direction, and p is the probability density function for scattering of which the Henyey-Greenstein phase function can be adopted as an approximation. Along with the trivial initial condition of the body temperature and the Neumann boundary conditions to account for the non-reflective solid boundary, the heat lost due to conduction/convection/radiation of the prostate surface, the above equation completely describes the local heat balance. However, to solve the governing equation to determine the soft tissue vaporization is not feasible, especially with respect to the application of surgical simulation. Firstly, some of the parameters and processes associated with the laser tissue interaction such as the soft tissue absorption coefficient, the scattering coefficient, the exact laser beam profile, and the boiling nucleation of soft tissue are not yet clear. Secondly, an appropriate numerical solution of the governing equation to determine the energy balance and transfer locally is impractical for the real time application requirement of the surgical simulation. Thus a phenomenological model of the laser tissue interaction to account for the global energy balance and transfer is more practical and preferable for the application of surgical simulation. In a global sense, with respect to the thermodynamic first law, the energy balance of the laser tissue interaction processes can be described as the following. r r r r ¥ r ¦ Where Qp is the due to blood perfusion, Qc is due to thermal damage and tissue denaturation, Qh is due to heating up the ablated tissue to the boiling temperature, Qa is due to phase change of the ablation, and Qb is the boundary condition. However, consider the fact that the power of the laser beam is adjustable, with respect to the sensitivity of the laser beam power, the global energy balance yields, ¥ K§ ¨ Q Q¥ § ª Q© Q© Q¥ « K§ ¬ f Q© Where P is the power, and Ic is the coagulation intensity, and Is is the saturation intensity. Ic corresponding to the laser intensity that is not enough to heat the soft tissue to the boiling temperature. There are two conditions that I ≤ Ic could happen. The first condition is that the laser beam power is less than 20 Watt, the second condition is that the fiber tip is too far away from the tissue surface and with the fact that the 120 Watt fiber has a 15° of divergent angle. The existent of the saturation intensity is due to physical constraint that there is a speed limit to heat the tissue to the boiling temperature due to conduction. When the laser intensity is greater than the saturation


753

intensity, the excesses energy is transferred to the ambient fluid environment due to nucleate boiling. Based on the above qualitative analysis and using the treatment speed TS (speed of the laser beam sweeping across the tissue surface, in mm/s), laser beam power P (in Watt), and working distance WD (in mm) as parameters, a phenomenological model of laser tissue interaction for the vaporization speed function is constructed as, where VS is the vaporization speed in mm3/s, and f(TS), h(P), and g(WD) are effects of the vaporization speed as functions of TS, P, and WD, respectively. The three functions are determined from experimental data. Utilizing the data provided from [1], we have,

where Pv=20W is the vaporization threshold power. Although the parameters of our model are determined using the experimental data, the available information is sufficient only for 80W laser power setting. We extrapolate the model to handle different laser power settings, e.g.; from 20W to 120W continuously. To validate the proposed model, the predicted results are compared with the experimental data from [1]. Comparisons are shown in Figures 2 and 3, which demonstrate that the proposed model can predict the behavior of the laser tissue interaction accurately within experimental errors.

Figure 2: Comparison of the proposed model and the experimental results.

754


Figure 3: Comparison of the proposed model and the experimental results.

2. Simulation Results Our simulation system is implemented on a Windows PC platform, which contains Intel Core2 Duo E6600 CPU, 4GB memory, and NVIDIA 8800GTX graphics board. We use an open source graphics package, called OGRE, as the rendering engine. OGRE is used for rendering special effects. Although our system is not designed with multi-threading support, we take advantage of the CPU/GPU parallelism. When all the rendering commands are submitted to the GPU, the main program does not wait for the rendering operations to be finished. Instead, it begins to process user input and perform geometry updates, including CSG difference evaluation, isosurface extraction, topology cleansing to remove tiny pieces, and collision handling. We test our system using several prostate models with different sizes. The largest one has a bounding box size of 61 x 52 x 60 mm and about 100 cm3 volume. In contrast, the laser beam has a diameter of only 0.75 mm. The largest model in our system contains about 344k vertices and 1.95 million tetrahedral elements. For all experiments, the grid cell size is set as approximately the maximum effective range of the laser beam, which is about 8 mm. Unless specified explicitly, we always use the laser power setting of 80 Watt and laser beam slope angle of 8 degrees. Figure 4 shows the simulation results, and Figure 5 shows the comparison with the operation video.

Figure 4: Laser PVP simulation.


755

3. User Validation We have asked two internal surgeons who are experienced in laser BPH therapy to validate our prototype implementation. In general, they were satisfied with the realism of the virtual environment created in our training system. From surgeons' point of view, realism in the behavior of the tissue vaporization model is more important than that of the graphical appearance of the prostate model. They reported that the Figure 5: Comparison of the simulation with the operation video. proposed phenomenological laser-tissue interaction model yielded a result that was very close to what the urologists felt in the operating room. Furthermore, we have sent our system to an annual urological conference for evaluation. About 40 urologists have tested the system. Although we were unable to schedule a comparison on both algorithms, in general, the experts were satisfied with the melting effect generated from our algorithm and the melting speed. Encouraged by this success, we are in the stage of planning some nation-wide, more rigorous user study activities. Details of the new updates and the results of validation study will appear in our future report. 4. Conclusion A phenomenological model of laser tissue interaction is proposed based on the qualitative study for the sensitivity of the global energy balance with respect to the laser intensity (power) is proposed. The proposed model not only can capture all the available experimental data points but also suitable for the real time application of the surgical simulation. And the proposed approach is also suitable to characterize the laser tissue interaction for different laser fiber designs if appropriate experimental data is available.

Acknowledgement Funding from American Medical System (AMS) is acknowledged. Computer support from Minnesota Supercomputer Institute (MSI) is gratefully acknowledged.

References [1] H. W. Kang, D. Jebens, R. S. Malek, G. Mitchell, and E. Koullick. Laser vaporization of bovine prostate: A quantitative comparison of potassium-titanyl-phosphate and lithium triborate lasers. The Journal of Urology, 180 (2008), 3675-2680



757

Subject Index 3 dimensional models 96 3D gaze calibration 616 3D interaction 372 3D lung dynamics 567 3D muscle 560 3D segmentation 552 3D visual guidance 400 3D visualization 372 3D-cranio-mandibular model 261 accuracy 524 activities of daily living (ADL) 730 adaptive signal processing 60 affective computing 132 aging 510 AISLE 476 anatomy 18, 96, 264, 280, 397 anatomy navigation system 354 anxiety disorder 696 Arbitrary Lagrangian-Eulerian method 710 arrhythmia 57 arthritis 18 arthroscopy 236 artificial neural networks 25 assessment 8, 86, 304, 493, 496 attention 192 augmented reality 336, 408 autism 132 autistic spectrum disorder 132 automatic volumetric segmentation 476 avatar collaboration 372 barriers to care 503 biofeedback 696 biofilms 394 bio-imaging 138 biomanipulation 231 biomechanical model 560 biopsy 242, 623 biosensors 86, 185, 496 bipolar disorders 496 bougie 65 BPH 574

brain anatomy 105 brain dynamics 329 brain neuro-machine interfaces 163 BRDF 105 bronchoscopy simulation 535 CAD/CAM 239 cadaver 397 cancer 691 cardiac 57 cardiac surgery 150, 716 catheter 594 cell microinjection 231 clinical breast exam 408 clinical examination 271 C-MAC 369 cognition 428 collaborative stereo visualization 264 collision detection 555, 560 computational fluid dynamics 567 computed tomography 18, 680 computer aided psychotherapy 44 computer graphics 389 computer vision 581 computer-based assessment 517 Computerized Cognitive Behavior Therapy (CCBT) 86 confirmation 611 connected components 359 containment detection 560 content validity 274 contextualized learning 144 corneas 653 Cosserat rod 466 CPU-GPU balancing scheme 354 curriculum 150 cutting simulation 311 CvhSlicer 354 CyberMed 386 cyberpsychology 44 630 da VinciTM Surgical System DARPA 730 data sets 670 data visualization 685

758

datasets 677 deformable surface models 560 deformation model 645 dental 653 depression 86, 496, 696 direct laryngoscopy 71 disability 510 dissection 397 dysesthesia 680 ecological validity 433 60 ECRTM education 57, 119, 264 educational content 242 EEG 329, 606 elastic object 638, 645 e-learning 202 electrical discharges 297 electrocautery 166, 311 electromagnetic tracking 479 electromagnetics 329 electromyography 630 60 electronic competency recordTM electroporation 297 emergency response 650 emotion sensing 132 emotions 44 Endo Stitch 461 endoscope 594 endoscopic surgery 743 Endoscopic Third Ventriculostomy 1 endotracheal intubation 400, 611 endovascular simulation 317 error analysis 304 ETI 611 eTraining 650 evaluation 743 evaluation/methodology 535, 542 face validity 274 feedback 119 fiberscope 68, 713 fidelity 147, 670 finite element method 415, 663 finite elements 31 flow-volume curve 25 force sensing 408 fractal dimension 606 fuzzy control 39 game-based learning 147, 254

gas phase plasma 297 gastrointestinal endoscopy 199 guide wire 594 guidewire/catheter 317 habituation 696 hand and wrist 397 haptic collaborative virtual environment 638 haptic feedback 224 haptic interface 199 haptic rendering 112, 645 haptics 135, 213, 397, 542, 555, 588, 591, 670, 691 hardware 135, 677 head modeling 329 head-mounted eyegaze tracker 658 hernioplasty 202 hierarchical segmentation 599 hip 18 hip re-surfacing 283 human-centered computing 535, 542 human computer interaction 400, 549, 552 hybrid reality 552 hydration 653 hydrocephalus 1 image analysis 138 image guidance 716 image guided surgery 283, 479 impulse response deformation model 645 indirect laryngoscopy 77 inertial measurement unit 479 infrastructure 723 integration 93 intelligent tutoring systems 60 interactive learning 254 interactive remote visualization 635 interactive simulation framework 213 internet 86 interreality 185 interscalene block 36 intubation 65, 68, 71, 74, 366 intubation training 77, 80, 83, 549, 688 knowledge transfer 147 language design 209 laparoscopic simulators 588, 591 laparoscopic surgery simulator 658

759

laparoscopic surgery 11, 581 laparoscopic training 588, 591 laparoscopy 348 laryngoscopy 74, 400 laser 394, 713 laser tissue interaction 749 layered depth images 224 learning curve 524 learning technologies 535, 542 lesions 359 level set 599 levels of realism 147 liver 348 localization 329 lower extremities 290 lung allometry 476 lung radiotherapy 567 machine vision 11 major incident response 650 Mammacare® 408 mass casualty 650 mass-spring model 317 master slave 524 medical 677 medical artist 397 medical education 173 medical robotics 716 medical simulation 199, 277, 542, 581 medical student education 271 medical training 277, 650 medical training simulator 51 meditation 696 metamorphopsia 336 micromanipulator 524 military healthcare 503 minimally invasive surgery 454, 723 mixed reality 144, 552 mobile eyetracking 616 mobile technologies 86, 496 modeling and simulation 156 modification 135 motion tracking 280 motor learning 119 motor skills 192 motor-neuroprosthetics 163 MRI 552, 716 MRI compatible robot 623 multifunction robotic platform 740

multi-level computer performance systems 638 multiple sclerosis 359 multi-tasking 192 Musculoskeletal Modeling Software (MSMS) 730 myoelectric prostheses 156 natural orifice surgery 740 Naval Hospital Camp Pendleton (NHCP) 696 navigation system 713 Navy Medical Center San Diego (NMCSD) 696 needle insertion 135 needle insertion simulation 710 network 93 neurofeedback 606 Neuropsychological assessment 433 neurorobotics 163 neurosurgery 51, 166 NeuroVR 8, 493 non-contact position sensing 549 NOTES procedure 743 numbness 680 occlusal contacts 261 Office of Naval Research (ONR) 696 Off-Pump Coronary Artery Bypass Surgery 147 open source 493 open surgery 202 operating room 93 ophthalmology 560 optical tracking 403 orthopedic surgery 283, 324 out-of-hospital 80 pain management 606 palpation 408 pancreas 691 parallel FEM 415 Parkinson’s disease 8 particle method 389 patent 351 patient education 96 patient model 524 patient specific surgical simulation 379 patient training and rehabilitation 156

760

patient-specific 447 patient-specific instrument guides 283 patient-specific model 112, 415 pectus excavatum 473 pelvic floor muscle 218 pelvis 280 penetration volume 224 perception 588 perceptual motor learning 428 percutaneous minimally invasive therapy 710 perioperative medicine 737 peripheral nerve block 552 Personal Health Systems 86, 496 phantom limb pain (PLP) 730 phenomenological model 749 physically-based simulation 461, 466 physics simulation 213 physiological monitoring 696 piezoelectric driven injector 231 piezoresistive sensor 703 plasma-medicine 297 pneumatic balloon actuator 703 pneumoperitoneum 348 postoperative 425 Posttraumatic Stress Disorder (PTSD) 696 precision 524 preoperative evaluation 737 preoperative 425 presence 44 prevention 86 probabilistic tractography 486 projective augmented-reality display 549 prostate brachytherapy 623 prosthodontic 422 prototype 351 psychological treatments 44 psychophysiology 433 pulmonary function test 25 PVP 574 real time 594 real-time interaction 236 real-time simulation 31, 213 real-time spatial tracking 400 reconstruction 447

regional anesthesia 36, 119 rehabilitation 163, 290, 703 rehabilitation robotics 39 remote consultant 93 renal surgery training 415 rendering 105 respirometry 25 Revolutionizing Prosthetics 2009 730 robot assisted surgery 274 robotic devices 247 robotic surgery 703 robotic surgical simulation 379 role-playing 173 Second Life 440 segmentation 138, 359 semantics derivation 209 sensor fusion 479 sensors 535, 542 sensory augmentation 703 serious games 147, 254, 606 shockwaves 394 SimCoach 503 SimTools 611 simulation 119, 125, 135, 150, 166, 173, 202, 271, 324, 447, 517, 599, 611, 630, 650, 677, 723, 743 simulation and 3D reconstruction 348 simulation-based training 400 simulation development 531 simulation meshes 670 Simulation Support Systems 535, 542 simulator maintenance 531 simulator 57, 242, 531 skills training 591 skin area 680 skin burns 653 369 SkypeTM sliding 663 SOFA 691 soft tissue grasping 663 software 677 software framework 343 software system 560 spiking neurons 685 stapedotomy 524

761

stereo imaging 454 stereo-endoscopy 1 stereolithography 18, 552 stereoscopic vision 680 stress 86, 185, 496 stress management 185 stress-related disorders 44 stroke 39 SUI 218 surgery 691 surgery simulation 31, 224, 311, 574 surgery simulation development 209 surgery training 11 surgical navigation 479 surgical planner 473 surgical rehearsal 112 surgical robotics 454 surgical robots setup 379 surgical simulation 112, 144, 236, 535 surgical simulator 389, 415 survey 277 suture 461, 466 suturing 31 system design 304 tactile feedback 703 task analysis 277 task decomposition 277 taxonomy 677 teaching 397 teaching curriculum 36 technical skills 517 technology 510 Technology Enhanced Learning 324 technology transfer office 351 telehealth 425 teleimmersion 290 telemedicine 93, 369, 425, 737 telerehabilitation 290 temporal bone surgery 112 terahertz imaging 653 therapy 493, 496 tongue retractor 68 training 125, 304, 447, 630, 723 training simulator 403 trajectory error 247 trauma 650 treatment plannning 422

tumor 691 two-handed interface 372 ultrasound 119, 138, 242, 447 ultrasound guided regional anesthesia 304 ultrasound image simulation 403 upper limbs 247 user interface design 343 user interfaces 549 user models 428 user studies 372 vaginal wall simulation 218 vergence eye movements 616 video conferencing 688 video laryngoscope 65, 74, 77, 80, 83, 366, 369 video laryngoscopy 71, 688 virtual environment 433, 555, 594 virtual humans 503 Virtual Integration Environment (VIE) 730 virtual patient 144, 173, 408, 440, 650 virtual preop 737 Virtual Reality 8, 44, 51, 96, 156, 163, 185, 202, 280, 264, 304, 386, 389, 397, 422, 428, 486, 493, 496, 510, 552, 574, 581, 685, 716 virtual reality articulator 239 Virtual Reality Graded Exposure Therapy (VR-GET) 696 Virtual Reality Therapy 696 virtual simulation 147, 254 virtual training 274 virtual world 125, 173, 440, 650 vision-motion alignment 743 visual impairment 336 Visual Programming 386 visualization 343, 549 VMET 8 Voice over Internet Protocol (VoIP) 83, 369 volume rendering 112, 264, 372, 635 volumetric 691 VTC 369, 425 Walter Reed Military Amputee Research Program (MARP) 730 War on Terror 696

762

web-based web-based visualization web-enabling

96 635 264

wound wound closure X3D

394 461 670


763

Author Index Abhari, K. Aggarwal, R. Agin, K. Ahn, W. Aisen, M. Akbar, M. Al-Allaq, Y. Alasty, A. Albani, G. Alcañíz, M. Allen, B.F. Allen, P. Amunts, K. Ando, E. Ankeny, M.L. Anstey, J.B. ap Cenydd, L. Arabalibeik, H. Arai, T. Aratow, M. Ardanza, A. Argun, O.B. Arikatla, V.S. Arizmendi, B. Armiger, R.S. Asghari, M. Atkins, M.S. Avis, N. Awad, I.T. Ayres, F. Babaeva, N.Yu. Backstein, D. Baillargeon, E. Baitson, G. Bajcsy, R. Bajd, T. Banerjee, P. Baniasad, M.A. Baños, R. Barak-Bernhagen, M.A. Barner, K.E. Batrick, N. Beenhouwer, D.

1 440, 650 25 213 510 39 119 39 8 44, 348 11 552 486 422 691 18 105 25 713 670 163 280 31, 311 433 730 663 658 304 36, 119 166 297 192, 254 549 304 290 290 510 39 44, 86, 496 77, 351, 737 224, 691 650 394

Beier, F. Bello, F. Bennett, D. Berg, D.R. Bergeron, B. Bernhagen, M.A.

51 202, 317, 599 653 57 60 65, 68, 71, 74, 80, 83, 688 Bhargava, A. 567 Biersdorff, S. 329 Bisley, J.W. 703 Bito, T. 638 Blevins, N.H. 112 Bloj, M. 105 Blossfield Iannitelli, K. 271, 531 Boedeker, B.H. 65, 68, 71, 74, 77, 80, 83, 351, 366, 369, 425, 688, 737 Boedeker, K.A. 74, 77 Boronyak, S. 552 Botella, C. 44, 86, 496 Branstetter, B. 552 Bregman, R. 549 Brown, A. 549 Brown, B. 611 Brown, E.R. 653 Bucholz, R.D. 93 Buckwalter, J.G. 503 Bulpitt, A. 599 Burck, J.M. 730 Burg, K.J.K.L. 588, 591 Burg, T.C. 588, 591 Burke, D. 96, 280, 574, 749 Cabral, A. 166 Caldwell, D.G. 135 Callahan Jr., J.B. 366, 369 Cameron, B.M. 99 Camilo, A. 623 Campbell, A.R. 680 Carelli, L. 8 Carlson, A. 57 Carrasco, E. 163 Carson, C. 574 Caspers, S. 486

764

Cassera, M.A. 743 Chan, S. 112 Chandrasekhar, R. 274 Charnoz, A. 403 Chaya, A. 549 Chen, B. 138 Chen, C.-P. 691 Chen, D. 311 Chen, E.W. 36, 119 Chen, H.Y. 231 Cheung, J.J.H. 36, 119 Chevreau, G. 242 Chien, J.-h. 428, 630 Chiesa, C. 366, 369 Chin, W.J. 740 Chodos, D. 125 Chowriappa, A. 274 Chui, Y.P. 354 Chung, S.Y. 132 Cipresso, P. 185, 493 Cline, A. 60 Cochran, A. 653 Cohen, D. 650 Cole, G.A. 623 Coles, T.R. 135 Constantinou, C.E. 138, 218 Courteille, O. 144 Courtney, C.G. 433 Cowan, B. 147 Cristancho, S. 147, 150, 254, 517 Culjat, M.O. 653, 703 Darzi, A. 440, 650 Davey, C. 329 Davies, M.G. 716 Davoodi, R. 156, 730 Dawson, M. 433 de Hoyos, A.L. 535 De Mauro, A. 163 de Ribaupierre, S. 1, 180 De, S. 31, 209, 213, 311, 555 Dech, F. 264 Delorme, S. 166 Deng, Z. 247, 716 Dev, P. 173 Devadhas, D. 691 Diederich, S. 51 Diesmann, M. 685 Dietz, A. 524 Difede, J. 503

Dillmann, R. Din, N. Dindar, S. Dittrich, E. Divo, E. Domont, Z.B. Donvito, G. Dubrowski, A.

635 599 723 524 567 271 493 36, 119, 147, 150, 192, 254, 517 Dumpert, J. 454 Durfee, W.K. 57 Dutson, E.P. 11, 703 Eagleson, R. 1, 180 Einhäuser, W. 616 Ellis, R.E. 18, 283, 680 Enders, L. 549 Enochsson, L. 144 Erdman, A.G. 280 Erhart, K. 567 Evestedt, D. 670 Faloutsos, P. 11 Fan, R. 703 Farahmand, F. 39, 663 Farritor, S. 454, 740 Felländer-Tsai, L. 144 Ferrari, M. 379 Ferrari, V. 379 Fischer, G.S. 623 Florez, J. 163 Foo, J.L. 343 Forbell, E. 503 Forest, C. 403 Fors, U. 144 Frantz, F.W. 473 Frizera Neto, A. 163 Fukushima, S. 239, 261, 422 Gaggioli, A. 8, 185, 493, 496 Gagliati, A. 493 García-Palacios, A. 44, 86 Gasco, S. 493 Gil, A. 163 Goretsky, M. 473 Gould, D.A. 135, 317, 599 Grassi, A. 185, 493 Grierson, L. 192, 517 Grundfest, W.S. 394, 653, 703 Gu, Y. 199 Gupta, V. 394 Guru, K. 274

765

Haake, D.A. Haase, R. Hald, N. Halic, T. Hammond, D. Hasegawa, S. Hata, N. Hattori, A. Head, M. Hedman, L. Hein, C. Hein, S. Heinrichs, W.L. Hemstreet, G.P. Hemstreet, J.L. Heng, P.A. Hirabayashi, R. Hirai, S. Ho, Y. Hofer, M. Hoge, C. Holmes III, D.R. Hostettler, A. How, T. Hu, R. Huang, H.B. Huang, M.H. Hubschman, J.P. Hughes, C.J. Hurmusiadis, V. Ikawa, T. Ilegbusi, O. Inuiya, T. Jafari, S. Jain, S. Janssoone, T. Jerald, J. Jiang, D. Jin, Y.-Y. John, N.W. Johnsen, K. Johnston, S. Jose, S. Jowlett, N. Juhas, M. Juhnke, B. Kadivar, Z. Kairys, J.C. Kaltofen, T.

394 524 202 209, 213 329 218 623 239, 261, 422, 713 740 144 447, 611 552 173 425 425 354 239 261 403 524 329 99 403 317 224, 691 231 403 653 594 236 239, 261, 422, 713 567 415 25 574 242 372 166 691 105, 135, 594, 670 408 696 343 192 552 343 247 691 560

Kapralos, B. 147, 254 Karmakar, M. 354 Kasama, S. 239, 261, 422, 713 Kaspar, M. 264 Kasper, F. 11 Kassab, A. 567 Katz, N. 8 Kaye, A.R. 271 Kazanzides, P. 476, 479 Kealey, C.P. 653 Kelly, R. 473 Kenny, P. 503 Kerr, K. 650 Kesavadas, T. 274 Khoramnia, R. 336 Kiely, J.B. 730 Kim, J. 503 Kim, Y. 581 King, D. 599 King, S. 125, 180 Kizony, R. 8 Kjellin, A. 144 Knisley, S.B. 473 Knott, T. 277, 677 Koffman, R.L. 696 Kohlbecher, S. 616 Konchada, V. 96, 280, 574, 749 Konety, B. 96 Korenblum, D. 138 Koritnik, T. 290 Kubota, Y. 415 Kuhlen, T. 277, 486, 670, 677, 685 Kunz, M. 283 Kupelian, P.A. 567 Kuper, G.M. 80, 83 Kurenov, S. 461, 466 Kurillo, G. 290 Kushner, M.J. 297 Kwak, H. 581 Kwok, W.H. 354 Lacy, T. 74 Lago, M.A. 348 Lange, B. 503, 510 Lanzl, I. 336 Larcher, A. 324 Laycock, K.A. 93 Lee, D. 581 Lee, D.H. 112 Lee, D.Y. 199

766

Li, K. 329 Li, P. 112 Liebschner, M. 247 Lind, D.S. 408 Lindgren, G. 144 Linnaus, A. 68 Littler, P. 317 Loeb, G.E. 156, 730 Long, X. 567 López-Mir, F. 348 Lövquist, E. 304 Lu, Z. 213, 311, 555 Luboz, V. 317, 599 Luengo, V. 324 Lüth, T. 524 Luzon, M.V. 594 Machado, L.S. 386 MacNeil, W.R. 93 Maier, S. 635 Makiyama, K. 415 Malony, A.D. 329 Männer, R. 51 Marescaux, J. 403 Markin, N. 688 Martinez-Escobar, M. 343 Martínez-Martínez, F. 348 Martin-Gonzalez, A. 336 Marx, S. 616 Matsuo, K. 351 Mauro, A. 8 McCartney, C.J.L. 36, 119 McDurmont, L.L. 93 McKenzie, F.D. 473 McLay, R.N. 696 Meeks, S.L. 567 Melnyk, M. 192 Meneghetti, A. 743 Meng, Q. 354 Merdes, M. 479 Merians, A. 510 Meruvia-Pastor, O. 359 Miljkovic, N. 80, 83, 366, 369 Miller, D.J. 65, 68, 74, 80, 83, 351, 366, 369 Mills, J.K. 231 Min, Y. 567 Mirbagheri, A. 663 Mitchell, R. 567 Mlyniec, P. 372

Moglia, A. 379 Moncayo, C. 150 Monclou, A. 150, 254, 517 Monserrat, C. 348 Moragrega, I. 86 Morais, A.M. 386 Moran, C. 730 Morgan, J.S. 688 Morganti, F. 8 Morikawa, S. 710 Mosca, F. 379 Moussa, F. 147, 150 Mozer, P. 242 Mukai, N. 389 Mukherjee, M. 630 Murray, W.B. 65, 68, 71, 77, 80, 83 Nagasaka, M. 415 Nakagawa, M. 389 Nakamura, A. 740 Nataneli, G. 11 Navab, N. 336 Navarro, A. 394 Needham, C. 397 Neelakkantan, H. 567 Nelson, C. 740 Nelson, D.A. 400, 549, 552 Neumuth, T. 524 Nguyen, M.K. 606 Nicholas IV, T.A. 68, 71 Nicolau, S.A. 403 Niki, K. 389 Nikravan, N. 119 Niles, T. 408 Noon, C. 343 Nuss, D. 473 O’Malley, M. 247 O’Sullivan, O. 304 Odetoyinbo, T. 317 Ogata, M. 415 Ogawa, T. 239, 261, 422, 713 Oh’Ainle, D. 304 Oleynikov, D. 428, 454, 630, 740 Omata, S. 218 Orebaugh, S. 552 Owen, H. 611 Oyarzun, D. 163 Ozawa, T. 713 Pagano, C.C. 588, 591 Paik, J. 428

767

Pallavicini, F. Panton, N.O.N. Papp, N. Park, E.S. Park, S. Park, S.-H. Parsad, N.M. Parsons, T. Pasquina, P.F. Patel, V. Patton, J. Peddicord, J. Peloquin, C. Peniche, A.R. Pérez, L.C. Peters, J. Peters, T. Petrinec, K. Phillips, N. Pignatti, R. Pinto, R. Polys, N.F. Pons, J.L. Pop, S.R. Porte, M. Potjans, T.C. Prabhu, V.V. Priano, L. Psota, E. Pugh, C.M. Punak, S. Pyne, J. Qayumi, K.A. Qin, J. Quero, S. Ragusa, G. Rank, D. Raspelli, S. Rasquinha, B. Rechowicz, K.J. Reger, G. Reihsen, T. Ren, H. Requejo, P. Rettmann, M.E. Rhode, K. Rick, T. Ritter, F.E. Riva, G.

185, 493 743 567 425 581 428, 630 264 433, 503 730 440, 650 510 343 343 535 454 723 1 447 105 8 552 670 163 594 254 685 591 8 454 271, 531, 535, 542 461, 466 696 743 354 44 510 479 8, 185, 493 18 473 503 57 476, 479 510 99 236 486, 685 428 8, 185, 493, 496

Rizzo, A.A. 503, 510 Robb, R.A. 99 Robinson, E. 549 Roehrborn, C. 574 Rojas, D. 517 Rolland, J.P. 567 Rothbaum, B.O. 503 Rotty, V. 96 Rudan, J.F. 18, 283 Ruddy, B.H. 567 Rueda, C. 150, 517 Runge, A. 524 Runge, H.J. 351 Rupérez, M.J. 348 Sabri, H. 147 Sagae, K. 503 Salisbury, J.K. 112 Salman, A. 329 Salud, J.C. 531, 535 Salud, L.H. 271, 531, 535, 542 Samosky, J.T. 400, 549, 552 Sanders, J.M. 691 Sankaranarayanan, G. 31, 213, 555 Santana Sosa, G. 560 Santhanam, A.P. 567 Sarker, S.K. 202 Sarosi, G. 723 Satake, K. 710 Sathyaseelan, G. 274 Savitsky, E. 447 Schaeffter, T. 236 Schmieder, K. 51 Schneider, E. 616 Schrack, R. 630 Schulte, N. 366 Schultheis, U. 372 Seagull, F.J. 372 Seixas-Mikelus, S. 274 Sensen, C.W. 359 Seow, C.M. 740 Sevdalis, N. 650 Shen, Y. 96, 280, 574, 749 Sherman, K. 236 Shigeta, Y. 239, 261, 422, 713 Shin, S. 581 Shirai, Y. 710 Shorten, G. 304 Silverstein, J.C. 264 Sinclair, C. 36

768

Singapogu, R.B. 588, 591 Singh, R.S. 653 Siu, K.-C. 428, 630 Smelko, A. 552 Smith, E.J. 18 Soames, R. 397 Sofia, G. 135 Soh, J. 359 Soler, F. 594 Soler, L. 403 Song, J.E. 329 Song, Y. 599 Sourina, O. 606 Spira, J.L. 696 Sprick, C. 611 Srimathveeravalli, G. 274 Stallkamp, J. 479 Stegemann, A. 274 Steiner, K.V. 224, 691 Stoll, J. 616 Strabala, K. 454 Strauss, G. 524 Strauss, M. 524 Stroulia, E. 125, 180 Su, H. 231, 623 Suh, I.H. 428, 630 Sukits, A.L. 549 Sung, C. 247 Suwelack, S. 635 Suzuki, N. 239, 261, 422, 713 Swanström, L.L. 743 Sweet, R. 96, 574, 749 Sweet, R.M. 57, 280 Sweitzer, B.J. 737 Syed, M.A. 716 Tagawa, K. 638, 645 Takanashi, S. 389 Tanaka, H.T. 638, 645, 710 Taylor, D. 440, 650 Taylor, Z.D. 394, 653 Tempany, C.M. 623 Tewari, P. 653 Thakur, M.L. 691 Thompson, M. 552 Thompson, Z. 247 Tien, G. 658 Tirehdast, M. 663 Toledo, F. 372 Tonetti, J. 324

Torres, J.C. Torricelli, D. Troccaz, J. Tsao, J.W. Tsekos, N.V. Tucker, D. Turini, G. Turovets, S. Ullrich, S. Unterhinninghofen, R. Vadcard, L. Vanberlo, A.M. Vemuri, A. Venkata, S.A. Vigna, C. Villard, P.-F. Volkov, V. von Kapri, A. Walker, M.I. Walker, R.B. Walter, A. Wampole, M. Wang, Q. Weaver, R.A. Webb, B. Webb-Murphy, J. Weeks, S.R. Wei, D. Weinhaus, A. Weiss, P.L. Westwood, J.D. White, S. Wickstrom, E. Wiederhold, B.K. Wiederhold, M.D. Wilding, G. Wilkinson, C. Williams, J. Winer, E. Winstein, C.J. Wong, C. Wood, D.P. Wood, G.C.A. Wottawa, C. Wu, H.S. Xiao, M. Yamaguchi, S. Yamamoto, T. Yamazaki, Y.

594 163 242 730 716 329 379 329 277, 670, 677 635 242 680 403 213 493 202 329 486, 685 688 688 105 691 606 549, 552 369, 737 696 730 218 280 8 v 653 691 185, 496, 696 696 274 397 503 343 510 574 696 283 703 403 359 710 713 713

769

Yeniaras, E. Yeo, Y.I. Yoganandan, A. Yoon, H.J. Yoshida, Y. Youngblood, P. Zeher, M.J. Zetterman, C.V.

716 723 372 132 218 173 730 737

Zhai, J. Zhang, N. Zhang, X. Zheng, B. Zhou, X. Zilles, K. Ziprin, P.

317 574, 749 740 658, 743 96, 574, 749 486 202


Medicine Meets Virtual Reality 18: NextMed - Volume 163 Studies in Health Technology and Informatics

Medical And Care 3 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Global Health Informatics Education (Studies in Health Technology and Informatics)

Advances in Information Technology and Communication in Health - Volume 143 Studies in Health Technology and Informatics

International Perspectives in Health Informatics - Volume 164 Studies in Health Technology and Informatics

Virtual Reality Technology

Medical and Care Compunetics 2 (Studies in Health Technology and Informatics, Vol. 114) (Studies in Health Technology and Informatics)

Research into Spinal Deformities 5 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Medicine Meets Engineering: Proceedings of the 2nd Conference on Applied Biomechanics Regensburg - Volume 133 Studies in Health Technology and Informatics

Plaque Imaging: Pixel to Molecular Level (Studies in Health Technology and Informatics, Vol. 113) (Studies in Health Technology and Informatics)

Connecting Medical Informatics and Bio-Informatics (Studies in Health Technology and Informatics)

E-Health: Current Situation And Examples Of Implemented & Beneficial E-Health Applications (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Virtual Reality in Geography

Virtual Reality

Biological Petri Nets - Volume 162 Studies in Health Technology and Informatics

Virtual Reality & Augmented Reality in Industry

Virtual Society? Get Real!: Technology, Cyberbole, Reality

Medicine Meets Virtual Reality 14: Accelerating Change in Healthcare: Next Medical Toolkit

Regional Health Economies and ICT Services (Studies in Health Technology and Informatics Shti)

Evaluation Methods in Biomedical Informatics (Health Informatics)

The Conservative Scoliosis Treatment:1st SOSORT Instructional Course Lectures Book (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

eHealth Beyond the Horizon - Get IT There:Proceedings of MIE2008 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Artificial Life and Virtual Reality

Artificial Life and Virtual Reality

MEDINFO 2007: Proceedings of the 12th World Congress on Health (Medical) Informatics (Studies in Health Technology and Informatics)

Genome Informatics 2007 (Genome Informatics Series, Volume 18)

Surgical Scene Generation for Virtual Reality-Based Training in Medicine

Surgical Scene Generation for Virtual Reality-Based Training in Medicine

Information Technology Solutions for Healthcare (Health Informatics)

The Lucent Library of Science and Technology - Virtual Reality

Medicine Meets Virtual Reality 18: NextMed - Volume 163 Studies in Health Technology and Informatics

Medical And Care 3 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Global Health Informatics Education (Studies in Health Technology and Informatics)

Advances in Information Technology and Communication in Health - Volume 143 Studies in Health Technology and Informatics

International Perspectives in Health Informatics - Volume 164 Studies in Health Technology and Informatics

Virtual Reality Technology

Medical and Care Compunetics 2 (Studies in Health Technology and Informatics, Vol. 114) (Studies in Health Technology and Informatics)

Research into Spinal Deformities 5 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Medicine Meets Engineering: Proceedings of the 2nd Conference on Applied Biomechanics Regensburg - Volume 133 Studies in Health Technology and Informatics

Plaque Imaging: Pixel to Molecular Level (Studies in Health Technology and Informatics, Vol. 113) (Studies in Health Technology and Informatics)

Connecting Medical Informatics and Bio-Informatics (Studies in Health Technology and Informatics)

E-Health: Current Situation And Examples Of Implemented & Beneficial E-Health Applications (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Virtual Reality in Geography

Virtual Reality

Biological Petri Nets - Volume 162 Studies in Health Technology and Informatics

Virtual Reality & Augmented Reality in Industry

Virtual Society? Get Real!: Technology, Cyberbole, Reality

Medicine Meets Virtual Reality 14: Accelerating Change in Healthcare: Next Medical Toolkit

Regional Health Economies and ICT Services (Studies in Health Technology and Informatics Shti)

Evaluation Methods in Biomedical Informatics (Health Informatics)

The Conservative Scoliosis Treatment:1st SOSORT Instructional Course Lectures Book (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

eHealth Beyond the Horizon - Get IT There:Proceedings of MIE2008 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Artificial Life and Virtual Reality

Artificial Life and Virtual Reality

MEDINFO 2007: Proceedings of the 12th World Congress on Health (Medical) Informatics (Studies in Health Technology and Informatics)

Genome Informatics 2007 (Genome Informatics Series, Volume 18)

Surgical Scene Generation for Virtual Reality-Based Training in Medicine

Surgical Scene Generation for Virtual Reality-Based Training in Medicine

Information Technology Solutions for Healthcare (Health Informatics)

The Lucent Library of Science and Technology - Virtual Reality

Recommend Documents