We are developing a visuo-haptic environment for simulating a variety of surgical procedures whose core component is bone drilling. The core environment is flexible, and has been used on otologic, craniofacial, and dental datasets. This page contains a high-level description of the system and screenshots/movies representing each of these applications. For more detailed
technical information about the underlying environment, see our publications
based on this environment:
- Morris D, Sewell C, Barbagli F, Blevins N, Girod S, Salisbury K. Visuohaptic Simulation of Bone Surgery for Training and Evaluation. IEEE Transactions on Computer Graphics and Applications, November 2006, pp. 48-57.
- Morris D, Sewell C, Blevins N, Barbagli F, Salisbury K. A Collaborative Virtual Environment for the Simulation of Temporal Bone Surgery. Proceedings of MICCAI, September 2004, Springer-Verlag Lecture Notes in Computer Science Volumes 3216 and 3217.
- Morris D, Girod S, Barbagli F, Salisbury K. An Interactive Simulation Environment for Craniofacial Surgical Procedures. Proceedings of MMVR (Medicine Meets Virtual Reality), January 2005.
Analysis and Metrics
- Sewell C, Morris D, Blevins N, Dutta S, Agrawal S, Barbagli F, Salisbury K. Providing Metrics and Performance Feedback in a Surgical Simulator. Computer Aided Surgery, Volume 13 Issue 2, March 2008, pp. 63-81.
- Liu W, Sewell C, Blevins N, Salisbury K, Bodin K, Hjelte N. Representing Fluid with Smoothed Particle Hydrodynamics in a Cranial Base Simulator. Proceedings of Medicine Meets Virtual Reality (MMVR), Long Beach, CA, February 2008.
- Sewell C, Blevins N, Peddamatham S, Tan HZ, Morris D, Salisbury K. The Effect of
Virtual Haptic Training on Real Surgical Drilling Proficiency. Proc. of IEEE World Haptics Conference, Tsukuba, Japan, March 2007, pp. 601-603.
- Sewell C, Morris D, Blevins N, Agrawal S, Dutta S, Barbagli F, Salisbury K. Validating Metrics for a Mastoidectomy Simulator. Proceedings of MMVR (Medicine Meets Virtual Reality), February 2007.
- Sewell C, Morris D, Blevins N, Barbagli F, Salisbury K. Evaluating Drilling and Suctioning Technique in a Mastoidectomy Simulator . Proceedings of MMVR (Medicine Meets Virtual Reality), February 2007.
- Sewell C, Morris D, Blevins N, Barbagli F, Salisbury K. Achieving Proper Exposure in Surgical Simulation. Proceedings of MMVR (Medicine Meets Virtual Reality), January 2006.
- Sewell C, Morris D, Blevins N, Barbagli F, Salisbury K. Quantifying Risky Behavior in Surgical Simulation. Proceedings of MMVR (Medicine Meets Virtual Reality), January 2005.
- Sewell C, Morris D, Blevins N, Barbagli F, Salisbury K. An Event-Driven Framework for the Simulation of Complex Surgical Procedures. Proceedings of MICCAI, September 2004. Springer-Verlag Lecture Notes in Computer Science Volumes 3216 and 3217.
Our simulation runs on a dual-CPU Windows PC, and was developed in C++. OpenGL is used for graphic rendering (optionally in stereo, via shutter glasses or a HMD), and two SensAble Phantom haptic devices are used for simulating a drill and a suction device. We typically use a Phantom 1.0 for the drill and a Phantom Omni for the suction device. Sound is rendered as a noisy sum of harmonics, whose frequencies are based on experimental recordings of surgical drill burrs.
Several features, including sound and dynamic bone dust movement, are best demonstrated via video; each section below contains a link to a short movie. Most of the movies are encoded in mp4, either as QuickTime mp4 files or as avi files. If you have trouble playing them, I recommend downloading the 3ivx codec, a nice (free) mp4 codec package.
This video (wmv) is an overview of the system, showing several clinical applications simultaneously.
Application 1: Temporal Bone Surgery
|This video (QuickTime mp4) demonstrates basic drilling on the surface of the mastoid bone. Note the simulated drill sound, whose frequency varies realistically with applied force. As bone dust accumulates, the surgeon uses his suction tool to make the surface visible again.|
|This video (QuickTime mp4) demonstrates the virtual nerve monitor that is incorporated into our simulation environment. The surgeon is operating deep in the ear canal and has exposed the facial nerve; when he approaches it with the drill, the visual and audio monitors indicate increased activity, a warning that the nerve is at risk of being damaged.|
|This video (3ivx-encoded avi) demonstrates the interface used in the simulator to select tools from a menu. The user pulls the haptic device out of the working field – as a surgeon would do before requesting a new tool – and selects a new tool from a “haptic menu”. Here the user demonstrates the drill, suction, flashlight, and camera tools.|
|This video (3ivx-encoded avi) demonstrates the implementation of drill surface nonuniformity (the tendency of the equator of a drill to remove bone more effectively than the pole) in our environment.|
|This video (3ivx-encoded avi) demonstrates the implementation of drill “running forces” (tangential forces) in our environment. The red arrow indicates the tangential force. This video demonstrates the sensitivity of this force to drill orientation.|
|This video (3ivx-encoded avi) demonstrates the use of our metrics console for the playback and evaluation of performance on the simulator.|
Our system was originally developed to simulate temporal bone surgeries, and was built around a set of polygonal models supplied by a collaborating otologist. The temporal bone itself is converted to a voxel representation, which is used for haptic rendering and for simulating material removal. The following sequence of images shows a typical manipulation of a temporal bone.
Important features displayed here are:
- The bone surface is re-triangulated as voxels are removed, allowing real-time rendering of the bone as a polygonal mesh
- Shadows provide a critical depth cue; the bone surface was difficult to find initially, even with haptic feedback, in preliminary experiments in which shadows were not present.
- A particle system is used to represent accumulated bone dust, which trains a practicing surgeon to use the suction tool frequenly to keep the relevant area adequately visible.
Click on any image for a larger version.
Further drilling reveals the complex inner structure of the temporal bone (note the gaps representing the mastoid air cells) and a gradual change in color as the user approaches the anatomy situated behind the bone (in this case a blue tint, representing the sigmoid sinus). This is a critical cue intraoperatively. Also note that before applying suction, the accumulated bone dust obscures this important information.
The following sequence of images also includes the virtual neural monitor, which displays the simulated activity of the facial nerve (in this example). If the drill approaches the monitored nerve, the activity meter begins spiking visually and via audio. It is also possible to stream neural activity information to a remote computer, simulating the neural monitoring station often
used in otologic procedures.
A final sequence of images displays a zoomed-in view that the surgeon might use for operating deep within the bone. Additional drilling also reveals the inner ear anatomy present at the end of the ear canal.
Application 2: Craniofacial Surgery
|This video (QuickTime mp4) demonstrates basic drilling on the surface of the skull, near a mandibular fracture. The user is making the first fracture typically introduced in a distraction osteogenesis procedure. Note that as the tool approaches the alveolar nerve, the virtual nerve monitor provides a visual and auditory warning. For demonstration, the user then exposes the nerve visually.|
|This image displays a complete skull, originally derived from a CT scan. The CT data was isosurfaced to isolate the bone, then converted into the simulation environment’s volumetric representation.|
|Here the user prepares to begin drilling near a mandibular fracture. Shadows again provide a useful depth cue, although depth presentation is enhanced significantly by the optional stereo display.|
|Here we represent a virtual nerve in blue, for visual clarity. Note the bursts of activity on the neural monitor, which result from close contact between the drill and the virtual nerve.|
|The user begins to drill on the surface of the bone. Again, bone dust accumulates and obscures the user’s view until suction is applied.|
Application 3: Dental Surgery
|This video (QuickTime mp4) demonstrates basic drilling on the surface of a tooth, illustrating the flexibility of our environment in loading diverse data sets.|
Additional images provide more detailed views of the bone surface during simulated drilling…