BodyWindows: enhancing a mannequin with projective augmented reality for exploring anatomy, physiology and medical procedures.

Augmented reality offers the potential to radically extend and enhance the capabilities of physical medical simulators such as full-body mannequin trainers. We have developed a system that transforms the surface of a mannequin simulator into both a display screen and an input device. The BodyWindows system enables a user to open, size, and reposition multiple viewports onto the simulator body. We demonstrate a dynamic viewport that displays a beating heart. Similar viewports could be used to display real-time physiological responses to interventions the user applies to the mannequin, such as injection of a simulated drug. Viewport windows can be overlapping and show anatomy at different depths, creating the illusion of "cutting" multiple windows into the body to reveal structures at different depths from the surface. The developed low-cost interface employees an IR light pen and the Nintendo Wiimote. We also report experiments using the Microsoft Kinect computer vision sensor to provide a completely hand-gesture based interface.