This paper describes the potential role and limitations of current three-dimensional (3D) virtual reality (VR), augmented reality (AR), and holography for image-guided surgery. We developed a new surgical spatial navigation system using VR, AR, and virtual holography. An interactive stereo display is used to view the interactions between the surgeon and stereo images of the patient’s anatomy depicted on the display by tracking the surgeon’s head and hand/arm positions. Sensing the surgeon’s head position creates motion parallax information, an immersive depth cue that can be added to the binocular parallax already present in the display. The beneficial applications of VR/AR devices (head-mounted devices, 3D tablets, and motion sensors) are also discussed. They allow the user to manipulate the spatial attributes of VR, which can enhance spatial reasoning and AR.