In this paper, we present a novel method for intra-operative registration directly from monocular endoscopic images. This technique has the potential to provide a more accurate surface registration at the surgical site than existing methods. It can operate autonomously from as few as two images and can be particularly useful in revision cases where surgical landmarks may be absent. A by-product of video registration is an estimate of the local surface structure of the anatomy, thus providing the opportunity to dynamically update anatomical models as the surgery progresses. Our approach is based on a previously presented method [Burschka, D., Hager, G.D., 2004. V-GPS (SLAM):--Vision-based inertial system for mobile robots. In: Proceedings of ICRA, 409-415] for reconstruction of a scaled 3D model of the environment from unknown camera motion. We use this scaled reconstruction as input to a PCA-based algorithm that registers the reconstructed data to the CT data and recovers the scale and pose parameters of the camera in the coordinate frame of the CT scan. The result is used in an ICP registration step to refine the registration estimates. The details of our approach and the experimental results with a phantom of a human skull and a head of a pig cadaver are presented in this paper.