A saliency attention model for predicting eye direction is proposed in this paper. This work is inspired by the success of the topological structure and Earth Mover's Distance (EMD) approach. Firstly, we extract visual saliency features such as color contrast, intensity contrast, orientation, and texture. Then, we eliminate disconnected regions in the feature maps to keep topological structure. Secondly, we calculate center surround difference using across-scale EMD between different scales feature maps, rather than utilizing the Difference of Gaussian (DoG), which is used in many other saliency attention models. Thirdly, we across-scale fuse the feature maps in different scale and same feature. Lastly, we take advantage of competition function to calculate feature maps in same feature to form a saliency map, which is use to predict eye direction. Experimental results demonstrated the proposed model outperformed the state-of-the-art schemes in eye direction prediction community.