We proposed a non-contact photoacoustic (PA) detection method using spectral domain optical coherence tomography (SDOCT). Two interference spectrums (A-lines) were acquired before and after the PA excitation with SDOCT. PA signal propagated within the sample causing the vibration. The vibration inner the sample introduced phase change between the acquired two A-lines. Thus, the PA signal can be detected by evaluating the difference in phase between the two A-lines. Based on the method, an OCT-PAM dual-mode imaging system was constructed. In the system, SDOCT served as the detection unit for PAM. Thus, the combination of the two imaging modalities was simplified. Another advantage of the system is that it realizes non-contact all-optic detection, which is attractive for biomedical imaging. Using the system, we imaged phantoms of carbon fibers, asparagus leaves and human hairs. Furthermore, the cortical vasculature of rat was imaged and the flow status was evaluated quantitatively.