Three-dimensional (3D) tissue imaging methods are expected to improve surgical management of cancer. In this study, we examined the feasibility of two 3D imaging technologies, optical coherence tomography (OCT) and optical coherence microscopy (OCM), to view human breast specimens based on intrinsic optical contrast. Specifically, we imaged 44 ex vivo breast specimens including 34 benign and 10 malignant lesions with an integrated OCT and OCM system developed in our laboratory. The system enabled 4-μm axial resolution (OCT and OCM) with 14-μm (OCT) and 2-μm (OCM) transverse resolutions, respectively. OCT and OCM images were compared with corresponding histologic sections to identify characteristic features from benign and malignant breast lesions at multiple resolution scales. OCT and OCM provide complimentary information about tissue microstructure, thus showing distinctive patterns for adipose tissue, fibrous stroma, breast lobules and ducts, cysts and microcysts, as well as in situ and invasive carcinomas. The 3D imaging capability of OCT and OCM provided complementary information to individual 2D images, thereby allowing tracking features from different levels to identify low-contrast structures that were difficult to appreciate from single images alone. Our results lay the foundation for future in vivo optical evaluation of breast tissues, using OCT and OCM, which has the potential to guide core needle biopsies, assess surgical margins, and evaluate nodal involvement in breast cancer.