Early diagnosis and therapeutic intervention for Alzheimer's disease (AD) is currently the only viable option for improving clinical outcomes. Combining structural magnetic resonance imaging (sMRI) and resting-state functional magnetic resonance imaging (rs-fMRI) to diagnose AD has yielded promising results. Most studies assume fixed time lags when constructing functional networks. Since the propagation delays between brain signals are constantly changing, these methods cannot reflect more detailed relationships between brain regions. In this work, we use a deep learning-based Granger causality estimator for brain connectivity construction. It exploits the strength of long short-term memory in ever-changing time series processing. This research involves data analysis from sMRI and rs-fMRI. We use sMRI to analyze the cerebral cortex properties and use rs-fMRI to analyze the graph metrics of functional networks. We extract a small subset of optimal features from both types of data. A support vector machine (SVM) is trained and tested to classify AD (n = 27) from healthy controls (n = 20) using rs-fMRI and sMRI features. Using a subset of optimal features in SVM, we achieve a classification accuracy of 87.23% for sMRI, 78.72% for rs-fMRI, and 91.49% for combined sMRI with rs-fMRI. The results show the potential to identify AD from healthy controls by integrating rs-fMRI and sMRI. The integration of sMRI and rs-fMRI modalities can provide supplemental information to improve the diagnosis of AD relative to either the sMRI or fMRI modalities alone.