Swept-source optical coherence tomography (SS-OCT) is a widely used imaging technique, particularly in medical diagnostics, due to its ability to provide high-resolution cross-sectional images. However, one of the main challenges in SS-OCT systems is the nonlinearity in wavelength sweeping, which leads to degraded depth resolution after Fourier transform. Correcting for this nonlinearity typically requires complex re-sampling and chirp compensation methods. In this paper, we introduce the first ultrafast time-stretch optical coherence tomography (TS-OCT) system that utilizes reservoir computing (RC) to perform direct temporal signal analysis without relying on Fourier transform techniques. By focusing solely on the temporal characteristics of the interference signal, regardless of frequency chirp, we demonstrate a more efficient solution to address the nonlinear wavelength sweeping issue. By leveraging the dynamic temporal processing capabilities of RC, the proposed system effectively bypasses the challenges faced by Fourier analysis, maintaining high-resolution depth measurement without being affected by chirp-introduced spectral broadening. The system operates by categorizing the interference signals generated by variations in sample position. This classification-based approach simplifies the data processing pipeline. We developed an RC-based model to interpret the temporal patterns in the interferometric signals, achieving high classification accuracy. A proof-of-the-concept experiment demonstrated that this method allows for precise depth resolution, independent of system chirp. With an A-scan rate of 50 MHz, the classification model yielded 100% accuracy with a root mean square error (RMSE) of 0.2416. This approach offers a robust alternative to Fourier-based analysis, particularly in systems prone to nonlinearities during signal acquisition.