The investigation of material properties based on atomic structure is a commonly used approach. However, in the study of complex systems such as high-entropy alloys, atomic structure not only covers an excessively vast chemical space, but also has an imprecise correspondence to chemical properties. Herein, we present a label-free machine learning (ML) model based on physics-based spectroscopic descriptors to study the catalytic properties of AgAuCuPdPt high-entropy alloy catalysts. Even if the atomic structures of two such alloys are different, these alloys may have similar catalytic properties if their spectral characteristics match closely. One cluster with the strongest CO adsorption exhibited high selectivity for C product generation, indicating that the spectra-based ML model can provide deeper chemical insight than one based on atomic structure. Moreover, such a model can be extended to other systems with consistent results, thus demonstrating its transferability and versatility. This not only underscores the potential of spectral analysis in identifying high-performance alloy catalysts, but facilitates the formation of a new spectra-based modeling approach and research theory in materials science.