As with excellent catalytic performance, palladium nanoclusters (PdNCs) have a wide range of applications. However, the traditional PdNCs are easy to agglomerate in the analysis system and lose their catalytic activity. A covalent organic framework (COF) has a definite structure, good stability, and easy surface functionalization. So, it is of great significance to develop stable PdNCs with high catalytic activity and then combine with advanced analysis techniques to analyze ultratrace small-molecule pollutants in the environment. In this research, a stable PdNC dispersed on a COF (PdTpPa) catalyst is prepared and we find it with strong catalysis for the NaHPO-HAuCl catalytic reaction. Furthermore, this nanocatalytic indicator reaction can be tracked by surface-enhanced Raman spectroscopy (SERS) and resonance Rayleigh scattering (RRS) dual-mode. Combined with a highly specific aptamer-modifying technique, a highly sensitive and selective SERS/RRS dimode assay platform for trace organic pollutants has been developed. The detection limits of oxytetracycline (OTC), glyphosate (GLY), tetracycline (TEC), and bisphenol A (BPA) are 0.64, 0.03, 6.2 × 10, and 0.53 × 10 ng/mL, respectively. This work also provides ideas for the application of COF materials and Pd nanocatalysts in the molecular spectral detection of trace pollutants.