Molecular determinants for recognition of RU 24969 analogs at central 5-hydroxytryptamine recognition sites: use of a bilinear function and substituent volumes to describe steric fit.

The putative serotonin (5-HT) agonist RU 24969 [5-methoxy-3-1,2,3,6-tetrahydropyridin-4-yl)indole; 5-MeO-THPI] has been extensively used in the study and classification of 5-HT receptors. In order to study molecular determinants for recognition of THPIs at central 5-HT recognition sites, about 25 additional THPI derivatives were synthesized, incorporating, among others, 16 different indole-5-substituents and three different pyridine-N substituents in various combinations. Two saturated derivatives (piperidin-4-ylindoles) and two 2-methyl analogs were also included. Binding affinities at 5-HT1A, 5-HT2, and total 5-HT1 sites were obtained and the data were incorporated in quantitative structure-activity relationships (QSARs) using a combined linear free energy/molecular modeling approach. The QSAR analyses suggest distinct differences in the structural features that determine optimal potency at 5-HT1A sites versus those directing optimal potency for 5-HT2 sites. The parameter of the indole-5 substituent that almost exclusively determines potency for 5-HT1A sites is volume, the optimal size being about 24 cubic angstroms (calculated by fitting the activity versus volume data to a bilinear function). This is approximately the size of a carboxamide group. In contrast, at the 5-HT2 site both volume and hydrophobicity play major but opposing roles for the 5-substituent. A balance between the smallest possible volume and the greatest possible hydrophobicity is required for maximal 5-HT2 potency. Benzyl groups on the indole-1 or pyridyl-1 positions also favor potency at the 5-HT2 site (probably largely due to increased hydrophobic binding) while decreasing potency at the 5-HT1A site. A minor electronic contribution to the QSARs involving the charge on the indole 5-carbon is of opposite sign for 5-HT1A versus 5-HT2 sites and thus may also be useful for selective drug design. The data are consistent with the possibility that the indole and pyridyl rings are in a coplanar configuration when binding at both 5-HT1A and 5-HT2 sites, because the indole-2-methyl substituent, which provides a large energy barrier to the coplanar configuration, greatly reduces the potency of THPIs at both binding sites. Similarities in analog selectivity patterns suggest that the indolic portion of these compounds binds similarly to that of other indole derivatives such as tryptamines; thus, it is possible that optimally selective substituents predicted by these QSARs may be extrapolated to tryptamines and other indoles.