Varenicline is a potent agonist of the human 5-hydroxytryptamine3 receptor.

Varenicline, a widely used and successful smoking cessation agent, acts as a partial agonist at nicotinic acetylcholine receptors. Here, we explore the effects of varenicline at human and mouse 5-Hydroxytryptamine(3) (5-HT(3)) receptors. Application of varenicline to human 5-HT(3) receptors expressed in Xenopus laevis oocytes reveal it is almost a full agonist (R(max) = 80%) with an EC(50) (5.9 μM) 3-fold higher than 5-HT. At mouse 5-HT(3) receptors varenicline is a partial agonist (R(max) = 35%) with an EC(50) (18 μM) 20-fold higher than 5-HT. Displacement of the competitive 5-HT(3) receptor antagonist [(3)H]granisetron reveals similar IC(50) values for varenicline at mouse and human receptors expressed in human embryonic kidney 293 cells, although studies in these cells using a membrane potential-sensitive dye show that again varenicline is a 4- or 35-fold less potent agonist than 5-HT in human and mouse receptors, respectively. Thus the data suggest that the efficacy, but not the affinity, of varenicline is greater at human 5-HT(3) receptors compared with mouse. Docking studies provide a possible explanation for this difference, because they suggest distinct orientations of the ligand in the mouse versus human 5-HT(3) agonist binding sites. Additional binding selectivity studies in a broad panel of recombinant receptors and enzymes confirmed an interaction with 5-HT(3) receptors but revealed no additional interactions of varenicline. Therefore, activation of human 5-HT(3) receptors may be responsible for some of the side effects that preclude use of higher doses during varenicline treatment.