Insulin resistance (IR) is associated with endothelial dysfunction. A defect in endothelium-dependent relaxation via outward potassium conductance has been observed in mesenteric arteries from IR rats. The purpose of this study was to assess whether this defect in endothelium-dependent relaxation was due to impaired endothelium-derived hyperpolarizing factor (EDHF) and to determine which specific potassium channel(s) are involved in relaxation. This was accomplished by using specific potassium channel inhibitors in the presence of nitric oxide synthase and cyclooxygenase inhibition. In addition, we sought to assess the function of smooth muscle cell adenosine triphosphate (ATP)-dependent potassium (K(ATP)) channels. Sprague-Dawley rats were randomized to control or IR. To determine EDHF-mediated relaxation, acetylcholine (ACh)-induced (10(-9)-10(-5) M) relaxation was measured (in vitro) in mesenteric arteries in the presence of indomethacin (10(-5) M) and N-nitro-L-arginine (L-NNA) (10(-4) M). Subsequently the combination of charybdotoxin (CTX) (0.1 microM) and apamin (0.5 microM) or glibenclamide (Glib) (10 microM) was added to the bath to inhibit KCa or K(ATP), respectively. In separate experiments, relaxation to pinacidil (10(-13)-10(-5) M), a K(ATP) activator, was assessed in vessels with intact endothelium, endothelium denuded, or with L-NNA. Maximal relaxation to ACh in the presence of L-NNA and indomethacin was 68+/-6% for control and 12+/-3% for IR (p<0.01). The addition of CTX + apamin almost abolished EDHF-mediated relaxation in control (Emax, 8+/-5% vs. 68+/-6%; p<0.01), whereas Glib had little affect. Neither CTX + apamin nor Glib had any affect on IR. Additionally, IR arteries were less sensitive to pinacidil than were controls (EC50, 1.5+/-0.9 microM vs. 5x10(-4)+/-3x10(-4) microM, respectively; p<0.01). Endothelial removal or L-NNA pretreatment of control arteries decreased the response to pinacidil similar to IR, whereas IR vessels were unaffected. EDHF-mediated relaxation is impaired in IR arteries. In addition, the K(Ca) channel appears to be imperative for activity of EDHF in rat small mesenteric arteries. Moreover, activation of K(ATP) channels by pinacidil is impaired in IR, and this appears to be a result of endothelial dysfunction.