To maintain normal blood glucose levels, pancreatic beta cells secrete insulin into the bloodstream at specialized regions at the cell periphery, often called secretion hot spots. While many secretory machinery components are located all over the cell membrane, directed secretion relies on distinct cortical patches of the scaffolding protein ELKS and the microtubule (MT)-anchoring protein LL5β. However, using total internal reflection fluorescence microscopy of intact mouse islets to precisely localize secretion events within ELKS/LL5β patches, we now show that secretion is restricted to only 5% of ELKS/LL5β patch area. Moreover, the majority of secretion occurs at the margins of ELKS patches. This suggests that additional factor(s) must be responsible for hot spot definition. Because the MT cytoskeleton plays a regulatory role in the insulin secretion process via both delivery and removal of secretory granules from the secretion sites, we test whether local MT organization defines secretory activity at hot spots. We find that the majority of secretion events occur at regions devoid of MTs. Based on our findings, we present a model in which local MT disassembly and optimal ELKS content are strong predictors of directed insulin secretion.