What are the innate neural mechanisms scaffolding the protracted development of sophisticated human cognition observable later in life? We investigate this question by focusing on the putative hub of the human semantic memory system-the temporal pole. Combining infant- and twin-based imaging analyses, we examine the ontogenetic mechanisms and network characteristics of the functional subdivisions within the temporal pole that are specialized for semantic processing of different types in adults. Our findings reveal topologically similar temporal pole parcellations in the adult and neonatal brains. Notably, the specific functional connectivity of the dorsal and ventrolateral subdivisions with semantic-related networks are evident in neonates, significantly heritable, and associated with semantic functions in adult twins. These results demonstrate the neonatal emergence of genetically programmed functional connectivity characteristics in the temporal pole parcellations that underlie its crucial role in semantic processing, highlighting the innate network mechanisms that support semantic cognition in humans.