Indoor physical environments appear to influence learning efficiency nowadays. For improvement in learning efficiency, environmental scenarios need to be designed when occupants engage in different learning tasks. However, how learning efficiency is affected by indoor physical environment based on task types are still not well understood. The present study aims to explore the impacts of three physical environmental factors (i.e., temperature, noise, and illuminance) on learning efficiency according to different types of tasks, including perception, memory, problem-solving, and attention-oriented tasks. A 3 × 4 × 3 full factorial design experiment was employed in a university classroom with 10 subjects recruited. Environmental scenarios were generated based on different levels of temperature (17 °C, 22 °C, and 27 °C), noise (40 dB(A), 50 dB(A), 60 dB(A), and 70 dB(A)) and illuminance (60 lx, 300 lx, and 2200 lx). Accuracy rate (AC), reaction time (RT), and the final performance indicator (PI) were used to quantify learning efficiency. The results showed ambient temperature, noise, and illuminance exerted significant main effect on learning efficiency based on four task types. Significant concurrent effects of the three factors on final learning efficiency was found in all tasks except problem-solving-oriented task. The optimal environmental scenarios for top learning efficiency were further identified under different environmental interactions. The highest learning efficiency came in thermoneutral, relatively quiet, and bright conditions in perception-oriented task. Subjects performed best under warm, relatively quiet, and moderately light exposure when recalling images in the memory-oriented task. Learning efficiency peaked to maxima in thermoneutral, fairly quiet, and moderately light environment in problem-solving process while in cool, fairly quiet and bright environment with regard to attention-oriented task. The study provides guidance for building users to conduct effective environmental intervention with simultaneous controls of ambient temperature, noise, and illuminance. It contributes to creating the most suitable indoor physical environment for improving occupants learning efficiency according to different task types. The findings could further supplement the present indoor environment-related standards or norms with providing empirical reference on environmental interactions.