The steerable needle becomes appealing in the neurosurgery intervention procedure because of its flexibility to bypass critical regions inside the brain; with proper path planning, it can also minimize the potential damage by setting constraints and optimizing the insertion path. Recently, reinforcement learning (RL)-based path planning algorithm has shown promising results in neurosurgery, but because of the trial and error mechanism, it can be computationally expensive and insecure with low training efficiency. In this paper, we propose a heuristically accelerated deep Q network (DQN) algorithm to safely preoperatively plan a needle insertion path in a neurosurgical environment. Furthermore, a fuzzy inference system is integrated into the framework as a balance of the heuristic policy and the RL algorithm. Simulations are conducted to test the proposed method in comparison to the traditional greedy heuristic searching algorithm and DQN algorithms. Tests showed promising results of our algorithm in saving over 50 training episodes, calculating path lengths of 0.35 after normalization, which is 0.61 and 0.39 for DQN and traditional greedy heuristic searching algorithm, respectively. Moreover, the maximum curvature during planning is reduced to 0.046 from 0.139 mm using the proposed algorithm compared to DQN.