In fine-grained image classification, there are only very subtle differences between classes. It is challenging to learn local discriminative features and remove distractive features in fine-grained image classification. Existing fine-grained image classification methods learn discriminative feature mainly via manual part annotation or attention mechanisms. However, due to the large intraclass variance and interclass similarity, the discriminative information and distractive information still are not distinguished effectively. To address this problem, we propose a feature relocation network (FRe-Net) which takes advantage of the different natures of features learned from different stages of the network. Our network consists of a distractive feature learning module and a relocated high-level feature learning module. In the distractive feature learning module, we propose to exploit the difference between low-level features and high-level features to design a distractive loss L, which guides the attention to locate distractive regions more accurately. In the relocated high-level feature learning module, we enhance the representing capacity of the middle-level feature via the attention module and subtract the distractive feature learned from the distractive feature learning module in order to learn more local discriminative features. In end-to-end model training, the distractive feature learning module and the relocated high-level feature learning module are beneficial to each other via joint optimization. We conducted comprehensive experiments on three benchmark datasets widely used in fine-grained image classification. The experimental results show that FRe-Net achieves state-of-the-art performance, which validates the effectiveness of FRe-Net.