Rail transport is a key stepping stone in the EU's transport policy and is pinpointed for investment and growth over the coming decades. This expanding infrastructure implies increased exposure to environmental stressors, such as noise and ground-borne vibrations. Little is known about the health impacts of exposure to these vibrations. The aim of this paper is to examine the association between annoyance from rail vibrations and the distance of residential dwelling from the railway. It reports the first results of a large epidemiological study, EpiVib, which was designed to investigate the long-term health effects of exposure to rail vibrations. The first part of this study examines a self-reported questionnaire. In total, 6894 individuals aged between 18 and 80 living within 1 km of a railway in west Sweden participated. Results presented here examine the association between distance to the railway and annoyance from vibrations and are stratified by train type. A positive association between closer distance and increased annoyance is seen. After adjustment for important modifiers, results showed that vibrations from freight trains and maintenance operations are reported to be moderately and highly annoying at distances of up to 400 m from the railway and diesel up to 300 m. Vibration from passenger and fast trains are significantly annoying up to 200 m from the track. Vibration from freight trains and maintenance operations were considered highly annoying up to 300 m from the track, diesel up to 400 m. Vibration from passenger and fast trains are not reported to be highly annoying after adjustment. Heavier, slower moving locomotives, in the form of diesel and freight trains, appear to be the source of annoyance at distances further from the railway compared to passenger and fast trains. This has implications in terms of property, transport, and infrastructure planning.