The experience of passage of time is assumed to be a constitutive component of our subjective phenomenal experience and our everyday life that is detached from the estimation of time durations. However, our understanding of the factors contributing to passage of time experience has been mostly restricted to associated emotional and cognitive experiences in temporally extended situations. Here, we tested the influence of low-level visual stimuli on the experience of passage and duration of time in 10-30 s intervals. We introduce a new paradigm in a starfield environment that allows to study the effects of basic visual aspects of a scene (velocity and density of stars in the starfield) and the duration of the situation, both embedded in a color tracking task. Results from two experiments show that velocity and density of stars in the starfield affect passage of time experience independent from duration estimation and the color tracking task: the experienced passage of time is accelerated with higher rates of moment-to-moment changes in the starfield while duration estimations are comparably unaffected. The results strongly suggest differential psychological processes underlying the experience of time passing by and the ability to estimate time durations. Potential mechanisms behind these results and the prospects of experimental approaches towards passage of time experience in psychological and neuroscientific research are discussed.