Interpersonal distance, an essential component of social interaction, is modulated by the emotion conveyed by others and associated physiological response. However, in modern societies with overcrowded and hyperstimulating environments, we can only surreptitiously glimpse the faces of others in order to quickly make behavioral adjustments. How this impacts social interactions is not yet well understood. In the present study, we investigated this issue by testing whether facial expressions that are difficult to identify modify the physiological response (Electrodermal Activity, EDA) and subsequent judgment of interpersonal comfort distance. We recorded participants' EDA while they provided comfort judgments to interpersonal distances with a Point-Light Walker (PLW). The PLW, with an emotionally neutral gait, moved toward and crossed participants at various distances after the latter were exposed to a negative (anger), positive (happiness) or neutral facial expression presented at the perceptual threshold. Bayesian analyses of the data revealed an increase versus decrease of interpersonal comfort distance with the PLW depending on the negative versus positive emotional valence of the facial expression. They also showed an increase in EDA when the approaching PLW violated interpersonal comfort distance after participants were exposed to an angry facial expression. These effects correlated with the subjective assessment of the arousal of facial expressions. Thus, previous exposure to barely visible facial expressions can alter the representation of social comfort space and the physiological response associated with a violation of interpersonal comfort distances, depending on the valence and arousal of the emotional social stimuli.