Behavioral & Cognitive Neuroscience Seminar : Immersed Cognition: studying active vision and memory

As we navigate the world, actions such as eye, head, and whole-body movements structure the input to the visual system and prioritize the maintenance of action-relevant information in visual working memory. By contrast, many studies in the domain of visual perception and cognition rely on experiments where observers passively fixate a point on a 2D computer screen. Here, we will motivate a research agenda of studying visual cognition in “immersed observers”—that is, active observers within environments that foster and model natural sensorimotor contingencies. First, we will show how saccadic eye movements constitute a selection mechanism in visual working memory and hence demonstrate how active exploration also shapes cognition. Based on a meta-analytic approach, we will demonstrate the robustness of saccadic selection in memory across observers and eye movement directions, as well as its association with saccade metrics on a single-trial level. Second, we will present how immersive VR, namely stereoscopic head-mounted displays that adjust the visual input in near-real time based on the user’s head movements, emerges as a promising tool to study comparable phenomena beyond traditional 2D setups. Newer generations of headsets offer integrated eye tracking, making them particularly interesting for researchers investigating eye movements. Based on recent research, in which we have combined VR with eye tracking and EEG, we will demonstrate opportunities and challenges that arise when investigating cognitive and perceptual phenomena such as visual working memory and attention using current VR technology.