Inferior temporal (IT) cortex is critical for visual memory, but it is not known how IT neurons retain memories while new information is streaming into the visual system. Single neurons were therefore recorded from IT cortex of two rhesus monkeys performing tasks that required them to hold items in memory while concurrently viewing other items. The neuronal response to an incoming visual stimulus was attenuated if it matched a stimulus actively held in working memory, even when several other stimuli intervened. The neuronal response to novel stimuli declined as the stimuli became familiar to the animal. IT neurons appear to function as adaptive mnemonic "filters" that preferentially pass information about new, unexpected, or not recently seen stimuli.