Inhibition of return (IOR) refers to the performance disadvantage when detecting a target presented at a previously cued location. The current paper contributes to the long-standing debate whether IOR is caused by attentional processing or perceptual processing. We present a series of four experiments which varied the cue luminance in mixed and blocked conditions. We hypothesised that if inhibition was initialized by an attentional process the size of IOR should not vary in the blocked condition as participants should be able to adapt to the level of cue luminance. However, if a perceptual process triggers inhibition both experimental manipulations should lead to varying levels of IOR. Indeed, we found evidence for the latter hypothesis. In addition, we also varied the target luminance in blocked and mixed condition. Both manipulations, cue luminance and target luminance, affected IOR in an additive fashion suggesting that the two stimuli affect human behaviour on different processing stages.