The veracity of claims made by researchers and clinicians when reporting the impact of lighting on vision and other biological mechanisms is, in part, reliant on accurate and valid measurement devices. We aim to quantify the intra- and inter-watch variability of a commercially available light sensor device which has been widely used in vision and other photobiological research. Intra- and inter-watch differences were investigated between four Actiwatch Spectrum Pro devices. The devices were used to obtain measurements on two separate occasions, under three different controlled light conditions; the Gretag Macbeth Judge II lightbox was used to produce Simulated Daylight (D65), Illuminant A (A) and Cool White Fluorescent (CWF) lighting. Significant inter-watch differences were noted when considering tricolour (red, green, blue) and the white sensor outputs under each of the three illuminants (p < 0.01). A significant interaction was also found between tricolour sensor and watch used (p < 0.01). Intra-watch differences were noted for the tricolour and for the white sensor outputs under the three illuminants (≤0.05), for all but one watch which showed no significant intra-watch difference for the white 'sensor output' under the D65 illuminant. Use of spectral sensitivity devices is an evolving field. Before drawing causal relationships between light and other biological processes, researchers should acknowledge the limitations of the instruments used, their validation, and the resultant data. The outcomes of the study indicate caution must be exercised in longitudinal data collection and the mixing of watches amongst study participants should be avoided.