In this paper, we present our approach to robust background modelling which combines visible and thermal infrared spectrum data. Our work is based on the non-parametric background model describe in 1. We use a pedestrian detection module to prevent erroneous data from becoming part of the background model and this allows us to initialise our bacjground model, even in the presence of foreground objects. Visible and infrared features are use to remove incorrectly detected foreground regions. Allowing our model to quickly recover from ghost regions and rapid lighting changes. An object-based shadow detector also improves our algorithm's performance.