Contextual information is vital for the robust extraction of semantic information in automated surveillance systems. We have developed a scene independent framework for the detection of events in which we provide 2D and 3D contextual data for the scene under surveillance via a novel fast and convenient interface tool. In addition, the proposed framework illustrates the use of integral images, not only for detection, as with the classic Viola-Jones object detector, but also for efficient tracking. Finally, we provide a quantitative assessment of the performance of the proposed system in a number of physical locations via groundtruthed datasets.