In this paper we propose a new approach to assess the performance of video anomaly detection algorithms. Inspired by the COCO metrics we propose a quartile based quality assessment of video anomaly detection to have a detailed breakdown of algorithm performance. The proposed assessment divides the detection into five categories based on the measurement quartiles of the position, scale and motion magnitude of anomalies. A weighted precision is introduced in the average precision calculation such that the frame-level average precision reported in categories can be compared to each other regardless of the baseline of the precision-recall curve in every category. We evaluated three video anomaly detection approaches, including supervised and unsupervised approaches, on five public datasets using the proposed approach. Our evaluation shows that the anomaly scale introduces performance difference in detection. For both supervised and unsupervised methods evaluated, the detection achieve higher average precision for the large anomalies in scale. Our assessment also shows that the supervised multiple instance learning method is robust to the motion magnitude differences in anomalies, while the unsupervised one-class neural network method performs better than the unsupervised autoencoder reconstruction method when the motion magnitudes are small. Our experiments, however, also show that the positions of the anomalies have impact on the performance of the multiple instance learning method and the one-class neural network method but the impact on the autoencoder-based approach is negligible.