This paper presents a preliminary exploration showing the surprising effect of extreme parameter values used by Support Vector Machine (SVM) classifiers for identifying objects in images. The Radial Basis Function (RBF) kernel used with SVM classifiers is considered to be a state-of-the-art approach in visual object classification. Standard tuning approaches apply a relative narrow window of values when determining the main parameters for kernel size. We evaluated the effect of setting an extremely small kernel size and discovered that, contrary to expectations, in the context of visual object classification for some object and feature combinations these small kernels can demonstrate good classification performance. The evaluation is based on experiments on the TRECVid 2013 Semantic INdexing (SIN) training dataset and provides initial indications that can be used to better understand the optimisation of RBF kernel parameters.