Modern Video Retrieval systems face challenges in computational efficiency and semantic depth when handling complex queries, particularly those with time-sensitive requirements. These systems typically rely on a “flat” index structure that encodes each frame independently, resulting in high search costs and difficulty capturing higher-level events or context semantics. To address these limitations, we propose a novel three-level hierarchical index concept that organizes video data at different semantic abstraction levels. The first level involves embedding vectors for individual frames to facilitate fine-grained retrieval. The second level groups visually similar frames into “shots” and encodes them into a semantic temporal representation. The top layer uses a Visual-Language Model (VLM) to identify and group frames related to narrative actions. This architecture allows the system to first quickly identify high-level related scenes or actions, and then refine the results by searching within individual frames within those groups. Our approach helps users to query data at the most relevant conceptual level.