Semantic description of scenes is a challenging problem in video retrieval. Early works enlarged the semantic gap owing to the retrieval mechanism based on text matching between textual metadata provided by the up loader and text-based query. The need for bridging the semantic gap between user’s query content and retrieved videos prompted the research in content-based semantic video retrieval. This chapter proposes a semantic video retrieval model for human habitual scene search without textual metadata annotation. The proposed semantic model relies on understanding of the query content in the video to retrieve user’s query over a highly challenging video database. This work encompasses automatic concept/object detection in frames using concept bank, relates the detected concepts using concept occurrence vector to retrieve the relevant videos for the user’s query. The proposed semantic model relies on pedagogical understanding of the query content in the video using the prototype to retrieve users query over a highly challenging video database by poisson binomial radius measure. To evaluate the proposed approach, we validated the concept bank driven semantic retrieval model on user-created database for the user’s query by example. The experimental observation shows the astounding performance of the proposed concept knowledge-based semantic retrieval model. © 2021, Springer Nature Singapore Pte Ltd.