The problem of classifying scenes from cricket video is addressed and a robust framework for this problem is proposed. It is proposed that the finite state machines (FSM) are suitable for detecting and classifying scenes and their usage is demonstrated for three types of events: wicket, six and four. This framework utilises the structural information of the scenes together with the low-level and mid-level features. Low-level features of the video including motion and audio energy, and a mid-level feature, body, are used in this approach. The transitions of the FSMs are determined by the features from each shot in the scene. The FSMs have been experimented on over 80 clips and convincing results have been achieved. Copyright © 2021 Inderscience Enterprises Ltd.