In video surveillance the moving object detection and tracking are important research area of computer vision. Tracking moving objects in a real time environment is not easy because of continual deformation of objects during movement. The moving item has many attributes in both temporal and spatial spaces. In the spatial area, objects vary in size while in temporal area they vary in moving speed. To track multiple object in video with optimal window size the Decomposition with Temporal Domain (DTD) is proposed. Thus the moving objects are detected and tracked until it disappears or losses it motion. The proposed strategy can identify and track objects all the while. It reduces the False Alarms Rate (FAR) and increases True Positive Rate (TPR). Moving items with various speeds and sizes are tracked with their optimal window size and the possible shadow is eliminated by the multi frame difference method. The proposed method is compared with Kalman filter and Optical Flow (OF) of Lucas-Kanade method using probabilistic approaches such as FAR and TPR performance is analyzed. Different test results have affirmed the legitimacy and adequacy of the proposed technique. © 2016 IEEE.