This paper proposes an approach for detection and tracking of multiple objects in a video. We detect multiple objects in the frames using an improved version of the Viola-Jones face-detector, extract Speeded Up Robust Features (SURF) from the detected objects and initialize an improved version of the Kanade-Lucas-Tomashi (KLT) tracker to track the objects throughout the video. We use Gradient Weighted Optical Flow (GWOF) feature to detect both the static and moving objects. The improvement over the KLT tracker is done using the GWOF measure, enabling the tracking system to work in videos with camera shaking. The proposed object tracking method is capable of dealing with multiple challenges like illumination changes, variable and uneven background and poor lighting condition. The efficacy of the proposed approach is tested on challenging datasets like ALOV++ and Honda/UCSD, compared to the state-of-the-art. © 2015 IEEE.