Human Activity Recognition in RGB-D videos has been an active research topic during the last decade. However, only a few efforts have been made, for recognizing human activity in RGB-D videos where several performers are performing simultaneously. In this paper we introduce such a challenging dataset with several performers performing the activities simultaniously. We present a novel method for recognizing human activities performed simultaniously in the same videos. The proposed method aims in capturing the motion information of the whole video by producing a dynamic image corresponding to the input video. We use two parallel ResNet-101 architectures to produce the dynamic images for the RGB video and depth video separately. The dynamic images contain only the motion information of the whole frame, which is the main cue for analyzing the motion of the performer during action. Hence, dynamic images help recognizing human action by concentrating only on the motion information appeared on the frame. We send the two dynamic images through a fully connected layer for classification of activity. The proposed dynamic image reduces the complexity of the recognition process by extracting a sparse matrix from a video, while preserving the motion information required for activity recognition, and produces comparable results with respect to the state-of-the-art. © 2020, Springer Science+Business Media, LLC, part of Springer Nature.