Dynamic object tracking in complex environments (surveillance, monitoring scenarios, etc.) is very challenging in accurately following objects while maintaining context awareness and respecting privacy. This project addresses this problem by exploring multi-modal semantic fusion techniques for enhancing context-aware dynamic object tracking. We aim to develop an advanced context- and semantic-aware Multimodal Object Tracking framework while preserving individuals’ privacy using trajectory prediction to deal with several challenges, such as occlusions, fast motion, varying lighting and scale, and background clutter.