Vision-Language models can understand Objects and even actions depicted in images really well. However they lack robust understanding of spatial relationships. Even when they are trained on auxiliary dense prediction tasks the integration of those localisation skills with the language falls short. This project aims to build better evaluation for spatial understanding and also to contribute stronger models in the future.