The VIMOS-VLT Deep Survey: the build-up of the colour density relation