The importance of intergalactic structure to gravitationally lensed quasars