Search for sterile neutrino mixing in the νμ → ντ appearance channel with the OPERA detector