A tale of two populations: Surviving and destroyed dwarf galaxies and the build-up of the Milky Way's stellar halo