The deepest X-ray view of high-redshift galaxies: Constraints on low-rate black hole accretion