Phonon decoherence in quantum high-overtone bulk acoustic-wave resonators is dominated by imperfections under the material surface. Removing the damaged subsurface layer substantially increases the performance of these devices.