We compute the Hochschild cohomology of Hilbert schemes of points on surfaces and observe that it is, in general, not determined solely by the Hochschild cohomology of the surface, but by its "Hochschild-Serre cohomology": the bigraded vector space obtained by taking Hochschild homologies with coefficients in powers of the Serre functor. As applications, we obtain various consequences on the deformation theory of the Hilbert schemes; in particular, we recover and extend results of Fantechi, Boissi\`ere, and Hitchin. Our method is to compute more generally for any smooth proper algebraic variety $X$ the Hochschild-Serre cohomology of the symmetric quotient stack $[X^n/\mathfrak{S}_n]$, in terms of the Hochschild-Serre cohomology of $X$.