We study the integrability to second order of infinitesimal Einstein deformations on compact Riemannian and in particular on K\"ahler manifolds. We find a new way of expressing the necessary and sufficient condition for integrability to second order, which also gives a very clear and compact way of writing the Koiso obstruction. As an application we consider the K\"ahler case, where the condition can be further simplified and in complex dimension $3$ turns out to be purely algebraic. One of our main results is the complete and explicit description of infinitesimal Einstein deformation integrable to second order on the complex $2$-plane Grassmannian, which also has a quaternion K\"ahler structure. As a striking consequence we find that the symmetric Einstein metric on the Grassmannian $ \mathrm{Gr}_2(\bbC^{n+2})$ for $n$ odd is rigid.