For a general affine connection with parallel torsion and curvature, we show that a post-Lie algebra structure exists on its space of vector fields, generalizing previous results for flat connections. However, for non-flat connections, the vector fields alone are not enough, as the presence of curvature also necessitates that we include endomorphisms corresponding to infinitesimal actions of the holonomy group. We give details on the universal Lie algebra of this post-Lie algebra and give applications for solving differential equations on manifolds.