A $p$-K\"ahler structure on a complex manifold of complex dimension $n$ is given by a $d$-closed transverse real $(p,p)$-form. In the paper we study the existence of $p$-K\"ahler structures on compact quotients of simply connected Lie groups by discrete subgroups endowed with an invariant complex structure. In particular, we discuss the existence of $p$-K\"ahler structures on nilmanifolds, with a focus on the case $p =2$ and complex dimension $n = 4$. Moreover, we prove that a $(n-2)$-K\"ahler almost abelian solvmanifold of complex dimension $n\geq3$ has to be K\"ahler.