We study the structure of the Albanese map for K\"ahler manifolds with nef anticanonical bundle. First, we give a result for fourfolds whose Albanse torus is an elliptic curve. In the general case of any dimension, we look at two cases: The general fiber of the Albanese map is a Calabi-Yau manifold or a projective space. In the first case, we show that the manifold itself must be Calabi-Yau. In the second case, we give a more topological proof of a result by Cao and H\"oring which says that the manifold must be a projectivization of a numerically flat vector bundle.