We compute odd-degree genus 1 quasimap (and Gromov--Witten) invariants of moduli spaces of Higgs $\mathrm{SL}_2$-bundles on a curve of genus $g\geq2$. We also compute certain invariants for all prime ranks. This proves some parts of author's conjectures on quasimap invariants of moduli spaces of Higgs bundles. More generally, our methods provide a computation scheme for genus 1 quasimap (and Gromov--Witten) invariants in the case when degrees of maps are coprime to the rank. This requires a careful analysis of the localisation formula for certain Quot schemes parametrising higher-rank quotients on an elliptic curve. Invariants for degrees which are not coprime to the rank exhibit a very different structure for a reason that we explain.