This paper explores the fibers of the principal minor map over a general field. The principal minor map is the map that assigns to each $n\times n$ matrix the $2^n$-vector of its principal minors. In $1984$, Hartfiel and Loewy proposed a condition that was sufficient to ensure that the fiber of the principal minor map is a single point up to diagonal equivalence. Loewy later improved upon this condition in $1986$. In this paper, we provide a necessary and sufficient condition for the fiber to be a point up to diagonal equivalence. Additionally, we establish a connection between the reducibility of a matrix and the reducibility of its determinantal representation. Using this connection, we fully characterize the fiber of symmetric and Hermitian matrices in the space of $n\times n$ matrices over any field $\mathbb{F}$. We also use these techniques to answer a question of Borcea, Br\"and\'en, and Liggett concerning real stable matrices.