We study border varieties of sums of powers ($\underline{\mathrm{VSP}}$'s for short), recently introduced by Buczy\'nska and Buczy\'nski, parameterizing border rank decompositions of a point (e.g. of a tensor or a homogeneous polynomial) with respect to a smooth projective toric variety and living in the Haiman-Sturmfels multigraded Hilbert scheme. Their importance stems from the role of border tensor rank in theoretical computer science, especially in the estimation of the exponent of matrix multiplication, a fundamental and still unknown quantity in the theory of computation. We compare $\underline{\mathrm{VSP}}$'s to other well-known loci in the Hilbert scheme, parameterizing scheme-theoretic versions of decompositions. The latter ones are crucial in that they naturally explain the existing severe barriers to giving good lower bounds on ranks. We introduce the notion of border identifiability and provide sufficient criteria for its appearance, which rely on the multigraded regularity of Maclagan and Smith. We link border identifiability to wildness of points. Finally, we determine $\underline{\mathrm{VSP}}$'s in several instances and regimes, in the contexts of tensors and homogeneous polynomials. These include concise $3$-tensors of minimal border rank and in particular of border rank three.