The family of Willmore immersions from a Riemann surface into $S^{n+2}$ can be divided naturally into the subfamily of Willmore surfaces conformally equivalent to a minimal surface in $\mathbb{R}^{n+2}$ and those which are not conformally equivalent to a minimal surface in $\mathbb{R}^{n+2}$. On the level of their conformal Gauss maps into $Gr_{1,3}(\mathbb{R}^{1,n+3})=SO^+(1,n+3)/SO^+(1,3)\times SO(n)$ these two classes of Willmore immersions into $S^{n+2}$ correspond to conformally harmonic maps for which every image point, considered as a 4-dimensional Lorentzian subspace of $\mathbb{R}^{1,n+3}$, contains a fixed lightlike vector or where it does not contain such a ``constant lightlike vector''. Using the loop group formalism for the construction of Willmore immersions we characterize in this paper precisely those normalized potentials which correspond to conformally harmonic maps containing a lightlike vector. Since the special form of these potentials can easily be avoided, we also precisely characterize those potentials which produce Willmore immersions into $S^{n+2}$ which are not conformal to a minimal surface in $\mathbb{R}^{n+2}$. It turns out that our proof also works analogously for minimal immersions into the other space forms.