We present the results of the first Dalitz plot analysis of the decay $D^0\to K^{-}{\pi }^{+}\eta$. The analysis is performed on a data set corresponding to an integrated luminosity of $953{\mathrm{fb}}^{-1}$ collected by the Belle detector at the asymmetric-energy $e^{+}{e}^{-}$ KEKB collider. The Dalitz plot is well described by a combination of the six resonant decay channels ${\overline{K}}^{*}(892{)}^0\eta$, $K^{-}{a}_0(980{)}^{+}$, $K^{-}{a}_2(1320{)}^{+}$, ${\overline{K}}^{*}(1410{)}^0\eta$, $K^{*}(1680{)}^{-}{\pi }^{+}$ and $K_2^{*}(1980{)}^{-}{\pi }^{+}$, together with $K\pi$ and $K\eta$ S-wave components. The decays $K^{*}(1680{)}^{-}\to K^{-}\eta$ and $K_2^{*}(1980{)}^{-}\to K^{-}\eta$ are observed for the first time. We measure ratio of the branching fractions, $\frac{\mathcal{B}(D^0\to K^{-}{\pi }^{+}\eta )}{\mathcal{B}(D^0\to K^{-}{\pi }^{+})}=0.500\pm 0.002(\mathrm{stat})\pm 0.020(\mathrm{syst})\pm 0.003({\mathcal{B}}_{\mathrm{PDG}})$. Using the Dalitz fit result, the ratio $\frac{\mathcal{B}(K^{*}(1680)\to K\eta )}{\mathcal{B}(K^{*}(1680)\to K\pi )}$ is measured to be $0.11\pm 0.02{(\mathrm{stat})}_{-0.04}^{+0.06}(\mathrm{syst})\pm 0.04({\mathcal{B}}_{\mathrm{PDG}})$; this is much lower than the theoretical expectations ($\approx 1$) made under the assumption that $K^{*}(1680)$ is a pure $1^3{D}_1$ state. The product branching fraction $\mathcal{B}(D^0\to [K_2^{*}(1980{)}^{-}\to K^{-}\eta ]{\pi }^{+})=(2.2_{-1.9}^{+1.7})\times {10}^{-4}$ is determined. In addition, the $\pi {\eta }^{\prime }$ contribution to the $a_0(980{)}^{\pm }$ resonance shape is confirmed with $10.1\sigma$ statistical significance using the three-channel Flatté model. We also measure $\mathcal{B}(D^0\to {\overline{K}}^{*}(892{)}^0\eta )=(1.41_{-0.12}^{+0.13})\%$. This is consistent with, and more precise than, the current world average $(1.02\pm 0.30)\%$, deviates with a significance of more than $3\sigma$ from the theoretical predictions of (0.51–0.92)%.

• Received 17 March 2020
• Accepted 18 June 2020
• Corrected 20 April 2023

DOI:https://doi.org/10.1103/PhysRevD.102.012002

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Funded by SCOAP³.

Published by the American Physical Society