Cyclotron radiation emission spectroscopy signal classification with machine learning in project 8

The ^83mKr source emits internal conversion electrons at several energies, which we divide into three nominal groups by the atomic shell of the transition as dictated in table 1.

Table 1.  Relative intensities of ^83mKr conversion electron lines under study in this work. From [6, 7].

Group	Line	Intensity (total $\equiv 1$)	Energy (eV)	Width (eV)
17 keV	K	0.36	17830.0	2.83
30 keV	L2	0.392	30424.4	1.84
	L3	0.581	30477.2	1.4
32 keV	M2	0.067	31934.2	1.99
	M3	0.105	31941.9	1.66
	N2 + 3	0.016	32140.9	0.59

Each transition is monoenergetic, up to a natural linewidth of order eV, which is substantially less than the energy resolution of Phase I. The 17 keV data is closest to the tritium endpoint (18.6 keV), and the higher-energy peaks provide important insight on the relative energy dependence of signal characteristics.

Electron signals emitted in the magnetic trap are received by an antenna and processed by a cryogenic receiver chain described in [1]. The signal is down-mixed twice using local oscillators and sampled at 200 MHz by a real-time spectrum analyzer (RSA). The RSA triggers an acquisition when it detects a high Fourier excess, and writes time-domain data for 10 ms per trigger with a pre-trigger time of 1 ms. The acquisition is then processed with a series of DFTs of size 8192 samples (0.040 96 ms) to produce a spectrogram like the one in figure 1, which displays a CRES event. The resulting spectrograms are scanned for high-power bins in collinear groupings called tracks; for an in-depth discussion the track finding algorithms and procedure, we refer to [3]. The initial frequency of a track signal is called the start frequency, and it is the start frequency of an electron event that contains primary energy information needed for spectrum reconstruction.

All events considered in this study are subject to a cut on the start time of the first track within ±0.25 ms of the pre-trigger time. This retains only events which have promptly triggered an acquisition and removes those which only triggered after some time due to sufficiently enough power; these low-power tracks often start even before the acquisition window, which prohibits a measurement of the start frequency altogether. Furthermore, in preparing the training set for the machine learning analysis we consider only the first track(s) in such events. In the example of figure 1, the second set of tracks would be cut, which is useful in labeling the ground truth, a process described in section 4; by only considering the first set of tracks in an event we can confidently label separate signal classes using both frequency information and other parameter space cuts.

We parameterize the axial motion of an electron with the pitch angle θ(t), defined as the angle between the momentum vector and the magnetic field:

$$\begin{eqnarray}&&\cos \theta (t)=\displaystyle \frac{\vec{p}\cdot \vec{B}(t)}{{pB}(t)}=\displaystyle \frac{{p}_{z}(t)}{p}\ \mathrm{for}\ \vec{B}={B}_{z}\hat{z}.\end{eqnarray} \tag{ 2 }$$

For a trapped electron, p_z has an oscillatory time dependence and therefore so does the instantaneous pitch angle¹¹ . By definition, θ = 90° at the turning points of the axial oscillation as p_z = 0; at the center of the trap, the pitch angle reaches a minimum along with B_z. Thus, the range in z explored by an electron is fully characterized by (a) the trap geometry and (b) the minimum pitch angle. Going forward, we will simply use θ to refer to this minimum pitch angle, rather than the time-varying instantaneous pitch angle θ(t). In fact, there is also a limit to the smallest pitch angle value due to conservation of energy which depends on the ratio of the trap depth and maximum magnetic field, for more on this see [4]. At a nominal 4 mT trap depth the lower bound is about 86° and may be higher depending on the chosen geometry.

A detailed mathematical discussion of trapped electron dynamics and the resultant CRES signal characteristics is outside the scope of this work but may be found in [4]; we refer to their nomenclature for the rest of the paper. There, a phenomenological model is developed using approximate Project 8 magnetic trap geometries to analytically describe the motion of electrons with θ < 90°. The model includes a short opposite the antenna, as is present in the Phase I detector; the result is an energy (frequency) and position-dependent interference effect between the incident and reflected radiation. The power spectrum P(ω) is calculated from the Poynting vector in the axial direction, and decomposed as a sum of waveguide modes. For a single mode denoted by λ, we take advantage of the quasi-periodic motion of the electron in the trap to express the power averaged over the axial period with equation (45) in [4], reproduced here:

$$\begin{eqnarray}\begin{array}{rcl}{P}_{\lambda }(\omega ) & = & 4{P}_{0,\lambda }\displaystyle \sum _{n=-\infty }^{\infty }{\left|{a}_{n}\left({k}_{\lambda }\right)\right|}^{2}{\cos }^{2}\left[({z}_{t}+l){k}_{\lambda }\right]\\ & & \times \,\left[\delta (\omega -{k}_{\lambda }{v}_{p,\lambda }))+\delta (\omega +{k}_{\lambda }{v}_{p,\lambda })\right]{\rm{w}}{\rm{i}}{\rm{t}}{\rm{h}}\,{k}_{\lambda }=\displaystyle \frac{{{\rm{\Omega }}}_{0}\,+\,n{{\rm{\Omega }}}_{a}}{{v}_{p,\lambda }},\end{array}\end{eqnarray} \tag{ 3 }$$

where P_{0, λ} and every a_n are amplitudes dependent on the magnetic trap shape, z_t + l is the distance from the short to the trap center, and v_p,λ is the mode phase velocity. This equation describes a comb-like spectrum with power concentrated at a central frequency Ω₀ (the average cyclotron frequency) and at frequencies shifted by integer multiples n of the axial oscillation frequency Ω_a. In the bathtub trap geometry ${{\rm{\Omega }}}_{0}={{\rm{\Omega }}}_{c}\times F({\tan }^{-1}\theta )$ where the function of tangent includes parameters describing the length and depth of the trap. The pitch angle dependence of P_λ(ω) comes from Ω₀, Ω_a, and in turn k_λ; in particular, the coefficients a_n(k_λ) describe the relative strength of each peak in the comb structure. Considerable discussion about these coefficients and their calculation for some models is included in [4].

We refer back to figure 1 in the previous section, now equipped to understand how this example event illustrates the behavior of equation (3).

• (a)  
  The signal takes the form of multiple parallel tracks corresponding to the different values of the frequency band order n. We call this structure a multi-peak track (MPT).
• (b)  
  At approximately 32 ms, the electron scatters with a residual gas molecule and changes the makeup of the MPT; this is consistent with an abrupt change in the pitch angle. In particular, the frequency and power of individual tracks is observed to be pitch-angle-dependent as expected.
• (c)  
  By contrast, the track slope—the change in frequency (energy) with respect to time—encodes the total radiated power and thus does not vary within tracks of any one MPT.

The dependence of the individual band power on the pitch angle for $n\leqslant 2$ is shown in figure 3 for a 32 keV electron in a bathtub trap; it is this individual band power, and not the total power, that corresponds to a single track in the spectrogram. Since high-order bands are never powerful enough for reconstruction, we in general restrict our discussions to only the mainband (n = 0) and one detected sideband order: n = 1 or n = 2 depending on the short interference effect. The dashed line in figure 3 shows an example detection threshold that is met only by the mainband and the n = 2 sideband for different but partially overlapping ranges of the pitch angle. This creates three allowed track types based on the pitch angle and the band order.

• (i)  
  Mainband high pitch angle: closest to 90° for n = 0.
• (ii)  
  Mainband low pitch angle: far from 90° for n = 0.
• (iii)  
  Sidebands: a single range for n = 2.

Zoom In Zoom Out Reset image size

Since the short interference is a wavelength-dependent effect, we expect the specific nature of the allowed pitch angle regions to vary with frequency. Indeed, in Project 8 Phase I the 32 keV tracks are well described by figure 3 and the three cases above; but the story is very different at 17 keV, where the mainband is almost completely suppressed and we detect only the n = 1 sidebands. For further discussion of this effect where the n = 1 sideband is visible we refer the reader to section VII of [4]. This exemplifies the powerful influence of the short, and the importance of understanding sideband effects.

So far, we have treated the axial motion as independent of the (x, y) plane on the basis that the magnetic field varies only with z, i.e. ${\rm{\nabla }}\vec{B}$ is always parallel to $\vec{B}$. However, to improve our description of the electron dynamics we must consider a small radial gradient of the form ${\rm{\nabla }}\vec{B}\times \vec{B}$ ¹² . This causes the guiding center of the cyclotron orbit to precess slowly (compared to the axial motion) in x and y, which in effect perturbs the one-dimensional magnetic trap profile B_z(z) with a slow time dependence. A small anti-symmetric tipping of the trap coils at angle ψ from $\hat{z}$ is the simplest way to recreate this ${\rm{\nabla }}\vec{B}\times \vec{B}$ perturbation in simulation. Consequently, this gradient induces a periodic drift in the axial frequency Ω_a:

$$\begin{eqnarray}&&{{\rm{\Omega }}}_{a}^{{\rm{{\prime} }}}(t)\approx {{\rm{\Omega }}}_{a}\left(1+\displaystyle \frac{r}{{l}_{1}}\sin \psi \sin ({{\rm{\Omega }}}_{m}t)\right),\end{eqnarray} \tag{ 4 }$$

where r is the radial position of the electron, l₁ is the characteristic trap length and Ω_m is the drift frequency.

The effect of this precession on the track signal now becomes clear: if the axial frequency varies sinusoidally, so will the frequency of the n > 0 (sideband) tracks. This oscillation has been observed in Phase I data, where it manifests as a track with an appreciable 'width' in frequency; figure 4 shows an example sideband track with this quality. Since the observed period of precession is $\sim 100\ \mu {\rm{s}}$, which is comparable to the DFT length ($40.96\ \mu {\rm{s}}$), the oscillation can be seen to some extent directly in the spectrogram.

Zoom In Zoom Out Reset image size

This observed frequency oscillation represents one of the primary motivations for a machine learning approach to signal classification. Its effect on the spectrogram is clear as illustrated in figure 4, and it is a unique property of sideband tracks which demonstrates the power to discriminate from mainbands. By extracting information from the spectrogram around a track, we can then apply machine learning techniques to identify sideband oscillation when it is present and label the track accordingly.

The cyclotron frequency Ω_c of a MPT structure can be calculated using the phenomenological model in [4] if both the reconstructed mainband frequency Ω₀ and the pitch angle θ of the electron are known. The result is in effect an energy correction, where the kinetic energy (and thus from many events, the energy spectrum) is calculated from the true cyclotron frequency rather than the frequency of any one reconstructed track. It is the end goal of the classification scheme to accomplish exactly this; first the identity of the mainband track must be established, and then pitch angle information extracted. We may extract the pitch angle in two ways.

• (i)  
  Axial frequency: for MPTs with a mainband and one or more sidebands, the axial frequency Ω_a is the frequency difference between the mainband and a sideband track divided by its order n. Ω_a may also be determined the same way from an event with multiple sidebands but no mainband. In Phase I, these cases comprise a minority of our data at about ∼10%.
• (ii)  
  Track slope: for MPTs with only a mainband, the pitch angle may be extracted from the track slope, which is proportional to the total radiated power in equation (3). Such cases comprise the majority of the data used for this work at about ∼90%.

The second method listed above has an ambiguity that must be addressed: in general, the track slope alone does not uniquely determine the pitch angle (this is evident from figure 3). To resolve this issue, we must also differentiate between mainbands of the high and low pitch angle regions. Our task is then to assign every track an appropriate topological label from the list in section 2.1: mainband high pitch angle, mainband low pitch angle, or sideband. Classification into these three groups will allow for an accurate measurement of the mainband frequency, the pitch angle, and in turn the true kinetic energy can be determined from the cyclotron frequency.

We approach this task with a machine learning model that uses a supervised learning method for classification. The overarching goal of the classification program is to use only those track features that are intrinsic to the signal itself and, through their inclusion, have the capability to improve the accuracy and robustness of the signal identification. As Project 8 moves to a tritium source, such a classification scheme will be vital to make an accurate measurement of the continuous spectrum and reach meaningful conclusions about the endpoint. In the remaining sections, we will develop the classification scheme and present the results on krypton Phase I data at all three energies of interest. The next steps of pitch angle calculation and the resultant energy correction are not yet implemented, but are of course a primary focus of future work to realize the full potential of track classification. We will also discuss the future impact of a more-developed classification process in the context of tritium endpoint sensitivity and full event reconstruction.

Primary track finding is the process of collecting high-power spectrogram bins into linear track signals, described thoroughly in [3]. At its conclusion, several parameters are formally calculated to describe each track candidate; these include the slope, start frequency, time length, and many others. We use the following three quantities as inputs to the classification model:

• TotalPowerDensity (W Hz⁻¹): the sum of power spectral density values in all bins that comprise the track cluster.
• TrackSlope (Hz s⁻¹): the slope of the track as extracted from regression analysis and a Hough transform [5].
• TimeLength (s): the difference between the track end time and start time.

Recall that the slope of a track is directly proportional to the total power emitted by the electron, and the slope and individual track power together determine the pitch angle information as illustrated in figure 3. Thus, the correlation between slope and track power has strong discrimination power between regions of high and low pitch angle. Figure 5 illustrates this correlation for 32 keV electrons and shows a good separation of the two mainband populations, which agrees with our understanding of the physical process from the phenomenological model. In this figure the sideband events populate the low PSD range across all slopes. This provides a clear motivation for the use of these two features as inputs to the classification model.

Zoom In Zoom Out Reset image size

The utility of the track length is primarily based in its correlation with the track power as well. Mainband tracks in general have a strong profile with power concentrated in one or a few of the frequency bins at each time slice. By contrast, sideband tracks often have power distributed across many bins due to the effect of the radial magnetic field gradient discussed in section 2.2. Sidebands also contain less power overall in the case of the 30 and 32 keV peaks. Consequently, the track power can have considerable dependence on the number of points which comprise the track, and the track length helps to bolster the discrimination ability between all three types in conjunction with the slope and power.

Radial magnetic field gradients create a sinusoidal variation in the axial frequency, effectively smearing sideband track power over several frequency bins while leaving the mainband track untouched. To extract a set of parameters from the spectrogram that quantify this difference, we first simplify the problem. After primary track finding, spectrograms with a known track are reprocessed with a 'Rotate-and-Project' operation, where they are effectively projected along the axis perpendicular to the track. This reduces the analysis to one-dimension—the projected spectrum—while preserving the most useful information about the track from the full spectrogram. The precise procedure is as follows.

• (i)  
  The known track is characterized by a slope q and intercept f₀.
• (ii)  
  The full spectrogram is reduced to a sparse spectrogram of only points that have SNR > 4.0 and that lie within the time bounds of the track. These points will be described by (t_j, f_j) where t denotes the time coordinate, f the frequency coordinate and j the point index.
• (iii)  
  The projected spectrum s at bin k is calculated as a function of the intercept, which we call ${\beta }_{k}$ and which sweeps the range f₀ ± Δf in discrete steps of δβ (both Δf and δβ are runtime-configurable parameters):

  $$\begin{eqnarray}&&{s}_{k}=\sum _{j}\exp -\displaystyle \frac{{({f}_{j}-{{qt}}_{j}-{\beta }_{k})}^{2}}{2{\sigma }^{2}},\end{eqnarray} \tag{ 5 }$$

  where β_k = f₀ − Δf + kδβ, q is the track slope, and σ is another runtime-configurable variable which describes the resolution of the spectrum. This calculation is a kernel density estimation with a Gaussian kernel and bandwith σ.

At a minimum, σ should reflect the inherent uncertainty in each point location, which is roughly the bin size; this way, the spectrum is not strongly affected by how precisely the choices of Δf, δβ, or the reconstructed value f₀ coincide with the discrete binning of the spectrogram. σ can also be made much larger than the bin size, and the projected spectrum gains sensitivity to structures which span a similarly larger bandwidth; however to retain good sensitivity to the sharp mainband tracks, we keep σ similar to the bin width. There is no advantage to matching it exactly with the bin size, so for convenience we choose σ = 50 kHz which is approximately 2 bins. For the step size, we choose δβ = 25 kHz, or half the resolution σ and approximately 1 bin. The only requirements on the sweep range 2Δf are that it should be many times larger than the frequency bin size, and at a minimum large enough to capture the full amplitude of the sideband oscillation (∼1 MHz). We choose Δf = 4 MHz.

Figure 6 shows typical projected spectra corresponding to a mainband and a sideband track. The qualitative differences between the two remain clear: sideband spectra typically feature a wide double peak structure, contrasted with the sharp and high-amplitude profile of the mainband spectrum. The sideband spectrum amplitude is largest near the edges of the signal region because the axial motion is slowest at its turning points, thus depositing more power per bin; this effect can also be seen in the spectrogram (figure 4). Next, we use the ROOT library TSpectrum [8] to characterize peaks in the projected spectrum. This library fits a linear background b_k = ak + b and labels a point k as a peak if it meets all of the following criteria.

• (i)  
  Value is at least twice that of the background level: s_k ≥ 2b_k.
• (ii)  
  Peak amplitude meets or exceeds a minimum fraction r of the highest peak: ${s}_{k}-{b}_{k}\geqslant r\ {\sup }_{j}\{{s}_{j}-{b}_{j}\}$.
• (iii)  
  Value is a local maximum within m bins: ${s}_{k}\,\gt \,{s}_{j}\ \forall j\ | 0\lt | j-k| \leqslant m$. The frequency range corresponding to m bins is $m\ \delta \beta$.

Zoom In Zoom Out Reset image size

We choose r = 0.4 and m = 5. The values of these and the other configurable parameters from equation (5) are listed in table 2. Once the peak locations are determined, the full spectrum is fit to a sum of n Gaussian functions where n is the number of peaks found. Only a handful of tracks in our studies produced a spectrum with 3 or more peaks, and none with more than 6.

From the spectrum s_k and the results of the Gaussian fit, we extract a total of 11 additional parameters for use with the classifier:

• Average, RMS, Skewness, Kurtosis: first four statistical moments of the spectrum s_k. Average and RMS are in units of MHz, and Average is shifted by f₀ so that 0 corresponds to the center of the spectrum.
• MeanCentral, SigmaCentral, NormCentral, MaximumCentral: extracted fit parameters of the Gaussian with mean closest to f₀ (the most central peak). MeanCentral is shifted by f₀ as described above, and MaximumCentral = b₀ + (2π)⁻¹ NormCentral / SigmaCentral where b₀ is the background level at the peak location. Both MeanCentral and SigmaCentral are in units of MHz.
• NPeaks: number of peaks found by TSpectrum for the Gaussian fit.
• RMSAwayFromCentral: rms ${\left(\langle {s}^{2}\rangle -\langle s{\rangle }^{2}\right)}^{1/2}$ of points greater than 3 times SigmaCentral away from the most central peak.
• CentralPowerFraction: average value $\langle s\rangle$ of bins within 3 times SigmaCentral of the most central peak divided by the average value of all points in the spectrum.

In the event that the Gaussian fit fails to converge, all of the parameters that depend on it are obviously unreliable and some can be undefined. To circumvent this, we simply remove any tracks from the analysis that have an unsuccessful fit, or which have any parameters undefined as a result of an improper fit. These represent between 5% and 10% of all tracks in various data sets we have used. Combining these with the track parameters discussed in the previous subsection, we have a total of 14 parameters to use for ML-based classification. The slope, power, and track length have the ability to distinguish between all three track topologies based on the phenomenological model; the projected spectrum provides an additional 11 parameters which distinguish sidebands from the two mainband topologies. In the next section we discuss the implementation of this new analysis.

In the case of a tritium spectrum, the region of interest will be a window spanning approximately 4 keV (200 MHz) around the endpoint value Q = 18.6 keV. A classifier will be necessary to understand the CRES signal in this region if sidebands are present and, overall, if energy corrections are to be applied in an event-by-event fashion. With ^83mKr as a calibration source gas, we may first study the classifier results by training our model on the 30 keV peaks and applying it to both 30 and 32 keV peaks simultaneously. This 2 keV energy separation¹³ serves as a test of classifier reliability across an energy range similar to the tritium window; we call this configuration the narrowband classifier.

The results of the optimization outlined in the previous section (in detail in appendix A), picked from a cross-validation accuracy of 92.0% ± 0.8% as a mean over three-folds, gives us the following SVM hyperparameters:

$$\begin{eqnarray*}\begin{array}{rcl}C & = & 108.01\\ \gamma & = & 2.947\times {10}^{-3}.\end{array}\end{eqnarray*}$$

These values are indicative of a 'smooth' model (γ ≪ 1) which captures little of the data complexity in the feature space, but is balanced by the large value of C ≫ 1 which allows for highly nonlinear terms in the loss metric minimization, recovering some complexity in the decision plane.

The respective test set accuracy on the 30 keV range is 91.2%. We also generate the ROC curves for each class and average them as shown in figure 7. Across all classes we observe an AUROC over 0.9 which indicates that our model does very well at separating any individual type of track from the rest. The ROC curve for low pitch angle mainbands (in pink) has the lowest AUROC, which is most likely due to its relatively small population relative to the other two classes; this comes into effect at One-versus-Rest level where the former population is pitted against the latter simultaneously. Both average curves achieve a ROC over 0.960 putting us in an excellent range of model stability to compliment the high test accuracy obtained.

Zoom In Zoom Out Reset image size

In figure 8(a) we see the resulting classified track start frequency spectrum in the 30 keV range, the range on which this classifier was trained. We observe a clean separation of mainband tracks in blue (high pitch angle) and pink (low pitch angle) to sidebands in yellow that are mostly concentrated above 1150 MHz. The broad peak between 1160 and 1170 MHz corresponds to upper sidebands of 2nd order, giving a rough estimate of the axial frequency f_a ≈ 22.5 MHz; the n = 1 sidebands are predicted to be suppressed due to the short effect discussed in section 2.1. To see the separation between mainbands of different types more clearly, we study the slope populations in the frequency range 1118–1123 MHz. The overlap of high and low pitch angle mainbands in frequency space seen in figure 8(a) around the low pitch angle peaks is now apportioned, as evident in figure 9(a). The blue scatter points in the region around the low pitch angle peaks constitute true high pitch angle carriers whose true start frequency has been missed during primary track reconstruction. This separation allows a single-valued reconstruction of the pitch angle for a mainband of a given slope; recall that energy corrections may be performed once the pitch angle information is available

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

We also apply the narrowband classifier to the 32 keV group and obtain an accuracy of 92.8%, which surpasses the test set score. As can be seen in figure 8(b), the frequency spectrum sports a clean separation between mainbands and sidebands; in this case both upper (around 1090 MHz) and lower (around 1010 MHz) sidebands are visible. The short peak around 1040 MHz has also been classified as a mainband with its respective low pitch angle tail. This corresponds to the 32.14 keV krypton line, which would be extremely difficult to spot by eye, as its relative intensity is very low. The mainbands of the 31.9 keV lines are separated neatly in slope-frequency space as seen in figure 9(b), once again giving way for possible energy reconstruction through pitch angle extraction. The success of the narrowband model at 32 keV is reassuring that a similar technique could be applied to the tritium endpoint region with training on the 17 keV krypton peak.

We can further improve the performance of the narrowband SVM model by examining how useful each parameter is for accurate classification. An exhaustive search of the unique combinations of features from the 14-dimensional feature space results in 2¹⁴ − 1 = 16 383 iterations of the training algorithm. We train a SVM and evaluate the accuracy and the AUROC for each feature subset. The subset with the best overall performance may have improved accuracy compared to the full feature set, and will require less intense computation resources to train on a large data campaign.

To evaluate each subset SVM with a single metric, we use the sum in quadrature of the accuracy and the AUROC value:

$$\begin{eqnarray}&&{{\rm{\Delta }}}_{\mathrm{opt}}=\sqrt{{x}^{2}+{y}^{2}},\end{eqnarray} \tag{ 6 }$$

where x is the accuracy of the model and y is the AUROC. The maximum possible value of Δ_opt is $\sqrt{2}\approx 1.414$. Maximizing this combined metric allows us to asses both the model stability and its predictive power simultaneously.

Perhaps unsurprisingly, the value of Δ_opt is in general large for subsets with many features; a complex model has a greater capability for increased performance. However, some single-feature models are able to achieve high values of Δ_opt as well. Average exemplifies this, which alone yields a model with 90.1% accuracy and Δ_opt = 1.314. Of course, models with a single feature for classification run the risk of lacking enough variance to generalize effectively so we discard them as viable candidates. A global maximum of Δ_opt = 1.359 is achieved with a model utilizing 6 final parameters: TotalPowerDensity, TrackSlope, TimeLength, MeanCentral, NormCentral, and MaximumCentral. This amounts to an accuracy of 94.9%, an increase of 3.7% compared to the original result in section 5.1, and an AUROC of 0.97 on the test set.

The 17 keV peak presents a unique problem for the classifier; as discussed in section 2.1, the CRES signal power distribution across the sideband spectrum is energy-dependent as a consequence of the waveguide short. We have come to understand that the 17 keV Phase I data studied here consists almost entirely¹⁴ of pairs of 1st-order sideband tracks, due to a large suppression of the mainband peak from the short interference effect. We have studied more closely the region between the 17 keV sideband peaks and found an excess of power corresponding to an average SNR of 1.26; much too low for track reconstruction, but adequate in combination with other studies to confirm the sideband hypothesis. Consequently, we cannot train the classifier on 17 keV tracks since we have no mainband tracks to offer it.

Instead, we can first use the same classifier that was trained on 30 keV to evaluate the 17 keV tracks, and the results are subpar, with an accuracy of 75.5%. While a sizeable portion of tracks are still classified properly, we are confident that nearly all of those classified as main carriers are incorrect. However, this result is not unexpected; the power and slope correlation discussed in section 3.1 is also energy-dependent, and from section 5.2 we now understand these two parameters to be among the most decisive in the classification scheme. In the training scheme, the classifier becomes familiar with the 30 keV power-slope correlation, and this has a considerable negative influence when applied to the 17 keV peak where the true power-slope correlation is different. It should be noted that this effect is also present at 32 keV when trained on 30 keV, but from the comparable accuracy scores and AUROCs we conclude it is insignificant for this small energy difference, much to our advantage.

To improve the classifier performance on 17 keV data, we will consider two approaches: first, we use only the rotate-and-project parameters, which have no energy dependence. Second, we train and evaluate the 14-dimensional classifier simultaneously on all three peaks.

With the rotate-and-project parameters only, we re-train and test the classifier on 30 keV data, and apply it at 17 keV. In the test set (30 keV), the total accuracy decreases to 86.1%; this is reasonable given that the slope and power information, which was especially useful in discriminating between the two mainband types, is missing. When applied at 17 keV, we observe an accuracy of 78.9%, which is a modest improvement over the narrowband model (75.5%). The ratio of the 17 keV accuracy to that of 30 keV is more substantially improved: 0.917 compared with 0.829 for the narrowband. This suggests, as we hypothesized, that energy-dependent parameters are partially responsible for the shortcoming of the narrowband classifier at 17 keV. However, the rotate-and-project only performance is still far less than ideal and poor compared to the narrowband model at 30 and 32 keV, thus it does not provide us with a satisfactory alternative.

The second approach we explore considers all three energy ranges simultaneously for training; we call this model the wideband classifier. The 17 keV classified spectrum from this model is shown in figure 10. It is immediately clear that this model has by far the best performance at 17 keV, with an accuracy of 96.1%. The largest population of mainband tracks is broadly centered about ∼1750 MHz, which is at least 30 MHz above the (suppressed) mainband peak. Informal inspection of the slope-power correlation among these tracks, as well as the individual spectrograms, suggests they are indeed most consistent with true mainband tracks. We interpret this as evidence for satellite shake-up/shake-off electrons [12] to be further investigated. Similar broad peaks of mainbands at the 30 and 32keV ranges have also been observed at a similar separation in frequency (see figure 8). Overall, the 30 and 32 keV lines themselves also show improved accuracy scores compared to the original narrowband model with the full feature set: 92.3% (+1.1%) and 95.6% (+2.8%) respectively.

Zoom In Zoom Out Reset image size

The signal from a single electron in general takes the form of many reconstructed tracks, via (a) sideband power deposition and (b) scattering interactions with residual gas molecules. The sideband comb structure creates a group of parallel tracks as discussed in section 2.1, and discrete energy loss from a scattering interaction creates a 'jump' in the signal frequencies and in the pitch angle. After track finding, those tracks that belong to the same event (electron) are grouped together to obtain only the start frequency of the event as a whole. In the current event building scheme, this is accomplished with two stages in sequence corresponding to the two items above. First, individual tracks are combined into MPT objects based on a coincidence between start and end times. Many such MPTs are then joined into a single event using a similar coincidence check on the timestamps, this time a head-tail comparison. A full treatment of this process is given in [13]. However, the present event builder (with no classifier information) makes no statement about the identity of the mainband within a MPT structure; the start frequency of an event is simply defined as that of the first track (in time) within the first MPT of the event sequence. The classification scheme thus creates the potential for a more intelligent event building procedure which takes advantage of the labeled topologies to determine the true main carrier start frequency of an event.

One simple improvement is to utilize the classification labels to add consistency checks in event building. A MPT structure should logically contain no more than one mainband, and the start frequency of the MPT should be determined by this mainband track alone (if present). MPTs with two or more sidebands may also be checked to ensure the frequency spacing between them is consistent with a unique axial frequency. If a mainband track decreases in frequency after a scatter, indicating a sharp increase in the pitch angle, the accompanying decrease in axial frequency of the candidate sidebands may also be used as a check. These examples are only some of the many possibilities in which classification labels can enhance our reconstruction process.

In figure 11 we show some typical interesting event topologies which could benefit from an event builder that utilizes the classifier results.

• Top left: an event with a scatter that changes the pitch angle from the high to low region, according to the classifier. A faint sideband is also visible above the mainband after the scatter, but it was not reconstructed. It is interesting to note that this change in topology would be very difficult for a human labeller to identify, but is clearly easy for the classifier.
• Bottom left: a MPT with two lone sidebands. In this case, we can reconstruct the hidden mainband start frequency from the axial separation and determine the pitch angle correction.
• Bottom right: a MPT that the classifier has identified to contain two mainbands. This indicates an error, either in the classification or the MPT construction. To address this, we might consider the relative probabilities that either track is in fact a sideband (i.e. work event building information into the classifier), or simply discard the event if it cannot be made sensible.

An improved event builder that works in tandem with the classifier is crucial for proper reconstruction of a continuous tritium spectrum using CRES. Complex event topologies and sideband proliferation from lower energies in the spectrum continuum will demand a sophisticated understanding of the underlying nature of tracks. For example with atomic tritium, assuming 1 × 10¹⁸ atoms m⁻³ and a cylindrical voxel¹⁶ of 1 cm in diameter and 10 m in length, for events with, on average, ten tracks of length 80 μs each we expect 3.14 × 10¹⁵ atoms/voxel. Then, given an activity of 5.6 × 10⁶ per voxel for the entire spectrum, we expect about 1.69 events (about two events) present at all times when looking at a 1 keV window below the endpoint where only a 2 × 10⁻⁴ fraction of the activity is present. With two possibly overlapping events in a given spectrogram, the use of an accurate classifier will be decisive in identifying and separating the constituents of each; we expect that the model presented here is a decisive step toward that success. In figure 12 we outline the analysis steps discussed for this work, including the future work regarding (a) the event builder as discussed in this section, and (b) pitch angle corrections to extract the true cyclotron frequency.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Along with the improved event builder, a classifier helps us reuse or remove all but the misclassified sidebands at a high confidence level. However, it is still important to study the effect of sideband proliferation through misclassification on the tritium spectrum. Using the Morpho [14] interface to perform Hamiltonian Monte Carlo simulations with the Stan package [15], we model the electron kinematic variables and compute the detected track frequencies according to the discussions in section 2.1. The kinetic energy is drawn from the tritium spectrum probability distribution function with an endpoint of exactly Q = 18 600 eV and zero neutrino mass. The power in each of the mainband and the pair of n = 2 sideband tracks is then calculated for a circular waveguide with the same bathtub trap configuration as for the data used in this paper. For those electrons that become trapped, a uniform detection threshold is enforced on each track. The detected tracks are collected into a mainband spectrum, denoted p₀(E), and a sideband spectrum denoted p₂(E)¹⁷ . Here, E represents the inferred kinetic energy from the detected track frequency, which in the case of sideband tracks would constitute an erroneous reconstruction; this allows us to study the effects of both misclassification and the lack of pitch angle corrections. It is important to note that this simulation serves only as a toy model and does not reflect the complete design of the Project 8 detector in Phase I (from which the data presented here in earlier sections was taken), or Phase II which has since reconstructed the first ever tritium spectrum with CRES. However, the toy model is relevant as a mean to highlight and evaluate some of the challenges that sideband presence will bring when reconstructing any CRES spectrum.

If the probability for a track to be wrongly classified (either mainband as sideband, or vice versa) is uniformly α, then the spectrum of classified mainband tracks is:

$$\begin{eqnarray}&&p(E)=(1-\alpha ){p}_{0}(E)+\alpha \ {p}_{2}(E).\end{eqnarray} \tag{ 7 }$$

With α = 0, we obtain the main-carrier-only spectrum from a 'perfect' classifier. Since the energy of each event is calculated from the measured (mainband) frequency and not the cyclotron frequency, we expect to observe a shift to lower energy (higher frequency) in the endpoint. Indeed, figure 13 shows the simulated Kurie plot for perfect classification α = 0 (in black) and the fitted Q-value:

$$\begin{eqnarray}&&Q(\alpha =0)=18.461\ \mathrm{keV}\end{eqnarray} \tag{ 8 }$$

which deviates from the true input value by 139 eV. In the same figure we show the spectrum with α = 0.5 (random classification); now, the region above the endpoint is contaminated by sidebands. As a result, the endpoint is measured at:

$$\begin{eqnarray*}&&Q(\alpha =0.5)=19.028\ \mathrm{keV}\end{eqnarray*}$$

which exceeds the true value by 428 eV, or roughly 20 MHz: quite similar to the observed axial frequency.

Zoom In Zoom Out Reset image size

Lastly, we perform the Kurie fit for many values of α to see the dependence of Q in a more continuous form; figure 14 illustrates these results. Each Q-value (shown as the dashed line) represents the mean of 50 unique spectrum simulations for a fixed value of α; the band in light red spans the standard error of the mean on either side, and is dominated by the statistical uncertainty of each simulation. Though the simulations are themselves independent, we use the same set of simulations for all values of α. Thus, the uncertainty band in Q does not reflect the endpoint measurement uncertainty of the simulation, nor, more importantly, of any real Project 8 phase. Comparing to α = 0, we observe that even for α ≈ 10⁻² which is small compared to our demonstrated models, the endpoint shift is significant; a precise requirement or bound on α, however, is again not necessarily transferable to Project 8. Still, we may safely conclude that going forward with the classification models we must have the highest reasonable standard for accuracy. Our thorough understanding of the trap geometry and related systematics helps in addition to reduce the effects of sideband contamination in future Project 8 phases, through both design and trap configuration.

Zoom In Zoom Out Reset image size

We have clearly demonstrated with this simulation that both energy corrections and track classification will have a substantial influence on future CRES tritium results, with an observed endpoint shift on the scale of ${ \mathcal O }$(100 eV). The design and configuration of future phases of Project 8 are guided in part by the goal to suppress detectable sidebands and achieve sub-percent level misclassification. Incorporation of this improved track classification and pitch angle considerations will enable a CRES experiment like Project 8 to take the next step and achieve a competitive eV-scale endpoint sensitivity.