Thinker invariance: enabling deep neural networks for BCI across more people

Despite the revolution that deep neural networks (DNNs) have brought to machine learning (ML) generally, they have had a more muted effect for brain computer interfaces (BCI) [1]. Despite some successes with shallow models [2–5], only limited attempts have been made to use deeper models with more than 2–3 layers, or modern architecture choices such as residual connections, attention, and adversarial losses [2, 6, 7]. A common claim as to why DNNs are not more consistently viable in BCI is that there are not enough data to train a deep model. This claim is well supported by the fact that the largest impacts DNNs have had on traditional ML have been accomplished using datasets that are far larger than popular public BCI datasets or data that are feasibly recorded by a single lab. Consider that the ImageNet dataset used to train state-of-the-art image classifiers consists of ∼ 14.2 million unique images ⁴ , and the BERT natural language model is pre-trained with two datasets: one of 800 million words, and the second of 2.5 billion [8]. Contrast this with the summary Roy et al collected in their 2019 review of deep-learning electroencephalography (EEG) analysis where, aside from a few outliers, the vast majority of datasets used in EEG research contain well under 1 million examples (despite including highly imbalanced data, and clinical data such as sleep and seizure datasets less relevant to BCI). However, there is an even more insidious challenge. Why is it that for most datasets, single-subject classification is still more accurate than inter-subject classification (leveraging multiple subjects) [4]? This is not accounted for by simply a lack of training examples—including more subjects dramatically increases the number of trials from which to learn; rather, this exemplifies the degree to which each subject differs. Indeed, single-subject neural-network classifiers succeed despite poorly leveraging the entirety of a dataset.

BCIs are affected by several sources of variability including equipment, subjects, sessions, and even trials (keeping all else largely constant). These differences in context cause differing degrees of data drift, so that classifiers trained in one context are sub-optimal in another, even when they are meant to classify the same or similar tasks or paradigms. The resulting decrease in performance is then unsurprising; it crucially violates a fundamental axiom of trained classifiers and ML generally, that data encountered in practice (or in evaluation) should be independent and identically distributed (i.i.d.) with respect to the original training data. The consequences of this violation abound in practice. For example, Lawhern et al found with all the DNNs they considered, that when trained with the BCI Competition dataset IV 2a, a common motor imagery (MI) benchmark, classification accuracy was nearly halved for cross-subject training as compared to within-subject [3]. This was despite the fact that they matched or surpassed state-of-the-art single-subject classification using their shallow neural network EEGNet, and their more traditional filter-bank common spatial pattern (FBCSP) baseline.

In response, transfer learning (TL) is meant to overcome the pitfalls of limited data, and the variability between contexts by transferring shared qualities. Data from other contexts are leveraged to enhance a target context, thus better leveraging the entirety of relevant data available. While TL is an effective technique, there are two slightly different sub-types of TL that are often conflated in the BCI literature. We suggest that this conflation has prevented leveraging these aspects jointly, as they are not mutually exclusive. These two concepts are better disambiguated by the terms in the wider ML literature: domain adaptation (DA) and domain generalization (DG). Both of these approaches can be leveraged to enable classifiers developed for one or more subjects to be useful for one or more new, unseen people.

The differences between DA and DG lie in the treatment of the so called target domain(s) relative to the source domain(s). For the moment, we loosely consider domains as the distribution of trials from a single subject ⁵ , though we provide more formal definitions in section 2. DA is primarily concerned with adapting the trials from a set of source domains (subjects) to be most useful for one or more target domains (subjects). DG, on the other hand, does not target any specific domain, but rather transforms data from all domains in such a way as to make them most useful for any new relevant domain. Neither of these techniques is new to BCI, but they are commonly both referred to as TL without distinction and rarely, if ever, synthesized. Perhaps the most well-known examples of DA are methods that learn to select (optionally weighted) features from a pool of source subjects based on some criteria evaluated on target features. For example, the invariant common spatial patterns (iCSP) method proposed by Blankertz et al [9] required discovering a unique parameter ξ (optimization and cross validation were required before ξ was viable) for each new subject [9], with respect to the original pool of subjects; in effect, this is a simple, target specific transformation (mostly selection) of source domain features. Data alignment approaches such as Riemannian alignment (RA) are relevant examples of DG [10, 11]. RA ensures that all trial covariance matrices are aligned with respect to an a priori reference matrix and thus all trials function as features that can be leveraged for any other aligned trials. Crucially, the alignment is not dependent on classifiers, features, or subsequent learning for any domain-specific target; in the case of RA, alignment is performed with respect to the mean of auxiliary data from each subject to model subject-specific variability. In some work, both DG and DA approaches are compared, but they are featured in juxtaposition as mutually exclusive alternatives [12]. There is, however, no reason why a DG approach cannot be leveraged to enhance DA.

Work at the intersection of BCI, DNNs, and TL [13–17] has more commonly involved DA, typically through fine-tuning. This means using a pre-trained model of some sort, like a model trained jointly with a pool of subjects [13, 15]; however, others pretrain using the same subject's data [14] and, in a few cases, models used weights previously tuned for different tasks such as image recognition [18]. Using the network weights from a pretrained model, most if not all the weights are slightly adjusted (typically by using a small learning rate) to fit a small pool of target domain data. Some weights may be kept unchanged to minimize the shift from the original features, but the model is never expected to remain useful to non-target or even original training domains. The particular combination of DG and DNNs in the BCI literature is rare. While there are some examples of multi-task DG [19, 20], or larger scale data (such as clinical sleep data) self-supervision [21], the work of Ozdenizci et al is, to our knowledge, one of the only examples of generalizing across subjects [6]. In that work, Ozdenizci et al used a variational autoencoder to encode a latent space that is alternatively implicitly or explicitly generalized to be domain (subject) non-specific. This is meant to generate features of specific trials such that they are independent of subject. Unfortunately, their performance was relatively underwhelming when compared against work such as Dose et al [15], who used the same public MI dataset and a shallower network. While this seems to imply that DG may be superfluous, there is good reason, outlined in section 2, to suggest that their results were due to their choice of shallow architecture, one that was not sufficiently expressive.

There are gaps in both the DA and DG approaches taken with DNNs in prior work. Fine-tuning was originally designed to transfer abilities from a large generic set of images to a specific (sub)domain, for example, leveraging general natural images to specifically predict skin-cancer from images of skin [22]. Once transferred, there is no need for the skin-cancer classifier to distinguish between, say, lions and tigers (as may have been the case in the original data). While there is an argument for completely personalized BCI classifiers, we suggest that preserving out-of-target predictions remains valuable. As a single model's subject-specific performance is improved, it is preferable that performance is improved (or at least unharmed) for unseen subjects. In contrast, considering the image analogy from above: with more images of skin, we would be indifferent to performance changes in the recognition of big cats. The current gap we observe in DG amounts simply to weak resulting performance, if using DG weakens the resulting accuracy of a classifier as compared to not using it, a loss of information is the cost of generalization, appropriate DG methods should perform no worse than unaugmented training.

In this work, we present three contributions to thinker-invariant DNN training: DG-focused training augmentations that leverage non-target subjects to outperform subject-specific shallow classifiers, a multi-domain learning (MDL) procedure that leverages both DG and DA to outperform either in isolation, and an application-appropriate deeper DNN architecture that can make better use of the benefits of the two former contributions. These are evaluated using five separate publicly accessible datasets: two MI datasets (in terms of subjects, one small and one large), two rapid series visual presentation (RSVP) datasets (a visual oddball paradigm and P300 speller), and an error-related negativity (ERN) dataset. For each dataset we limit pre-processing, we perform no additional filtering unless needed to prevent aliasing, we include trials marked as 'unusable' for training, and no dataset-specific normalization with respect to baselines or otherwise is done. We simply crop the trials in accordance with the task and scale values between −1 and 1. We do this in an effort to demonstrate this DNN architecture and overall methodology can be easily adapted to new applications with minimal prior knowledge. To aid this further, we have made all of the code used in our work available at https://github.com/SPOClab-ca/ThinkerInvariance including both the PyTorch implementation that we used for our own experiments, and a reproduction using the more approachable Keras API as a convenience for those interested in applying this work elsewhere.

A supervised learning problem in its most general form is taking some sort of feature space $\mathcal{X}$ and a label space $\mathcal{Y}$ and learning a function $f: \mathcal{X} \to \mathcal{Y}$. Any individual instance of a supervised task can be seen as $(x, y) \in \mathcal{X} \times \mathcal{Y}$ with a probability distribution P(x, y). The goal is learning f to approximate $P(y\,|\,x)$. For such problems, we can define domains as $\mathcal{D} = \{\mathcal{X}, P(x)\}$, where P(x) is the posterior probability of each $x \in \mathcal{X}$ for our task $\mathcal{T} = \{\mathcal{Y}, f(\cdot)\}$. Furthermore, if we focus on a BCI classifier with a set of possible subjects T that constitute our domain, we can restate our domain as $\mathcal{D} = \bigcup_{t \in T} \mathcal{D}_t$. We make the assumption that all $\mathcal{D}_t \subset \mathcal{D}$ given the common difficulty of leveraging one subject's data for another. In other words, we assume no single subject can represent the entire task domain alone.

Consider the possible sources of error that prevent an ideal determination of $P(y\,|\,x)$, which we use mostly interchangeably with f(·). Empirical risk minimization characterizes the process of learning a supervised task using a loss function l, which makes up the vast majority of modern ML, neural networks or otherwise [23, 24]. We can define risk for some hypothesis h of the label function f

$$\begin{equation*} R(h) = \int l(h(x), y)dP(x, y) = \mathbb{E}[l(h(x), y)] \end{equation*}$$

Following Wang et al [23], we can break this risk down into two terms, considering three different hypotheses:

• 1.  
  $h_{opt} = argmin_hR(h)$, the optimal hypothesis in terms of risk.
• 2.  
  $h_{arch} = argmin_{h \in \mathcal{H}}R(h)$, the optimal hypothesis given the constraints imposed by the hypothesis space.
• 3.  
  $h_{emp} = argmin_{h \in \mathcal{H}} R_{emp}(h)$, the hypothesis constrained by the empirical estimate of risk for a given hypothesis space.

We assume that there exists an h_opt that is uniquely optimal for each subject, and the union of all people. The error of the empirically constrained solution, with respect to the optimum, can now be expressed as the sum of two terms: the estimation error and the approximation error [23]:

$$\begin{align*} \mathbb{E}[R(h_{emp}) - R(h_{opt})] & = \mathbb{E}[R(h_{emp}) - R(h_{arch})] \\ & \quad + \mathbb{E}[R(h_{arch}) - R(h_{opt})]\\ & = \mathcal{E}_{est} + \mathcal{E}_{approx} \end{align*}$$

The more expressive a model is, the greater its theoretical capacity to mimic some arbitrary function, which is accounted for by $\mathcal{E}_{approx}$. Similarly, $\mathcal{E}_{est}$ represents the better understanding captured with more or better data during training. Consider a classifier (e.g. a shallow network, or constrained kernel-based classifier) exhibiting high performance (thus low risk) within a single subject domain D_ss —which error terms are likely to be low? Under our assumption of a unique optimum, we can make some inferences about its error from the change in performance from adding new subjects. If performance remains high, both terms are already likely low. The empirical estimate of P(x, y) would seem to satisfactorily cover additional domains, and the architecture satisfactorily approaches the globally relevant optimum. However, what if performance deteriorates with the addition of new subjects? There should be little change in $\mathcal{E}_{approx}$ if using an appropriate classifier, as if it were globally appropriate, adding more relevant data should not affect this error (i.e. it is not dependent on R_emp ). If there were a change, it implies that the classifier was only optimal in a single domain (subject) and not globally optimal—in other words, overfitting. If the classifier were appropriate, then $\mathcal{E}_{est}$ has increased due to R_emp . DG is then an attempt to transform g(·) each person as domain D_t to a new invariant domain $D_{TI} \approx g(D_i), \forall i$, so that $\mathcal{E}_{est}$ is consistent across different people, and each additional subject seen during training should serve to bring h_emp closer to h_arch .

Consider that, for example, EEGNet has been highly successful on a subject-specific basis, but not when leveraging more subjects [3]. This would seem to encapsulate the challenge we describe above, so we propose to use DG to improve E_est , and then minimize $\mathcal{E}_{approx}$ using a potentially more suitable hypothesis space to find $h_{arch} \approx h_{opt}$, for example a DNN that is deeper and more expressive. This is in general supported by two important factors: the hypothesis space itself, i.e. the choice of architecture and the operations it can perform, and secondly, how it is searched (i.e. how the argmin_h is performed). Crucially, some $h \in \mathcal{H}$ are easier to find than others, and thus the optimization procedure employed plays a large role [23]. As is detailed further in the sections below, to avoid biasing our results by using better optimization techniques than might not have been available to our baselines, we re-implement and train both ours and baseline models using the same optimization procedures. We address the choice of hypothesis space by presenting what we claim is an improved DNN architecture for BCI generally (where it is closer to h_opt than prior work).

3.1. TIDNet: DenseNet-inspired isolated spatio-temporal operations

Zoom In Zoom Out Reset image size

The most commonly used DNNs in BCI work are the shallow convolutional network (SCN; its deep counterpart seemed to be featured less often in the work we investigated) proposed by Schirrmeister et al [2] and the EEGNet architecture proposed by Lawhern et al [3]. The common factor in these networks, and others inspired by them [15, 20, 25], is the marked separation of temporal and spatial convolutions. Our proposal is also inspired by this feature, but we offer two intertwined criticisms of the previous approaches that we address with our own: both the SCN and EEGNet architectures do not use a non-linear activation between the isolated temporal and spatial networks, but rather apply two linear operations. Secondly, the structure of these architectures provides no clear approach with which to increase the number of layers used, particularly while respecting spatiotemporal isolation. These are related criticisms in some sense as, if there were a non-linearity between the temporal and spatial operations, simply repeating either operation may be an effective strategy, which we explored in our own previous work [26], at increasing architecture depth and model complexity.

Interestingly the major drawback of the absent non-linearity is likely in fact the advantage of these two approaches, as this operation remains nicely constrained. While the absence of non-linear activation preserves an overall linear nature to the composition of the temporal and spatial operations, the convolution operations impose constraints. Given the apparent advantage of these isolated operations over convolution operations that span time and channels (space), these constraints are crucial. If these linear layers were simply repeated, the composition still remains of course linear, but through successive composition the overall operation is less constrained. In other words, simply adding more linear layers amounts to loosening the necessary constraint and enabling the sub-optimal operation that spans both time and channels (space). The challenge then is ensuring that appropriate spatiotemporal isolation constraints are preserved, while increasing the depth and complexity.

To address the concerns above, and to focus on extracting appropriate features, we construct a thinker-invariant DenseNet-inspired DNN (TIDNet). It features two clear semi-isolated stages, with operations that focus on extracting temporal and spatial features respectively, followed by a simple classification layer. Residual connections exist throughout, but crucially take two different forms depending on the stage. Similarly, standard non-linearity operations are used throughout the architecture, in this case, the leaky rectified linear unit [27]. By design, the architecture depth can be easily extended, independently in each stage, to model more complex temporal or spatial features. Although, for consistency of dataset treatments, we do not test this feature here, opting to focus on its features 'out-of-the-box', any hyperparameters not detailed in the next sections can be found in Appendix A.

3.1.1. TIDNet: temporal stage

Looking specifically at the temporal filtering stage of TIDNet, figure 2 shows the time series of a single channel and two convolution layers. We show a single sequence at each layer for simplicity, but there is typically more than a single channel, and TIDNet consists of many more filters at each layer. Each convolution operation spans only along the temporal sequence of each channel, in keeping with the principle of isolation we discuss above. The convolution weights are regularized using weight normalization [28], and the result of each convolution operation is followed by our non-linearity and by filter-level dropout; the effect of different filters are removed during training [28].

Zoom In Zoom Out Reset image size

The major highlight of this structure is in including both dilation and a residual connection, similar to work in convolution-based sequence modelling [28]. Dilated convolution operations are performed by skipping d − 1 samples in the weighted sum of the convolution for a dilation rate of d. So, for a dilation rate of d and samples s_t , t ∈ [0, T), if a convolution operation with three weights ${w_1, w_2, w_3}$ was applied, the subsequent output is $\tilde{s_t} = w_1 \cdot s_{t-d} + w_2 \cdot s_{t} + w_3 \cdot s_{t+d}$. Dilations are used in TIDNet to increase the span of temporal operations over subsequent layers, without needlessly increasing the number of parameters. Since many BCI datasets are normally sampled at rates well above what is necessary, and knowing that handcrafted BCI features focus on trends over tens of samples rather than immediate changes, we begin the earliest layer with the largest dilation rate, equal to the number of layers in the stage, and reduce the dilation rate by 1 for each subsequent layer. The last layer is then a standard convolution with d = 1 (which is in fact the opposite of what would be more commonly used in speech processing [28] or in other uses with EEG [29]). To illustrate how this dramatically increases the potential capture for long-term trends with fewer parameters, consider the number of parameters needed to roughly match the temporal span of EEGNet (64 samples in length). With only two layers and 11 weights at each layer, the final output includes samples from the original sequence between t − 30 through t + 30, but consists of only 22 convolution parameters (roughly a third of the parameters needed by EEGNet). We find through our early design process that setting the number of weights per convolution at approximately 5% of the sequence (N_weights  = 0.05 · T) provided a good trade-off between speed and accuracy.

While dilation allows TIDNet to capture long-term trends, a residual connection facilitates integrating short-term trends. Residual connections vary in type, but here we simply perform the weighted sum of the unmodified input series with the output of the temporal stage up to this point. This also facilitates the stage constructing features in terms of difference with respect to the input, facilitating deeper structures and more stable training [28, 30]. Finally, it is important to note the dramatic increase in dimensionality that results. Multiple convolution kernels (i.e. filters) at each stage requires the expansion of the original matrix to a three dimensional tensor. Including batch dimension during training, the expansion is from 3 to 4 dimensional tensor. The temporal aspect of the output of this stage is therefore max-pooled to both limit the memory requirements and regularize the features.

3.1.2. TIDNet: spatial stage

The primary goal of this stage is to transform the channel space sequences to another latent space with strong discriminatory power. Given that the most common problem utilizing an overpowered network with limited data is the tendency to overfit, care is still needed when increasing the depth of the network. As discussed in the introduction, despite leveraging the data of many people, the number of training examples still remains small relative to other classical ML fields, so the regularizing effect of many examples is less reliable. While the dilations above help reduce the tendency to overfit, they are not similarly suited to channel space data. While the sequence of C channels is loosely spatial, it is only this way with some immediate neighbouring elements (and not so with other neighbours). There have been different attempts at re-creating the spatial nature of channel space for training DNNs [31, 32], but these are relatively less common and do not perform quite as well as networks that use channel space directly. Instead we opt for a system that reuses the activations of local integration from previous layers multiple times.

The DenseNet [33] family of DNNs is well known for its strong performances in image classification tasks while retaining parameter efficiency through its unique residual structure. We attempted to circumvent our overfitting concerns, and the poor spatial relation of our data format by simply translating the DenseNet structure to 1D convolution kernels (rather than the standard 2D image kernels). The advantage of this is that despite the depth of the network, the number of parameters remains small by the continual reuse of layers (other DNNs would need to re-learn partial transformations) [33]. An additional advantage is that gradients easily propagate back through the DenseNet network [33], potentially allowing a short-circuit when temporal information is of primary importance (e.g. ERP datasets).

The spatial stage then consists of multiple layers of translated DenseNet-like operations (figure 3a) that we called DenseFilters, followed by a final channel mixing step. The goal is to iteratively expand the filter space into a set of strong local features that are augmented by progressively global ones, and then remove the remaining effects of channel space, to ensure that the former has expressed the needed variation of the latter. Each DenseFilter (figure 3b) is composed of two layers, each consisting of a LeakyReLU, batch normalization [34], and convolution operation. The first layer creates a bottleneck—a 1x1 convolution (filter space transform) that expands the number of filters as a composition of incoming filter activations. The second layer, with activations from the expanded/composed filter set performs spatially isolated convolutions with a kernel span of 5 in channel space (1 in temporal space for isolation), selected for simplicity. The activations of the resulting filters are then concatenated with the original incoming filters.

Zoom In Zoom Out Reset image size

3.2. Euclidean alignment

Given our framing of DG as minimizing the empirical risk estimate of a single invariant domain rather than a set of possible subject domains, a simple methodology follows. We leverage characteristics consistent across all domains and try to express instances with respect to this reference. Euclidean alignment (EA) [10] is a data alignment technique that can be seen as narrowing the scope of P(X), and consequently P(y, x). As a DG strategy, it follows the methodology of projecting data into a domain-invariant space, leveraging the fact that EEG is a continuous multivariate time series, implying it has a symmetric positive definite covariance matrix. It takes its inspiration from RA, which shifts each trial for each subject with respect to the Riemannian mean of a set of resting trials, but instead trials under EA are expressed simply with respect to the identity matrix. While RA asserts perhaps a stronger prior, and thus may be more effective [1, 35], it requires the subsequent use of Riemannian classifiers, which operate on covariance matrices in Riemannian space rather than raw time-series. Although, previous work attempted to perform manifold learning with DNNs generally [36], the vast majority of DNN research is still tailored towards learning in Euclidean space. The advantage of EA is that operations are performed exclusively in Euclidean space and thus do not exclude standard DNN approaches. The alignment process of EA is centred around asserting a consistent mean covariance matrix of all the trials for each subject—in this case, the identity matrix. Consider a subject with n trials each represented by matrix M:

$$\begin{equation*} \bar{R} = \frac{1}{n} \sum_{i = 0}^n M_i M_i^T \end{equation*}$$

Alignment is then simply performed by using the matrix square root of the arithmetic mean:

$$\begin{equation*} \tilde{M}_i = \bar{R}^{- \frac{1}{2}}M_i \end{equation*}$$

After re-weighting M_i , the mean of all covariance matrices becomes the identity matrix I:

$$\begin{align*} \frac{1}{n} \sum_{i = 0}^n M_i M_i^T & = \frac{1}{n} \sum_{i = 0}^n \bar{R}^{- \frac{1}{2}} M_i M_i^T \bar{R}^{- \frac{1}{2}} \\ & = \bar{R}^{- \frac{1}{2}} \Big( \frac{1}{n} \sum_{i = 0}^n M_i M_i^T \Big) \bar{R}^{- \frac{1}{2}} \\ & = \bar{R}^{- \frac{1}{2}} \bar{R} \bar{R}^{- \frac{1}{2}} = I \end{align*}$$

Note that, while data for the subject is needed to establish the unaligned mean, labels are not needed. In the original presentation of EA, He et al visualized the pre- and post-alignment distributions of trials. The alignment procedure created much more homogeneous and less sparsely distributed data for each subject (P(x)). This suggests that it requires less data to achieve an accurate empirical estimate of the data distribution, as it is isolated within a smaller subspace. For convenience, alignment was computed for (and applied to the trials within) each single recorded file for each single subject. Finally, this process is notably similar in methodology as that of the more conventional process of spatial whitening [37], where all trials are first concatenated and then a global covariance is used to align data, instead of the mean covariance across trials.

3.3. Mixup

The second approach we consider is mixup regularization. This technique has the effect of enforcing more linear model behaviour in the immediate vicinity around training points [24], the degree of this is tuned by a hyperparameter α. It can also be interpreted as a data-augmentation technique as, in effect, linear interpolation is used to create artificial points between training points. However, it was originally motivated from the standpoint of improving the empirical estimate of P(y, x). It trades the naive empirical estimate, which is a probability distribution P(x, y) made up of the sum of Dirac masses located at each training point $(x_i, y_i)$, with a vicinal estimate that assumes a vicinity distribution ν around each point. The probability of finding the virtual point $(\tilde{x}, \tilde{y})$ is:

$$\begin{equation*} P(\tilde{x}, \tilde{y}) = \nu(\tilde{x}, \tilde{y} | x_i, y_i) \end{equation*}$$

Thus, mixup is an assumed prior distribution for the empirical learning problem that functions as a stochastic interpolation between existing points, driven by the parameters λ ∼ Beta(α, α) and $\alpha \in (0, \infty)$:

$$\begin{align*} \nu_{mixup}(\tilde{x}, \tilde{y}) & = \frac{1}{n} \sum_{j = 0}^n \mathbb{E}[\delta(\tilde{x} = \lambda \cdot x_i + (1 - \lambda) \cdot x_j, \\ \tilde{y} & = \lambda \cdot y_i + (1 - \lambda) \cdot y_j)] \end{align*}$$

Furthermore, Zhang et al suggested another potentially relevant feature of mixup that could benefit the approach we take here. As we are particularly interested in employing a relatively deep and computationally expressive neural network, it becomes easier for the network to fall into the degenerate case of memorizing training points [24]. Since mixup constantly presents a modified version (the degree of this is tuned by increasing α, if α = 0 we never present virtual points) of the original points, the exact values of the points become much more difficult to memorize outright [24], minimizing this risk when using a network with a greater capacity for memorization.

3.4. MDL as DG-DA fusion

3.5. Datasets

3.6. Model training and evaluation

We train each model using the Adam optimizer with the AMSGrad fix [49], and we used a cosine learning rate policy with warmup. This means the rate would peak at a default value of 0.001 after a linear warm-up period which lasted one fifth of the total epochs, and then a cosine decay for the remainder of training, which has been beneficial in many instances of deep network training [50]. The learning rate at any epoch e of total epochs E is:

$$\begin{equation*} \eta (e) = \left\{\begin{array}{ll}\eta_{\textrm{max}} \times \frac{5e}{E} & e \lt \frac{E}{5}\\ \eta_{\textrm{max}} \times \frac{1}{2} \Big( 1 + \textrm{cos}\big(\pi \frac{(e - \frac{E}{5})}{\frac{E}{4}} \big) \Big) & e \ge \frac{E}{5}\end{array}\right. \end{equation*}$$

We include a second-order weight decay (L2) of 0.01 across all network parameters and dropout throughout the TIDNet is 0.4. We also employ label smoothing as implemented by He et al [50] at a factor of 0.2 to penalize overly confident predictions, and model some noise in the labels. As done by Zhang et al in the original work on mixup, we first gather batches and apply mixup within each minibatch. Specifically, a minibatch is first accumulated with a set of trials, and then we apply mixup to each trial by matching it with another from the minibatch with replacement (this could lead to scenarios where a batch is matched with itself). The number of total epochs, batch size, α_mixup and number of final epochs to smooth weights over are the only parameters that vary between datasets (and they do not vary much), determined heuristically by finding values that ensured validation accuracy mostly increments over the first training fold/subject of the respective datasets. A summary of these hyperparameters for each dataset can be found in Appendix A. Wherever there was an imbalance in examples between classes, we under-sampled the majority class(es) without replacement so that the total samples drawn is equal to the number of classes multiplied by the number of examples in the minority class. Furthermore, as hinted above, to stabilize the final model weights, we save the parameters from the last N_ewma epochs and sum all the exponentially weighted parameters for each epoch e by $w^e_{ewma} = c_e \cdot 0.5^{N_{epochs}-e}$, where $c_e = w^e_{ewma}(\sum_{i}^{N_{ewma}}w^i_{ewma})^{-1}$).

Electro-ocular channels are ignored for the three datasets where they are provided (IV-2a, P300 and ERN) for consistency with the other datasets which are only available this way (see Appendix B for an ablation on the effects of EOG on TIDNet and baselines). The A1 and A2 reference electrodes from the P300 dataset are included in training to allow for learned referencing, see the end of Appendix B for results that also exclude these references. Secondly, we also include those trials that were originally annotated as bad, meaning predominantly artifacts or unusable. While these are normally rejected to increase classifier performance, in our experience with larger models, these additional trials do not hamper training. Filtering is limited to that performed by the original recording equipment or data providers, and down-sampling is only used in the P300 and N2PC datasets due to their much higher sampling rates. These are detailed in section 3.5. Each trial is normalized to range between −1 and 1, preserving relative scale between channels (trial values are divided by the maximum absolute value per trial), but no baseline subtraction is performed. Thus, any dataset-specific pre-processing that may be necessary is left to TIDNet to perform internally, as our approach aspires to be as much of a self-contained and end-to-end solution as possible. While we do not explicitly evaluate this, any specialized filtering or normalization that does increase the downstream signal to noise ratio should provide similar boosts in performance to any selected classifier, save for coincidental overfitting (where the pre-processing is likely not affecting signal to noise ratio at all).

3.6.1. Multi-domain learning

For our implementations of MDL, we reserve target data by taking the first of multiple runs if more than one and fewer than four runs are performed. Otherwise, we select a quarter of the test dataset, selecting the first $n_c \times p_c$ points, where n_c is the number of points in that class and p_c is the probability of that class observed for that subject. This is done for both validation and test subjects, so that the validation domain is an accurate simulation of test conditions, albeit with different subjects. Furthermore, in LMSO experiments, the above is performed for each test and validation subject in their respective folds. Thus, every MDL experiment has data from every subject for each respective dataset. We make the same statistical comparisons to baseline models under MDL evaluation.

We also compare how effective MDL is in comparison to fine-tuning the general model to specific individual subjects, as was done in previous work [4, 15], and as is common in other ML fields. Fine-tuning DNN parameters (or, at the very least, re-training the final classification layers) is generally a simple yet effective method for TL [51]. What is less often considered in the context of BCI is the potentially catastrophic effect on generality after multiple training stages [52]. We expected that a MDL approach would remain effective on unseen subjects while fine-tuning would not—a potential advantage if fine-tuning and MDL target performance are largely similar in performance. To this end, we focus only on the two datasets that we trained in the LMSO scenario, providing a test-bench subset of four tasks, three from the MMI dataset and one from the ERN. The reason we focus on LMSO datasets is to study the performance of both target subjects and unseen subjects. A subject-wise paired Wilcoxon test again identifies statistically detectable differences between fine-tuning and MDL for each task and either target or unseen subjects.

Focusing on the best performing LMSO configuration TIDNet (with EA and/or Mixup added), we target each subject in each test fold. To fine-tune a target-specific model, we refine the general model in two phases, using 50% of target data. The first phase trains a new classification layer over ten epochs, keeping all other parameters frozen. This means only the final layer's weights are updated after each batch. Then, the entire model, including previously frozen parameters and the new classification layer, is updated for five epochs using a tenth of the original learning rate. Both phases again use a cosine learning rate decay but no warmup. This is compared against a limited MDL approach, which consists of augmenting the training data (those that produced the general model above) with the same 50% of the target subject's data. The training procedure remains the same as for the general model. Each of these approaches results in a single model tuned for a single subject, using the same training data cumulatively. We then compare the performance of each of these models on the remaining target data and the remaining subjects in each of their respective test folds.

3.7. Number of subjects regression

Table 2 captures the average absolute improvement on the selected metrics for each dataset in section 3.5 with respect to the re-implemented reference models (performances from table 1), and table 3 shows the same when equivalently augmenting the reference model training with EA and Mixup. While TIDNet alone or with only mixup shows mixed results, once alignment is included there is a nearly universal performance benefit to the deeper network (TIDNet), this improvement is much more consistent when both mixup and EA are used, with only a single (statistically insignificant) case showing very mildly worse performance. While the more temporally discriminative datasets, such as the P300 and ERN, show relatively mild improvements overall, there are more dramatic performance benefits for both the IV-2a and MMI datasets, in some cases 7%–8% absolute increase in classification accuracy.

The MMI results show a strong increase over prior work, particularly when considering the 3- and 4-way comparisons. Interestingly, in all three tasks, our baseline outperformed what was to our knowledge the previous state-of-the-art [15]. Dose et al showed 80.38%, 69.82% and 58.58% for the three tasks respectively, whereas our EEGNet baseline achieved 80.93%, 72.39% and 61.53% (our re-implementation of the model by Dose et al was 81.42%, 71.76% and 48.93%, but was not stable training with EA or mixup, often degenerating into chance performance, we however are more suspicious of the model architecture than anything else for this). TIDNet alone outperformed all of these, with the 3 and 4-way comparison statistically significant. After including augmentations, in each task we significantly surpass state-of-the-art. Focusing on the largest increase in performance, we find a significant 8.91% increase in 4-way classification performance using EA, jumping from 61.53% to 70.44%.

Looking at the IV-2a dataset alone, results are more mixed as previous work by He et al used this dataset in their original presentation of EA and scored an average LOSO accuracy of 73.53% [10] using a more traditional set of features and a linear discriminant analysis classifier. While this significantly outperforms the LOSO approach which would be the closest equivalent comparison, the MDL approach is where TIDNet with EA and mixup score very well, which is (detectable with $p\lt 0.004$, but not significant after correction) 5.867% higher than the results by He et al. Furthermore, this score is 5.7% higher than the original subject-specific SCN [2], while the MDL SCN has a more modest 3.7% improvement (significances are unknown as only mean results were published).

With the P300 dataset we see improvement over baseline performance across all permutations considered. Notably, we can make a comparison to previous work where multiple P300 datasets are leveraged to augment performance [19]. Their performance on this dataset, without seeing it, but pre-trained on 5 other P300 tasks is 0.806 ± 0.004 3 (mean and standard deviation across ten fold cross-validation). They then fine-tuned the pre-trained model on 80% of the target dataset (this same P300 dataset), notably randomly from all subjects. To compare our own performance to theirs, the highest performing results are 0.824 ± 0.068 LMSO and 0.856 ± 0.061 7 MDL (which similarly includes data from all subjects, but leverages 83% of the dataset). Our results exhibit higher variability, but appear on par with their results, without any pre-processing beyond our rough down-sampling, and without pre-training with five additional datasets.

Considering the ERN dataset, TIDNet in conjunction with EA and/or mixup again outperforms the baseline, in a notably significant fashion, although these are relatively minor increases. This implies that the performance benefit, although mild, is highly consistent on a subject-to-subject basis. Furthermore, while EEGNet (our employed baseline) to our knowledge was the highest previously performing DNN on this dataset, and our results indicate that TIDNet with EA might be better, a previous method using covariance-based classification well exceeded our own performance scoring 0.846 mean AUROC to our 0.748, although this performance notably was the result of 500 bagged models trained, in aggregate, across all subjects and included meta-features beyond raw trial data (see https://github.com/alexandrebarachant/bci-challenge-ner-2015 for more information). Considering the closest comparison to our own arrangement, that similarly employed LMSO cross-validation, the performance is much lower at 0.729 AUROC, 0.024 lower than the best performing TIDNet configuration.

4.1. Targetted MDL vs. fine-tuning

4.2. Subject regression

By plotting the performance of TIDNet and EEGNet, both with and without DG against the number of subjects used for training, we model some of the differences between these conditions as a function of training subjects. Figure 4 shows the mean performance across 10 different cross-validation stratifications for 20 logarithmically spaced subsets of subjects on a log–log plot. We add log-space regression lines fit using individual test performances (of all points, not of the mean line) against the number of subjects used to train the model. When performing the regression, we exclude subsets with fewer than five subjects, as the mean performance below this point is not well suited to a logarithmic relationship. The 2-way figure stands out due to the relative saturation of performance levels compared to the 3 and 4 way tasks. It gives context to the scale of improvements in performance, as by 20 training subjects each configuration is well within 5% of the maximum value. Other features that stand out include the tendency for the DG-free shallow network to have good performance with fewer subjects, and for this performance to trend below the remaining configurations as more subjects are added. Similarly for TIDNet+EA, performance begins quite low, but makes steady improvement coming out ahead in terms of absolute performance and trend in the 3- and 4-way tasks. The addition of EA does not seem particularly helpful to TIDNet in the two-way task, but given the saturation of the performance, it is unclear. Otherwise, in terms of both trend and peak performance, EA seems consistently beneficial to EEGNet as well as TIDNet.

Zoom In Zoom Out Reset image size

While each regression line is determined with $p\lt \lt 10^{-5}$, the r-values are quite low for the 2-way classification (sub-figure (a)) with 0.49, 0.52, 0.53, and 0.34 for the TIDNet+EA, TIDNet, EEGNet+EA, and EEGNet lines, again indicating that, at least with this level of performance, the marginal addition of subjects provides little performance gain, but the shallow model without DG is the most affected. In contrast, the 3- and 4-way task exhibit much higher r-values, with EEGNet alone consistently having the lowest r-values in its group. In terms of model slope, we find a mostly consistent pattern of increasing slope values in the order of EEGNet, EEGNet with EA, TIDNet, and then TIDNet with EA. The only inconsistency was the promotion of EEGNet with EA to have the highest slope in the 2-way task, with all other positions staying consistent. This strongly suggests, at least for this dataset, that there is a greater capacity for leveraging additional subjects when using TIDNet and EA.

We suggest that the weakest aspect of TIDNet is an unnecessary loss of spatial information caused by treating each channel as an element of a largely unordered list, one that is crucially one-dimensional rather than that which would naturally better represent a 2D or 3D sensor layout. We preserve this for two main reasons: in this way, a deeper architecture is much more computationally tractable and, as of yet, there is no established common way to represent the sensor layout topology for use by a neural network. While there exists two somewhat common methods to do this—direct pixel to channel mapping [31] and an interpolated image of sensor values assigned locations based on azimuthal equidistant projection of sensor locations on the scalp [53] (and similarly with an MEG helmet in our previous work [26])—it is our opinion that neither of these has been established as superior even to the single list.

A notable difference in how data is presented to both TIDNet as well as the reference designs in this work is the normalization of trials between −1 and 1 rather than by z-score normalization or like variants [2, 3]. Early in our experimentation we noticed a slight performance improvement when using our normalization scheme over z-scoring. Given the sensitivities that neural networks have to the exploding and vanishing gradient problem as a result of the distribution of incoming data (whether at the input, or within the network) [34, 54], we suspected that incoming values were poorly distributed, potentially undermining the theory grounding things like weight-initialization schemes that are meant to prevent exploding or vanishing gradients [30, 54]. Our concern was specifically that maximum values where too widely distributed. Figure 5 plots the distribution of absolute maximum trial values after z-score normalization of the IV-2a dataset. Notice that there was quite a skewed distribution, far from the sorts of normally distributed activations that neural networks theory prefers [54] to consider. While these represent all the absolute maximums, and thus are not a direct parallel to activations or generally inputs as such, we decided that our strict confinement of these extremes to two values should mitigate this possible concern, while leaving the distribution of internal activations to be addressed by batch normalization [34]. While this calls to question whether z-scoring (along channels) is an appropriate input normalization for raw BCI-trial DNN classifiers, it is far from conclusive. Future work would do well to consider this problem more directly and whether alternative normalization schemes would further improve results.

Zoom In Zoom Out Reset image size

Above, we ultimately ignored any provided EOG channels rather than use them to perform any ocular signal removal. It is well known that these signals obfuscate the underlying neural signals, and are an undesirable feature of classification of exclusively neural signals. While this is known, we were unaware of any previous work that investigated the effects of EOG channels/ocular artifact rejection and shallow DNN variants like EEGNet and SCN in particular. In an effort to characterize how these signals affected TIDNet and our shallow references, we have appended a short ablation study in Appendix B, which found that even when including EOG channels, or when simple ocular artifact rejection (via highly correlated independent component analysis (ICA) components to EOG channels), a similar trend emerged as in the main body of this work, where TIDNet excelled when augmented with EA and mixup. We suggest that this is due to a higher reliance on signals that originate from neural sources but do not confirm this, feeling it was out of the scope of this work. While we feel the black box nature of DNNs is overstated when compared to the sometimes spurious collection of features that enable 'classical' classifiers to work, DNNs remain a very obfuscated tool nonetheless. It is therefore hard to say what exactly the DNNs are leveraging to make decisions. Here, we relied on the fact that performance carried over to completely unseen subjects to justify the applicability of TIDNet as a classifier for BCI. We do however believe that the use of 'explainable AI' (XAI) techniques will be crucial using DNN classifiers in BCI moving forward, if only for its potentially unique ability for hypothesis through data-mining [55], where even potentially new hypotheses might be developed by considering the operations learned by DNNs, rather than simply endeavouring to show that DNNs pick up on what are considered to be known features.

Strangely, a few of the shallow reference configurations, when paired with EA, failed to score above chance. It is hard to say what exactly caused this, but we suggest that a possible explanation might be the inability of the shallow classifier to separate trials when they are better distributed. Consider that when the distributions are sufficiently disparate, an ensemble of simple internal classifiers (one for each disparate grouping) might discover rules for each cluster. When the clusters are more aligned, the complexity of the arrangement may have been too difficult to overcome without greater degrees of freedom. This may also explain a few other observations: in figure 4, EEGNet+EA is consistently the poorest performing when only a handful of subjects are available, and also why in Appendix B, when ocular/EOG components were increasingly controlled for, the augmented shallow models were more strongly negatively affected. This latter point would be consistent with the ocular/EOG signals serving as a high signal-to-noise-ratio feature that could be used to, in some sense, select for the appropriate internal model. This is highly speculative to say the least, and even XAI techniques may serve simply to obscure the truth in the matter, but notably, we never noticed a case where TIDNet in conjunction with EA was similarly incapable of scoring above chance.

We originally hypothesized that combining EA and mixup would imbue performance increases akin to added data scale in a naturally complimentary way. Our empirical results are clear that using both consistently increases performance but, in terms of magnitude of increase, EA alone is many times better performing. Currently, using mixup alone without first aligning the data is never particularly helpful, which is consistent with the idea that many trials are proximate after alignment, but are not sufficiently nearby before alignment to take advantage of a vicinal prior. This was true for our baseline models as well, and we would therefore recommend their use in other LO/MSO situations, whether TIDNet is used or not. Furthermore, it should be noted that we narrow our focus of potential DG approaches to domain invariant features and data augmentation (EA and mixup, respectively), but there are other possible avenues of DG, MDL, and TL that have developed in recent years from the DNN research community [38, 56, 57].

We elected to compute possibly multiple alignments for each subject by performing EA with respect to each recorded file (no file had more than one person's data). This had the potential benefit of also addressing variations between sessions (in these datasets files and sessions are synonymous), which are an additional source of domain shift, but a large number of sessions, with less data to leverage per session may contribute to a poorer $\bar{R}$. In this vein we note that the relatively poorer performances of the N2PC and P300 datasets correlated with the extremes of this process: a single alignment procedure and up to 20 separate alignments respectively. It may be necessary to consider this trade-off in future work, but further study is required.

These results show a consistent performance gain using DG/MDL, most consistently prominent with the deeper TIDNet model, for offline public datasets—a step that is crucial for establishing a comparable benchmark to other work. Perhaps the most important next step is determining how exactly these techniques are applied to practical settings. It should be kept in mind that even awareness of established performance benchmarks serves as a sort of data over-fitting, and performance is likely exaggerated over novel settings. That said, LOSO/LMSO models (without EA) have a clear path to practical classification of new subjects with data collected from similar hardware and settings: offline global models serve as ready-to-use classifiers for new subjects. Slightly more challenging is how MDL could be leveraged in practice. The simplest answer seems to be to develop continually adaptive classifiers that would not otherwise be practical by fine-tuning. An 'online' paradigm where a single model could be constantly optimized by re-training with additional incoming data from new subjects. Given our results from section 4.1, this is preferable to fine-tuning as the target performance is both higher and less (if at all) destructive towards non-target subject accuracy. A single model then remains nearly universal, despite the addition of small sets of novel subject data (that is appropriately over-sampled as we do above). For example, starting from an original model M₀ trained with the original data pool D₀, a new target session is recorded: D₁. During this session, M₀ can be used for an already functional BCI, but during or prior to subsequent sessions for data D_n + 1, new models M_n are trained using $\bigcup_{i = 0}^{n}D_i$. This discussion however is yet to consider EA, which presents a slight challenge to this approach, particularly for the first session, as it requires a mean covariance matrix of trials. It relies on at least some representative domain data to align subsequent data, and so far these are notably trials, not simply resting data (although it is possible this may work as well). While we did not model the number of trials needed to successfully reap the benefits of EA, we explored the worst-case scenario of classifying completely unaligned trials (that is, raw trials) with a model previously trained with aligned data. Looking at non-target MDL performance (the same procedures as 4.1) for the 3- and 4-way MMI tasks (2-way performance was maximized without EA), and the ERN task, we find mean (completely unseen subject) performance drops to 69.07%, 56.14%, and 0.649 AUROC respectively. Interestingly, the MMI performance drops to very similar levels as successful prior work; i.e. Dose et al achieved LMSO accuracies of 69.82% and 58.58% respectively [15]. On the other hand, the ERN performance fared worse, scoring 0.035 lower than a non-augmented EEGNet, but these decreases in performance were much less dramatic than we expected, and indicate that EA could likely still be used in an online-bootstrapping fashion as we describe.

Another natural extension of this work is to use similar DG/MDL techniques to improve performance for similar tasks recorded in different recording contexts, e.g. different recording equipment with their own number of channels, sampling frequencies, and analog-to-digital conversions. While prior work has learned like-tasks embeddings, creating a projection to a common task subspace for later classification [19], integrating more explicit DG methodology or a MDL approach may further improve their performance. We hope to develop this idea further and invite others to consider integrating these approaches.

The large improvement in performance we observed when expanding our trial window to 6 s in section 4.1, and the relatively little improvement over prior work we saw with the N2PC dataset (eight channels) would make it unsurprising if TIDNet was not well suited to data collected with a limited number of electrodes. In general, it appeared to us that our approach is best used with long sequences and increased electrode densities and when no particular bands of import have been selected for. Part of our goal is to develop classifiers that are successful without the need for dataset-specific pre-processing, and greater performance using longer sequences and many electrodes suits this nicely.

Our results should help inform how TL in BCI, and particularly with DNNs, is performed. We showed that disambiguating DG and domain adaptation has particular merit, and that leveraging them in a more complementary fashion can be beneficial. TIDNet's performance was improved through both fine-tuning (DA), and also generally through a mixture of EA + mixup (DG), but the largest improvements are seen through our proposed MDL framework that leverages both DG and DA, where a single model is well suited to new (i.e. target) subjects with little training data and to completely unseen ones. TIDNet, our proposed deeper DNN, while not necessarily the strongest choice before DG, is most consistently preferable after, which suggests to us that deeper models (TIDNet or otherwise) have thus far been prohibited by the poor empirical estimation of P(x, y), and that deeper models may excel both with limited data but appropriate application of DG, and with the continued addition of more subjects (and consequently data). In either case, this results in a single, more universal classifier than prior convention. Our results are all empirically validated using publicly accessible data, and the code used is available in its entirety, which we hope will help continue this line of research by others.

We would like to emphasize our thanks to those who make data they have recorded publicly accessible to all. Rudzicz is supported by a CIFAR Chair in Artificial Intelligence. Also, this work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government [20ZS1100, Core Technology Research for Self-Improving Integrated Artificial Intelligence System].

A.1. TIDNet

A.2. Dataset-specific

While it is well known that EOG electrodes may increase performance (and will confound results to say the least), we were unaware of any previous efforts to determine the susceptibility of common DNN architectures (such as EEGNet or SCN) in particular to EOG signals. We performed a short ablation study that compared the effects of: 1. Simply not using EOG electrodes (the results presented in the main body of this work), 2. Including the EOG electrodes during training, and 3. Performing a simple Pearson-correlation-based ICA component artifact rejection on the three datasets that included EOG channel recordings: IV-2a, P300 and ERN. In particular, the ICA rejection involved decomposing the N_c EEG channels into $\lfloor \frac{N_c}{2} \rfloor$ components using fastica [58]. Then using the automated (simply thresholded Pearson correlation values to EOG channels) EOG rejection provided by the MNE library (https://mne.tools/stable/generated/mne.preprocessing.ICA.html?mne.preprocessing.ICA.find_bads_eog#mne.preprocessing.ICA.find_bads_eog), no more than 3 (most correlated) components are rejected before translating ICA components back into EEG channel space. Table B2 shows the differences in performance across each of these three configurations across all three datasets. While the sheer number of comparisons make statistical analysis of all these performances difficult, we elected to include these as an appendix to discuss the trends we believed these data indicated.

It is immediately clear from the first two blocked rows of table B2 that the EOG electrodes and signals increased classification accuracy with the IV-2a dataset. With a continual decrease in performance with each measure introduced to remove EOG. In fact, performing classification using only the three EOG electrodes produced an average classification accuracy of 59.51% and 61.09% accuracy for TIDNet and SCN respectively (paired Wilcoxon test $p\lt 0.14$). The SCN seemed to perform better with no training augmentations, but TIDNet consistently made good use of the training augmentations, particularly so even when EOG correlated components were extracted (ICA-EOG rows), in which case SCN was more catastrophically affected.

The ERN dataset on the other hand indicates that EOG electrodes are relatively detrimental with the +EOG condition being the worse performing in nearly all cases, although interestingly, this effect is somewhat mitigated with EA and EA+mixup (Both) conditions. Otherwise however, we again see the reference model not benefiting as much as TIDNet from the EA and EA+mixup (Both) conditions, with the best performing condition for EEGNet being the Base condition. Notably however, it performs very similarly to TIDNet with augmentations. It should also be noted, that unlike IV-2a, single digits here are a 0.01 AUROC, which is probably less of a material difference than a percentage point.

The P300 dataset shows a similar decreasing pattern of performance, but is markedly less pronounced, with the decreases for the most part uniform between the two architectures. It is also worth noting that we considered the effect of simply ignoring the A1 and A2 ear reference channels recorded for the P300 dataset (at the same time, simply ignoring the EOG electrodes). We do not add them here, but interestingly its effect was largely the same as the ICA-EOG data configuration. Again, much like the other conditions with the P300 dataset, the effects were small and relatively uniform between models and training augmentations.

Our observations from these extended tests further compound the observations in the main body, that TIDNet excels when data is aligned, making better use of more similarly distributed data. On the whole, the reference implementations tended to respond more poorly to training augmentations the more ocular effects where removed, juxtaposed with the fact that in 16 of the 18 data configurations in table B2, the augmented TIDNet outperformed the base reference. The two instances where this was not the case, were neither statistically detectable differences, nor relatively large, with absolute differences of 0.2 and 0.3 expressed in the metrics above. We therefore suggest that the shallower EEGNet and SCN models might be more susceptible to strong confounding signals like ocular artifacts, and augmented TIDNet training is the safer choice in general.