On assessing neurofeedback effects: should double-blind replace neurophysiological mechanisms?

Sir,

We read with great interest the recent article of Schabus et al. entitled ‘Better than sham? A double-blind placebo-controlled neurofeedback study in primary insomnia’ published in Brain (Schabus et al., 2017) and its commentary ‘Neurofeedback or neuroplacebo?’ (Thibault et al., 2017). In recent years, EEG-neurofeedback (NFB) has benefited from a revival of interest, although its clinical efficacy remains a controversial and delicate issue (Micoulaud-Franchi and Fovet, 2016; Thibault and Raz, 2016; Sitaram et al., 2017). In the context of a general reproducibility crisis in science (Baker, 2016), we can only be delighted that negative results are being recognized in leading neurology journals such as Brain. The findings of Schabus et al. are of great scientific interest, contributing to a stimulating debate in the domain of EEG-NFB. However, we believe that caution is needed before generalizing these results to sensorimotor rhythm (SMR) frequency training as well as the entire field, as well as the entire of field of EEG-NFB. This is because irrespective of whether results from a single study are positive or negative, ‘one swallow does not a summer make’. Moreover, despite the study’s double-blind design, inconsistencies exist between earlier and current results published by the same research group.

It may at first appear surprising that Schabus et al. were not able to replicate some key neurophysiological relationships that have consistently emerged in their previous work. In 2008, with a sample of 27 healthy subjects, the same team demonstrated that 10 NFB sessions of SMR (12–15 Hz) upregulation was successful in: (i) conditioning an increase in relative SMR amplitude; (ii) eliciting positive changes in sleep parameters (sleep spindle number and sleep onset latency); (iii) eliciting changes in declarative memory performance (enhancement in retrieval score computed at immediate cued report) (Hoedlmoser et al., 2008). In the same vein, in 2014, Schabus et al. reported in 24 patients with insomnia disorder (iv) a positive correlation between SMR-NFB training enhancement, overnight memory consolidation and sleep spindle changes; and (v) a significant effect of SMR-NFB on objective sleep quality (a decrease in the number of awakenings, a trend towards decreased sleep onset latency and an increase in slow wave sleep) (Schabus et al., 2014).

Paradoxically, in their double-blind study involving a relatively lower number of 16 patients with insomnia disorder and nine patients with misperception insomnia, they did not reproduce a significant effect of SMR-NFB on objective measures of sleep quality, either in the active or in the placebo group (Schabus et al., 2017). However, in a well-controlled double-blind study unrelated to insomnia, Kober et al. replicated finding (i) and extended finding (iii) from immediate recall to 24-h delayed response (Kober et al., 2015a). Therefore, the use of double-blind control alone cannot explain all the null results of Schabus et al. (2017). Definite conclusions can only be drawn based on more solid data.

In particular, we think any discussion of NFB is incomplete without considering its basis from the point of view of neurophysiological mechanisms. Indeed, the previously established correlations between post-NFB sleep spindle generation and within-session NFB control highlight key neurophysiological mechanisms that are difficult to reduce to simple placebo processes and/or a single-blind design (Hoedlmoser et al., 2008; Schabus et al., 2014). From a historical perspective, seminal experiments by Sterman and colleagues (1970) were the first to show that waking SMR activity may be operantly conditioned to be more strongly expressed during subsequent sleep. Since long-term effects manifest in the same direction dictated by the training, a candidate mechanism may be Hebbian plasticity (see Ros et al., 2014 for a review). In fact, a host of other NFB studies indicate a similar Hebbian relationship between within-session and post-session EEG changes (Cho et al., 2008; Zoefel et al., 2011; Engelbregt et al., 2016). Importantly, online control of spectral power is necessary but insufficient as a demonstration of brain plasticity induction (i.e. a lasting change outside of the training session). Hence, a crucial question of mechanistic importance is why there was no association between the NFB training in the 2017 study with any change in offline spindle activity and/or offline SMR activity; as observed twice in the team’s previous work (Hoedlmoser et al., 2008; Schabus et al., 2014)?

In that respect, we noticed a potentially important methodological change between Schabus et al.'s 2008, 2014 and 2017 studies. Each time the authors used a different rule for setting and adapting the reward threshold. They decreased the threshold based on no particular rule in 2008, following ‘<5 success in a 3-min block’ in 2014, and ‘<13 success in a 5-min block’ in 2017. We wonder why such changes have been made, since the last two rules are not proportionally related, while in contrast the proportionality between the rules governing the threshold increase were preserved. Since this modification effectively makes the NFB task less challenging, it may have led to a decrease in the SMR activity during the 3-s baseline periods preceding each trial. The latter are used as references for each trial and may therefore have significantly changed when compared with the spontaneous (i.e. resting state) SMR activity of each subject prior to NFB. Hence, given that Fig. 2 only displays per cent evoked power from this respective 3-s baseline, it is hard to verify whether absolute SMR power during NFB actually exceeded the values of the resting state activity recorded before the start of NFB training. From a mechanistic standpoint, altering EEG activity significantly from its resting state value may be a critical determinant of plasticity induction (Ros et al., 2014). We therefore invite Schabus et al. to clarify this issue by submitting supplementary data providing absolute SMR changes during NFB relative to the initial 2-min resting state, unreferenced to the (potentially variable) 3-s baseline(s) before each trial. Such differences (Hoedlmoser et al., 2008; Schabus et al., 2014, 2017) may arise by simply using different learning indices, and/or the influence of the experimenter’s knowledge/un-blindness; however, they might also reflect differences in NFB learning and ultimately contribute to a reduced impact on brain plasticity.

Finally, and separate from the issue of whether brain plasticity was actually induced by Schabus et al.’s (2017) paradigm, logical reasoning should have restricted Thibault et al. to a simpler claim: that this specific NFB training, for this specific application, was not better than placebo. Instead, they appear to overgeneralize the null findings in the treatment of insomnia to the greater field of EEG-NFB. Such a position appears to us to be more ideological than scientific and contradicts the overall spirit necessary for advancing medical research. By jumping to conclusions, Thibault et al. forget that caution needs to be applied to both positive and negative findings. NFB should be considered as a unique tool for targeting specific neural activities rather than as a panacea for all brain disorders. Successful deployment of NFB critically depends on our knowledge of the brain’s inner workings, which still remains incomplete. Hence, to properly exploit this approach, there is an urgent need for more research to both optimize NFB learning (e.g. number of trials in a NFB block, threshold rules, number of training sessions) (Micoulaud-Franchi et al., 2016; Enriquez-Geppert et al., 2017) and to select the most appropriate training protocol for each disorder (Kober et al., 2015b).

In conclusion, the negative findings published by Schabus et al. excitingly generate more questions than answers for the field. Moreover, the last 50 years since NFB’s discovery cannot be considered a homogenous record, given several decades of relative dormancy in terms of research output, before a resurgence in the early 2000s. Hence, these results call for more research rather than less, including a deeper exploration of the neural mechanisms and methodological nuances emerging from this embryonic field—preferably before premature launches of double-blind clinical studies. It is more conceivable that the story of NFB is a simple reflection of the general scientific process, which, through its twists and turns, remains in the safer judgement of posterity.

Funding

T.F. is partially supported by a doctoral fellowship from the Pierre Deniker Foundation. F.L. received funding support from the French National Research Agency with the REBEL project (grant ANR-15-CE23-0013-01) and the European Research Council with the BrainConquest project (grant ERC-2016-STG-714567). R.J. is Principal Investigator of the INTRUDE project, awarded by the Agence Nationale de la Recherche (ANR-16-CE37-0015). This project will use fMRI-based Neurofeedback to relieve hallucinations and intrusive thoughts. This funding source had no direct involvement in the preparation of this letter. F.B.V. received funding from the company Urgotech (France) to investigate neurofeedback; this sponsor had no involvement in the preparation of this letter.

References